indexstore-db integration #229
Replies: 4 comments
-
for the geometry, i think the right approach is to write an overlay type around the the overlay should transform the base sequence into a new symbol-aware sequence of a more meaningful element type than a |
Beta Was this translation helpful? Give feedback.
-
now that we’ve implemented the syntax coloring, we can move on to the next task which is actually emitting link resolutions for these symbols. there are is a subtlety here that i did not originally anticipate IndexStoreDB returns everything, which is ideal - that is what we want it to do. in a normal project, this includes
we can filter out 1 by querying all the symbols in the current Snippet and excluding them from from the symbol table. we can also restrict exclusively to 3 by building a whitelist out of the symbols in the current package, which we have access to at build time. however, this is not what we want for the render API use case - we want to build documentation for we have a few alternatives. parse mangled symbols to discover module namethis is theoretically possible! (and would also solve some other outstanding issues in Unidoc) unfortunately the Swift library ecosystem kind of sucks and there is no up-to-date library that actually implements this, even though the Swift compiler ( link everything anywaywe can give up on filtering entirely and emit links for every even with our (very aggressive) request filtering, just 2.6 percent of our requests actually come from real, human developers, and the percentage decreases with each passing week as bots get more and more sophisticated. so enabling this might backfire in predictable ways. do the filtering at link timewe could emit all references at build time, and add logic to the linker to recompile the markdown at link time. but the problem is by then, the code snippets have been inlined into the larger article, and it would be complicated to treat the code snippet references differently than say, a normal cross-package symbol link, since they look the same at that stage of the pipeline. have the symbol graph compiler to perform whole-tree buildsa lot of passes that run on the linker don’t really make sense to run there, they only run there because the symbol graph compiler does not perform whole-tree builds. we originally designed it this way to avoid having to re-parse the standard library JSON with every documentation build, as JSON parsing is very, very slow in Swift. but now that we want to implement this feature, the tradeoff might be different. it would be a non-trivial refactoring effort, however. |
Beta Was this translation helpful? Give feedback.
-
implemented in #261 . we basically went with option 2, under the principle of “spend money now, ask questions later” |
Beta Was this translation helpful? Give feedback.
-
the PR #228 lands most of the far-reaching changes i anticipate that we will need for IndexStoreDB integration, without breaking compilation due to the complexities of depending on IndexStoreDB.
all of the IndexStoreDB-dependent stuff is currently gated by
#if canImport(IndexStoreDB)
. to enable the IndexStoreDB stuff, uncomment theindexstore-db
package dependency, and the.product(name: "IndexStoreDB", package: "indexstore-db"),
dependency of theMarkdownPluginSwift_IndexStoreDB
target.tasks
the IndexStore integration is not anywhere near a state where we can ship it; there are many outstanding issues that need to be solved and algorithms that need to be written. the purpose of #228 is to enable us to work on it in branches that touch a smaller subset of the code base.
in
SSGC.Toolchain.swift
, we have hardcoded the path/usr/lib/libIndexStore.so
, which is Linux-specific.SSGC.Toolchain
does not know how to locate the Swift runtime and will need to become able to do so.in the
MarkdownPluginSwift
module, we need to somehow correlate two-dimensionalSymbolOccurrence
s with SwiftSyntax’s one-dimensional syntax classifications, which are already non-contiguous due to Snippet slicing and de-indentation.this could be complicated, because
MarkdownPluginSwift
is already cuttingSnippetParser.Slice
ranges out ofSyntaxClassificationCursor
(making the highlights non-contiguous with respect to the original source text), so there is a lot of tough computational geometry that needs to be implemented here.on the bright side, this should be a relatively self-contained task that doesn’t require understanding anything outside the
MarkdownPluginSwift
module, except for the Markdown bytecode ABI.depending on how aggressively we want to link identifiers in the Snippets, we probably also to intern the USRs pulled out of the Snippets by the
Markdown.SwiftLanguage
plugin, and add them to the Symbol Graph reference table. but this can wait until after we have implemented the range intersection algorithm described in the previous bullet.unidoc becomes really difficult to build and test due to the extra C++ flags, and it becomes impossible to use swift-unidoc as a SwiftPM library because indexstore-db has no stable releases. we need to figure that out before merging this stuff into
master
.iteration
#228 adds a small test project
TestPackages/swift-snippets
.right now the stub implementation just dumps the 2D coordinates of the IndexStore symbols:
Beta Was this translation helpful? Give feedback.
All reactions