docs(util): state that TurboQuant's packed codec has no storage path

[sroussey]

Small; land with or right after #801. Documentation only — one module-header section, no code changed, nothing un-exported, no storage wired.

The gap

TurboQuantize.ts ships two independent encoders and nothing said so:

encoder	output	storage path
turboQuantizeToTypedArray	plain Int8Array / Int16Array	✅ any IVectorStorage fixed-width column
turboQuantize / turboDequantize / turboQuantizedInnerProduct / turboQuantizedCosineSimilarity (+ the two sizing helpers)	packed TurboQuantizeResult	❌ none

The packed codec is the interesting half — sub-byte widths (1–8 bits/dim), the exact 1-bit angle correction, pairwise comparability enforcement — and it is currently unreachable from any persistence path.

Verified: git grep for the six packed-codec names and TurboQuantizeResult across the branch, excluding the module and its own test, returns zero hits.

And the storage layer cannot accept the shape even if a caller wanted to:

• assertVectorShape (packages/storage/src/vector/assertVectorShape.ts:22-52), called by every backend on write and on query, requires an array-like whose length equals the declared dimensionality with every entry a finite number. A TurboQuantizeResult is a record and fails on the first check; its packed codes buffer fails on the second, since at 4 bits it holds two coordinates per byte and its length is ceil(paddedDimensions * bits / 8), not dimensions.
• Client-side-scoring backends — InMemoryVectorStorage (:161), SqliteVectorStorage (:151), SqliteAiVectorStorage (:660) — all call cosineSimilarity(query, vector) on raw numbers, with no hook to substitute turboQuantizedCosineSimilarity, the only function that can read these codes.
• pgvector-backed backends — PostgresVectorStorage (operators <=> / <-> / <#>) and SupabaseVectorStorage (a match_<table> RPC) — compute the distance server-side, where no client-side scorer can be injected at all.

Two rejected alternatives, and why documentation is the fix

Un-exporting was evaluated and rejected: the ~1200-line test suite imports those six names from @workglow/util/schema, so hiding them forces a cross-package deep import into packages/util/src (which nothing in packages/test does) or deleting ~700 lines of verified-correct code plus its tests.

Wiring storage needs a new column type, a client-side scoring path, and per-backend fallbacks for the server-side-distance engines — a multi-package feature, not a review fix.

So the fix is to state the split at the point of use: which encoder has a storage path, why the other cannot have one, what the packed record IS good for (in-process comparison — an in-memory candidate cache, a client-side rerank over a shortlist retrieved by other means), and an explicit "do not persist it expecting a retrieval path to exist."

Real IVectorStorage integration is tracked in #798, linked from the new section.

Note on the review's backend list

The review named "pgvector/sqlite-vec/DuckDB" as three server-side-distance engines. Re-checked against the tree: there are two pgvector-backed ones (PostgresVectorStorage, SupabaseVectorStorage); SqliteVectorStorage and SqliteAiVectorStorage in fact score client-side with cosineSimilarity, and this repo has no DuckDB vector storage at all (only DuckDbTabularStorage). The doc section is written to the verified picture. The conclusion is unchanged — and the client-side ones are blocked too, just by a different constraint (no scorer hook rather than server-side evaluation).

Verification

• bunx vitest run --project test packages/test/src/test/util/TurboQuantize.test.ts → 72 passed
• bunx eslint packages/util/src/vector/TurboQuantize.ts → exit 0
• bunx prettier --check packages/util/src/vector/TurboQuantize.ts → All matched files use Prettier code style!

---

Generated by Claude Code