fix(util,ai): TurboQuant norm overflow, ranking claim, decode warning, heterogeneous batches

[sroussey]

Stacked onto #354 (claude/integrate-arxiv-paper-VF55c), which is the base of this PR — not main. Six commits, deliberately not squashed, so review and revert are granular where the risk is.

Commits

1. fix(util): reject a TurboQuant record whose norm overflows a double

normalizeToUnit fixed the sumSquares accumulator overflow, but the norm is reconstructed as maxAbs * rootS and that product still overflows to Infinity for a large-magnitude finite input. turboQuantize returned it unvalidated, because assertQuantizeResultShape runs only on read paths:

turboQuantize(new Float64Array(4).fill(1e308), { bits: 8, seed: 42 })
// -> { norm: Infinity, ... }

turboDequantize, turboQuantizedCosineSimilarity and turboQuantizedInnerProduct all then threw on it — the encoder was writing a record nothing could read, and reporting it three calls later phrased as if the reader were malformed. It now throws at encode time, with a message that names maxAbs, states that the true L2 norm is not representable as a double, and gives the two ways out: prescale the input (cosine is scale-free, so prescaling changes no similarity), or use turboQuantizeToTypedArray, which discards the norm entirely and handles this input unchanged.

The check is deliberately not in normalizeToUnit: turboQuantizeToTypedArray calls it and destructures values only, so a 1e308 input is fine on that path today and stays fine. Also asserts assertQuantizeResultShape on the constructed record — the structural half, that every record this module hands out passes the check every reader applies, which today holds only by coincidence. O(1).

Rejected alternative: rescaling. There is nothing to rescale to — the overflow means the true L2 norm exceeds Number.MAX_VALUE and no double holds it. Representing it would need a log-norm or (mantissa, exponent) field plus a TURBO_QUANTIZE_VERSION bump, to serve inputs whose coordinates sit near 1e308.

2. test(util): measure TurboQuant's ranking fidelity per bit width — lands before the doc commit so the recorded numbers are measured rather than asserted. See the table below.

3. docs(util): correct TurboQuant's ranking claim to what the measurement shows

The module doc said ranking "was correct before and is correct now" and listed "Preserves the RANKING induced by inner products and cosine similarity" as a property. Those are two claims and only the first is true by construction. The true half is kept and sharpened — the correction is strictly monotone, so it moves every score and moves none past another, and switching it on or off cannot change a ranking — and now says explicitly that this is a statement about the correction. The Properties bullet carries the measured per-width ordering fidelity instead of the false blanket claim.

4. docs(util): warn that turboDequantize's output is not shrinkage-corrected

turboDequantize's docstring never said its output carries the uncorrected estimator at 1-4 bits. That warning lived only on turboPrepareQuery, which a caller reaches only if they already understood the problem — while turboDequantize is the function on the natural integration path, since a Float32Array is the only output shape an IVectorStorage backend accepts. Adds a section: decode-then-compare with plain cosineSimilarity reads a true 0.80 as 0.59 at 1 bit and low-but-less-so at 2-4; the corrected number comes from turboQuantizedCosineSimilarity or turboPrepareQuery + turboPreparedCosineSimilarity; at 5-8 bits the two routes are interchangeable, and that boundary is MAX_CORRECTED_BITS.

Promotes the private MAX_CORRECTED_BITS to an exported TURBO_MAX_CORRECTED_BITS = 4 so a caller can branch on that boundary programmatically instead of reading a docstring and hoping. The internal name stays as an alias — one literal.

No "corrected-compare" helper, and the docstring says why: the correction inverts a map defined on the code-domain ratio dot / (codeNormA * codeNormB), and a decoded Float32Array has already been renormalized to the recorded norm, destroying that ratio — so any helper taking two TurboQuantizeResults and returning a corrected number is turboQuantizedCosineSimilarity. The real trap (an IVectorStorage backend scoring decoded vectors server-side with no hook) is not fixable in this module and is already tracked as #798, which the module doc cites. The docstring says that rather than implying a workaround exists.

New test: sweeps bits 1-8 on one correlated pair and requires the corrected-vs-decoded gap to exceed 0.002 iff bits <= TURBO_MAX_CORRECTED_BITS — exporting a constant creates a new way for docs and behavior to diverge, and this fails in both directions.

5. fix(ai): reject a heterogeneous vector batch under method "turbo"

VectorQuantizeTask's dimensionality guard asks a per-vector question (is this length a power of two?) about a batch-level invariant, so [Float32Array(512), Float32Array(1024)] passed it unchanged: each vector was rotated in its own paddedLen under its own sign table (getSignTable(seed, paddedLen) is a different basis per length), while originalDimensions reported only vectors[0].length. Same with turboPadToPowerOf2: true and lengths 700/1100 (→ 1024/2048). Turbo output is comparable only against output with the same seed and, implicitly, the same padded length.

Turbo-only in this commit. An empty array now throws naming the input (it previously reached getVectorType(undefined) and threw "Unknown vector type: undefined"). Otherwise the first differing vector is reported with its index, both lengths, both padded lengths, the fact that the two live in different bases and are not comparable even after padding, and the remedy: quantize each dimensionality in a separate call.

6. fix(ai): reject a heterogeneous vector batch under every method

Moves the guard before the method branch with a method-aware consequence clause (for linear: the outputs keep their input lengths, so cosineSimilarity throws downstream — but originalDimensions misreports before that, and a fixed-width storage column accepts one and rejects the other), and folds in the mixed element-type case, which misreports originalType.

This is the only commit touching released behavior. VectorQuantizeTask's array input predates #354, unlike the codec and the turbo branch, so it is isolated so a reviewer who objects drops exactly this commit and the branch stays coherent. The justification: such a caller already receives a result whose originalDimensions/originalType describe only vectors[0] and whose outputs cannot share a storage column, so there is no correct use of that input shape — the break is from "silently wrong" to "loud". The input schema is additionalProperties: false with no opt-out, so there is no gradual path anyway.

Measured ranking fidelity (commit 2, quoted by commit 3)

Deterministic corpus, d = 1024 (a power of two, so nothing is padded and the case measures the estimator rather than the padding): 480 background documents, 10 planted at true cosine 0.70-0.88 (0.02 apart) and 10 near-misses at 0.30-0.48. The reference ranking is cosineSimilarity on the unquantized vectors, never the planted t — a finite sample's realized cosine differs from t by ~1/√d.

bits	recall@10	top-10 order identical	Kendall tau over the top 20
1	1.00	no	0.9368
2	1.00	yes	0.9789
4	1.00	yes	1.0000
8	1.00	yes	1.0000

Retrieval of a shortlist is unaffected at every width — the planted set comes back whole even at 1 bit — while the order within it is not: 1 bit transposes documents inside the top 10, and 2 bits reproduces the top 10 exactly and still transposes a pair further down. The 0.22 gap between the 10th and 11th planted document makes recall@10 insensitive to noise at every width, while the 0.02 spacing inside the top 10 makes ordering sensitive — the two metrics are decoupled so neither sits on a knife edge. recall and orderIdentical are pinned two-sided, which is what pins the corrected wording in both directions; tau is a floor, non-decreasing in bits.

Compatibility

packages/util/src/vector/TurboQuantize.ts is added by #354, so every codec-level export is unreleased and commits 1-4 carry no compat risk — the behavior commit 1 replaces is "returns a record all three read paths throw on", so the throw merely moves earlier. Commit 6 is the only released-behavior change; see its note above.

Verification

• bun scripts/test.ts util vitest — 54 files, 836 passed, 10 skipped (baseline before any edit, and again after).
• bun scripts/test.ts util rag vitest — 77 of 78 files passed, 1081 tests passed, 10 skipped, 1 failed. The one failure is src/test/rag/EndToEnd.integration.test.ts timing out at 160s: an ONNX-download RAG pipeline test (3.6 GB RSS, onnx:Qwen3-Embedding-0.6B) that references nothing this branch touches. Run on its own it passes on both sides — 8/8 in 73s on this branch, 8/8 in 77s on the base commit — so it is a resource/timeout flake under the sweep's 40-worker parallel load, not a regression.
• bun scripts/test.ts rag unit vitest — 21 files, 225 passed.
• bun run build:types — 41/41 workspaces, and TURBO_MAX_CORRECTED_BITS confirmed to reach @workglow/util/schema (declaration emitted, and importing it through the package subpath resolves to 4).
• bun run format — no changes to any file this branch touches (prettier --check clean on all four). Prettier did rewrite 12 unrelated files carrying pre-existing drift on the base branch; those were reverted rather than folded in.
• The three commit-5 rejection cases were confirmed red against the pre-fix task before the fix landed, and the pre-existing homogeneous cases should handle array of vectors and should handle array of vectors with turbo method were verified green by name after commit 6.

Run with Node v22.22.2 (the container has no Node 24 available); everything went through bun / bunx vitest as the repo scripts do.

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	60.07%	37630 / 62635
🔵	Statements	59.59%	39497 / 66276
🔵	Functions	60.98%	7255 / 11896
🔵	Branches	48.23%	19073 / 39538

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/ai/src/task/VectorQuantizeTask.ts	13.26%	0%	9.09%	16.04%	201-441, 450

Generated in workflow #3164 for commit d74a298 by the Vitest Coverage Report Action