Review fixes for PR #354 (TurboQuant): NaN-scoring prepared query, exact shrinkage endpoint, engine-pinned loading factors, corrected docs

[sroussey]

Addresses the review findings on #354 in four commits. The review found no security issues (all 15 hostile decode shapes throw) and reproduced the maths independently — GAUSSIAN_LOADING_FACTORS matches Max (1960) at all eight widths, the shrinkage quadrature matches (2/π)asin(ρ) to 2e-6 over ρ∈[0, 0.97], and the rotation is genuinely orthogonal. What follows is everything else.

1 — A prepared query that cannot span its candidate scored NaN, not an error

turboPreparedCosineSimilarity re-checked the three compatibility scalars (bits, seed, dimensions) but nothing about the query's coordinates, while the dot product is bounded by the candidate's paddedDimensions. A query.values shorter than that read past its own end. Every such read is undefined → NaN, NaN survives finishCosine to become the score — and a ranking sort((a, b) => scores[b] - scores[a]) over NaN is implementation-defined, not an error. The failure surfaced as an arbitrarily ordered shortlist with nothing reported at any layer. A throw is strictly better than a plausible-looking wrong answer.

Three guards, placed before the codeNorm === 0 early return so a malformed zero query cannot slip past. The codeNorm check is one item beyond the review's literal ask — it is the same defect on the other field, exactly as assertQuantizeResultShape already argues for norm, and it cannot break a well-formed caller since turboPrepareQuery only ever emits a Math.sqrt result or 0.

The shrinkage map now reaches exactly 1 at ρ = 1

shrinkageAt cannot evaluate ρ = 1: the conditionalSd floor turns the inner conditional expectation into a step function landing on a bin boundary, so Simpson loses ~1.7% of the cross term at 1 bit while square stays exact. Measured, the tabulated last knot came out at 0.983377 / 0.989762 / 0.994138 / 0.996838 at 1/2/3/4 bits — against a comment asserting g(1) = 1. Every ratio above that knot had no bracket to interpolate inside and collapsed to exactly ±1, so at 2 bits every pair from |r| ≥ 0.9898 upward scored identically.

Returning 1 analytically (the pair is degenerate there — y = x, so E[Q(x)Q(y)] is E[Q(x)²]) improves the one exact reference this machinery has. Sweeping invertShrinkage(1, ·) against cos(π(1−r)/2) on the test's own ±200 grid:

	worst disagreement	at
before	2.776e-4	r = −0.985
after	1.158e-4	r = −0.96

and the table is still strictly increasing at all four widths.

No TURBO_QUANTIZE_VERSION bump. The shrinkage map is a scoring-time computation; the version covers only the codes → reconstructed-values mapping. Blast radius, measured: finishCosine uses the closed form at 1 bit, so no 1-bit score moves; at 2-4 bits only ratios above the previous last knot move — true cosines above ~0.9995. Self-similarity still returns exactly 1, and the bit-for-bit prepared-vs-pairwise test (which uses t = 0.999) is preserved by construction since both routes share finishCosine.

Deviation from the review's suggested assertion, stated plainly. It proposed expect(invertShrinkage(1, 0.99)).toBeCloseTo(cos(π·0.01/2), 4). That fails: the two sides now agree to 5.027e-5, just outside that matcher's 5e-5 window (the residual is the 65-knot linear interpolation, not the endpoint). Loosening to precision 3 (5e-4) would not have caught the pre-fix value either, since the old error was 1.234e-4. The pin is an absolute < 1e-4 instead — ~2x headroom over the real disagreement, and it still fails on the pre-fix behaviour.

2 — The IVectorStorage width contract was asserted only in prose

TurboQuantize.ts's module doc claims the typed-array encoder "drops into any backend declared at the padded width". Nothing ran it. Three things had no test anywhere: that turboQuantizeToTypedArray's output is a shape assertVectorShape accepts entry by entry (the packed codec deliberately has no storage path, so this is the only turbo encoder that does), that the length is turboPaddedLength(d) and not d end to end, and that a padded vector still retrieves its neighbour.

The round trip goes through putBulk (which is what runs validateVectorEntities) into an InMemoryVectorStorage declared at 1024 for a 768-dimensional embedding, then back out of similaritySearch. The companion case executes the footgun four separate error messages in this codebase warn about: a store declared at d rejects the widened vector on write, which is precisely why turboPadToPowerOf2 defaults to false.

Added to the existing rag test file rather than a new one — it already owns the task's turbo fixtures (turboOf) and timing hooks, so no new license header.

3 — Documentation that sent readers the wrong way

turboQuantizedInnerProduct said "for maximum accuracy, dequantize both sides and take a real dot product". That is the uncorrected estimator below TURBO_MAX_CORRECTED_BITS and reads systematically low with nothing reported. Measured at d=1024 over 25 Gaussian pairs at true cosine 0.80, against a true mean inner product of 816.8:

bits	this helper	dequantize-then-dot
1	823.8	611.2 (−25%)
2	820.2	739.7 (−9%)
3	817.9	791.2 (−3%)
4	817.2	808.7 (−1%)
5	814.1	814.1
8	816.7	816.7

5-8 bits agree exactly, which is the boundary the constant names.

The module doc's bias table is captioned "BEFORE the correction and as shipped", but the 1-bit row's "before" column is already Goemans-Williamson-corrected — that correction landed in an earlier commit than the 2-4 bit one. Read as-is the table says 1 bit has no shrinkage to correct, while the code says the raw statistic reads a true 0.80 as 0.59. Caption fixed, row marked, and a note gives the genuinely uncorrected 1-bit bias from the closed form: −0.167 at a true 0.50, −0.210 at 0.80, −0.152 at 0.95 — roughly 200x the largest number in the table. The row's measured numbers are left alone; they are honest measurements of the shipped estimator.

VectorQuantizeTask's method schema description is the one UI-visible surface and repeated the "TurboQuant" name with no disclaimer. It now carries the same one the module doc does.

The rename is deliberately deferred

rotatedScalarQuantize would touch 16 exported symbols re-exported wholesale by schema-entry.ts, but the blocking reason is worse than breadth: QuantizationMethod.TURBO's "turbo" is a persisted enum value in serialized workflow JSON and in VectorQuantizeTaskOutput.method, and turboSeed / turboPadToPowerOf2 are named input ports. Renaming breaks dataflow edges in saved graphs and invalidates cached task outputs (cacheable = true). Deprecated aliases cover function names but not port names or the persisted enum without a migration. Filed as a follow-up next to #798; only the schema disclaimer is done here.

4 — Engine-dependent loading factors, and a boxed code array

optimalLoadingFactor solved the two level counts the signed typed-array path reaches (255 for int8, 65535 for int16 — never powers of two, so absent from the table) by ternary search over quantizerDistortion, which calls Math.exp ~1200 times per evaluation. ECMAScript leaves Math.exp implementation-approximated, and this repo runs the same tests under JSC (bun test) and V8 (vitest on Node). Measured here:

levels	Node v22.22.2 (V8)	Bun 1.3.11 (JSC)
255	3.9206374677710735810	3.9206374677728756950
65535	5.9382386136258178411	5.9382386136258178411

A relative 4.6e-13 at 255 — small enough to produce the same bytes today, not small enough to guarantee it, since a code is Math.round((clamped / scale) * max) and a value near a half-integer can flip either way. The golden-bytes test asserts an exact sequence derived from this constant, so it was a weaker pin than it read. Both values are now literals in SOLVED_LOADING_FACTORS, consulted via Object.hasOwn (a bare index resolves inherited Object.prototype keys); the ternary search stays as the fallback. Tabulating the exact doubles is preferred to rounding to N significant digits, which would perturb scale and could silently change the checked-in bytes.

The golden-bytes test gains an INT16 block over the same fixture — the existing int8 block pins the 255 literal, and nothing pinned 65535 at all. Expected values were captured from the failure diff, not hand-computed, and the test was run under both runners producing identical bytes (below).

The encoder's boxed number[] becomes a Uint8Array — safe by construction (assertBits caps bits at 8 so every code is in [0, 255]; packCodes reads only .length and codes[i], so its signature widens to ArrayLike<number>). Re-measured peak RSS growth around a single turboQuantize at d = 2²⁰:

	peak RSS growth	bytes / padded coordinate
before	~31 MB	~31
after	~25 MB	~25

That is a ~20% move, so MAX_TURBO_DIMENSIONS's docstring figure is re-measured rather than left stale — leaving it would have been a smaller version of exactly the doc-vs-code defect this PR fixes.

Verification

bun install && bun run use-source
bun run format                            # All matched files use Prettier code style!

bun scripts/test.ts unit rag  vitest      # Test Files 21 passed (21) | Tests 227 passed (227)
bun scripts/test.ts unit util vitest      # Tests 1 failed | 841 passed | 10 skipped (852)
bun scripts/test.ts unit util bun         # 786 pass | 10 skip | 8 fail
bun scripts/test.ts unit storage vitest   # Test Files 3 failed | 39 passed | 1 skipped (43)

Every named test passes (117/117 across the two touched files):

✓ turboQuantize > should match hardcoded golden bytes (cross-process determinism)
✓ turboQuantizedCosineSimilarity > the shrinkage map reaches exactly 1 at rho = 1, so only |ratio| = 1 saturates
✓ turboQuantizedCosineSimilarity > invertShrinkage reproduces the 1-bit Goemans-Williamson closed form
✓ turboPrepareQuery / … > rejects a prepared query whose values cannot span the candidate
✓ turboPrepareQuery / … > matches turboQuantizedCosineSimilarity bit-for-bit at every bit width
✓ ranking fidelity > the shrinkage correction is strictly monotone at every corrected width
✓ VectorQuantizeTask > turbo method > IVectorStorage round trip > a padded turbo vector round-trips through a store declared at turboPaddedLength(d)
✓ VectorQuantizeTask > turbo method > IVectorStorage round trip > a store declared at d rejects the widened turbo vector on write

The golden bytes are identical under both runners — the whole point of the loading-factor pin:

$ npx vitest run … -t 'golden bytes'   →  Tests  1 passed | 84 skipped (85)
$ bun test         … -t 'golden bytes' →  1 pass, 0 fail, 3 expect() calls

The three suites that must NOT move did not move — the RMSE ceiling, the per-bit-width signed bias bands, and ranking fidelity all pass unchanged (they top out at true 0.95/0.88, below the ~0.9995 where the ρ=1 fix takes effect).

The failures above are all pre-existing and environmental, each verified

This box was running three other sessions' full test suites concurrently, and every failure is a timeout or a runner-API gap:

• unit util vitest — 1 failed: TestCredentialPreload.test.ts timed out at 15 s (the file took 162 s under load). Passes 5/5 in isolation. It does not import TurboQuantize.
• unit util bun — 8 failed: all in WorkerServerBase.race, Base64 and dataUri, all TypeError: vi.stubGlobal / vi.unstubAllGlobals / vi.advanceTimersByTimeAsync is not a function — vitest-only APIs Bun's runner does not implement. None of those files were touched by this PR.
• unit storage vitest — 3 files failed: all PGlite Postgres tests hitting Test timed out in 15000ms; the count varied between two runs of the same command (4 vs 5 tests). Verified by reverting this PR's four files to the base commit (they passed), then restoring them and re-running twice (passed both times, 6/6). This PR touches no storage code.

Risks

• The ρ=1 fix changes returned numbers at 2-4 bits for true cosines above ~0.9995. No 1-bit score moves, and self-similarity is still exactly 1.
• The golden bytes fail loudly if a literal is mistyped. Work order was: add literals → run the golden test under both runners → then add the int16 pin.
• The codeNorm guard is the one item beyond the review's literal ask (rationale above; it is the first thing to cut if scope is challenged).

Out of scope

The rename; a storage path for the packed codec (#798); VectorQuantizeTask.quantizeToInt8's L2-then-×127 scaling; re-deriving GAUSSIAN_LOADING_FACTORS / SHRINKAGE_TABLE_POINTS / SHRINKAGE_NODES_PER_UNIT; any version bump.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01HJRf3YFa8DjmjsZvXz8xDT

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	60.57%	39200 / 64714
🔵	Statements	60.06%	41157 / 68516
🔵	Functions	61.06%	7559 / 12379
🔵	Branches	48.83%	19994 / 40945

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/ai/src/task/VectorQuantizeTask.ts	13.26%	0%	9.09%	16.04%	201-441, 450

Generated in workflow #3204 for commit 3a41c76 by the Vitest Coverage Report Action