fix(util,ai): restrict TurboQuant to signed targets; harden input validation

[sroussey]

Review fixes stacking on #354 (claude/integrate-arxiv-paper-VF55c) — not on main. No PR template exists in this repo (.github/, root, docs/), so this uses plain headings.

CRITICAL: unsigned turbo quantization encodes a DC offset that breaks cosine similarity

turboQuantizeToTypedArray's unsigned branch mapped

x -> (x + scale) / (2 * scale) * max

an affine map. Its DC offset (127.5 for uint8) lands on every stored coordinate. Cosine similarity is invariant to scaling but not to translation, so that shared component dominates every comparison.

Measured on this branch (d=1024, uint8, seed 42, mulberry32 vectors), replicating the removed branch exactly:

quantity	value
true cosine of the pair	0.0139
cosine after uint8 turbo	0.9018
range of uint8 cosine over 40 random pairs	[0.8930, 0.9066]

The plan's independent measurement on a different random stream reported the same failure shape (true 0.0559 → 0.9071, range collapsing to ~[0.90, 1.00]). Negative similarities become impossible and absolute thresholds are meaningless — silently, with no error anywhere.

Decision: reject unsigned targets outright. uint8 / uint16 now throw; only int8 and int16 are supported.

The alternatives do not work:

• Storing a per-vector offset — cosineSimilarity(a, b) (packages/util/src/vector/VectorSimilarityUtils.ts) receives only the two arrays. There is nowhere to thread an offset.
• A "subtract the midpoint first" contract — cannot be honoured by pgvector / SQLite / DuckDB backends, which compute the distance server-side.

Nothing has landed on main, so there is no migration burden, and unsigned buys zero storage over the signed type of the same width.

Each fix

#	Fix	File
T1	INTEGER_TARGET_RANGES → SIGNED_TARGET_RANGES (int8/int16 only). Guard is now Object.hasOwn(...) before indexing — INTEGER_TARGET_RANGES["constructor"] previously resolved an inherited Object.prototype key, so if (!range) saw a truthy function and turboQuantizeToTypedArray(v, "constructor") returned an all-zero Uint16Array instead of throwing. Unsigned branch and signed flag deleted; JSDoc rewritten (signed-only, comparability caveats, the removed "works transparently with … similarity search" claim qualified).	TurboQuantize.ts
T2	Extracted normalizeToUnit(), shared by turboQuantize and turboQuantizeToTypedArray, which throws on a non-finite norm. Previously new Float32Array([1,2,NaN,4,5,6,7,8]) made norm NaN, failed norm > 0, and returned the freshly-allocated all-zero buffer — the NaN silently discarded. Zero-vector behaviour is preserved.	TurboQuantize.ts
T3	nextPowerOf2 used p <<= 1, a 32-bit signed shift: at p = 2^30 it wraps to -2147483648, then 0, and p < n stays true forever. Changed to p *= 2, plus MAX_TURBO_DIMENSIONS = 2**24 (the padded Float64Array working buffer is 128 MB there) and assertDimensions / assertBits, now called from both exported helpers. turboQuantizeStorageBytes(1.5e9, 4) hung the event loop permanently; bits of 0, -5, NaN silently returned 1.	TurboQuantize.ts
T4	fastWalshHadamard assumed a power-of-2 length; at n=6 it read data[j + halfSize] past the end (undefined → NaN), so a TurboQuantizeResult with paddedDimensions: 6 decoded to [NaN x 6] with no error. Since that interface is plain and serializable (intended for storage), a persisted or mismatched record corrupted silently. The transform now enforces the invariant, and turboDequantize / turboQuantizedInnerProduct re-validate bits, dimensions, seed, norm and paddedDimensions === nextPowerOf2(dimensions) before use.	TurboQuantize.ts
T5	Documented, not "fixed". randomRotate returns paddedLen coordinates (correctly — that is what makes it invertible) but only the first d are kept, so for a non-power-of-2 d this is a random projection, not an orthogonal rotation. Fixed-length output makes that unavoidable, and 768/1536 are the two most common embedding dims among this repo's providers, so throwing is not acceptable. Measured cosine RMSE (int8, seed 42, 40 pairs) is now in the JSDoc: d=1024 → 0.001, d=1000 → 0.006, d=1536 → 0.013, d=768 → 0.019 (worst single pair 0.052), with a recommendation to zero-pad at the call site. Mirrored into the method schema description.	TurboQuantize.ts, VectorQuantizeTask.ts
T6	Extracted getSignTable(seed, paddedLen) returning three Uint8Array flip masks drawn in exactly the previous order, memoized in a bounded Map (16 entries, oldest evicted) so a hostile seed stream cannot grow it. inverseRandomRotate previously rebuilt a boxed boolean[][] of 3 × paddedLen on every dequantize. unpackCodes now returns a Uint8Array (bits ≤ 8) instead of a boxed number[] — turboQuantizedInnerProduct, documented as "faster than dequantizing", was allocating two 1024-element JS arrays per comparison. Output bytes are unchanged (pinned by the new golden test).	TurboQuantize.ts
T7	VectorQuantizeTaskOutput carried only vector/originalType/targetType, so nothing downstream could tell a rotated Int8Array from a linear-quantized one — measured, cosineSimilarity(a, turboQuantizeToTypedArray(a, INT8, 42)) ≈ 0.05, a vector near-orthogonal to itself. A collection re-indexed with turboSeed 42→43, or a row written with method: "linear", produced garbage rankings with no error. method and turboSeed are now in outputSchema (with method required) and on the output type. The turbo branch rejects non-signed targetType early, and the normalize description notes it is ignored for turbo.	VectorQuantizeTask.ts
T8	License year 2026 on the two files created 2026-04-01 (TurboQuantize.ts, TurboQuantize.test.ts); the pre-existing files stay at 2025 per CLAUDE.md. TurboQuantizeOptions fields → readonly bits: number | undefined / readonly seed: number | undefined. createPrng: state ^= state >> 17 → >>> 17 (cosmetic — the JSDoc calls it xorshift32 — but it changes output bytes, so it was done before the golden literals were generated).	TurboQuantize.ts, tests

Tests

10 new tests plus an extended zero-vector case. All fidelity/statistical tests use a local deterministic mulberry32 PRNG — never Math.random.

TurboQuantize.test.ts: unsigned rejection; DC-offset guard (d=1024 int8 — asserts |mean| < 2 and ≥30% negative coordinates, both of which an affine map fails); NaN/Infinity on both entry points; out-of-range dimensions and bits (incl. 2**30 + 1 under a 2 s timeout so a hang fails rather than stalls CI); tampered paddedDimensions on turboDequantize and turboQuantizedInnerProduct; prototype-chain target names; cosine fidelity across d ∈ {768, 1000, 1024, 1536}; and hardcoded golden bytes ([117, 165, 180, 133] and a 16-value int8 prefix) — the existing determinism test only compared two in-process calls, which cannot detect a Math.random regression introduced elsewhere.

VectorQuantizeTask.test.ts: unsigned targetType rejection for turbo; method/turboSeed reported on the output.

What was verified vs. not

Verified — commands run, output seen:

• bun install → 1830 packages installed.
• bun run build:types → 41/41 tasks successful (full workspace, tsgo).
• bunx vitest run packages/test/src/test/util/TurboQuantize.test.ts packages/test/src/test/rag/VectorQuantizeTask.test.ts → 67 passed / 67.
• Fails-before check. With the pre-change sources restored (git show HEAD:…) and the new tests in place: 9 of the new tests failed — NaN/Infinity, golden bytes, tampered paddedDimensions, float-target message, unsigned rejection, prototype-chain names, zero-vector (uint8 arm), and both VectorQuantizeTask tests.
• T3 fails-before, separately. The out-of-range test could not be run under the old code at all: turboQuantizeStorageBytes(2**30 + 1, 4) spins synchronously, so it blocks the event loop and vitest's own timeout never fires — a first attempt burned a full 10-minute shell timeout. Isolated with timeout 15 bun -e … it exits 124 (hang) on the old source and throws immediately on the new one. That is the DoS, demonstrated.
• Both CRITICAL measurement tables above were produced by me on this branch, not copied from the plan.
• prettier --write and eslint clean on all four changed files (pre-commit lint-staged also ran).

Not verified / caveats:

• Two new tests (DC-offset and cross-dimension cosine fidelity) also pass against the pre-change sources. Both use INT8, whose signed map was already correct — they are forward regression guards (against reintroducing an affine map, and against the documented RMSE drifting), not reproductions of a current failure. Flagging this because the plan asked that every new test fail before.
• The documented RMSE figures are the plan's, kept because my independent measurement came in at or below every one of them (0.0164 / 0.0047 / 0.0008 / 0.0126 vs. documented 0.019 / 0.006 / 0.001 / 0.013), so the test's ×1.5 bound holds with margin. The one adjustment: worst single pair documented as 0.052 rather than the plan's 0.048, because I measured 0.0511 at d=768.
• Test execution requires bun run use-source (this repo's use-source rewrites package.json exports rather than writing dist stubs as CLAUDE.md describes). Those package.json edits were reverted before committing — the commit touches exactly 4 files. Under dist resolution the suite cannot run without a prior build.
• Only these two test files were run. The wider suite was not executed; build:types across all 41 packages is the only whole-workspace signal here.

Deviation from the approved plan

One, stated explicitly. The plan asked to drop ? from method / turboSeed on VectorQuantizeTaskInput (making them readonly method: QuantizationMethod | undefined). I added readonly but kept the ?, because the task's inputSchema lists neither in required — they genuinely are optional inputs — and removing ? makes every existing vectorQuantize({ vector, targetType }) call site a type error, including the pre-existing tests in VectorQuantizeTask.test.ts. It would also have been inconsistent with the pre-existing normalize?, which the plan explicitly left alone. TurboQuantizeOptions did get the full treatment (readonly bits: number | undefined) as specified — it has no external callers, so the change is contained. On the output type, method is required and turboSeed is number | undefined (set to undefined for linear), exactly as planned.

---

🤖 Generated with Claude Code

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	64.55%	31999 / 49568
🔵	Statements	64.37%	33121 / 51453
🔵	Functions	65.58%	6007 / 9159
🔵	Branches	53.51%	16464 / 30768

File Coverage

No changed files found.

Generated in workflow #2905 for commit 43458f0 by the Vitest Coverage Report Action