docs(util,ai): correct TurboQuant's accuracy claims against an honest int8 baseline

[sroussey]

Blocks merge of #354. Docs, one rename, one unshipped signature change, one new output field, and tests — no numerics changed. #354's core numerics (Max 1960 loading factors, the clipping integral, the orthonormal self-inverse WHT, inverse rotation composition, the exact 1-bit angle correction, bit packing, edge cases, no caller mutation) were verified correct and are not touched.

The problem: an 8x win measured against a defective baseline

The branch claimed padded turbo was "substantially MORE accurate than linear", quoting a ~8x gap. Re-measured on the branch module with the test's own generator (makeRandom(d), a[i] = rnd() - 0.5, 40 pairs, int8, seed 42) — cosine RMSE against the exact similarity:

d	padded turbo	linear as shipped (L2 then ×127)	max-abs int8	max int8 code the shipped linear emits
768	0.00034	0.00269	0.00024	8
1536	0.00024	0.00331	0.00015	6
3072	0.00021	0.00263	0.00012	4

The middle column is not turbo's achievement. quantizeToInt8 divides by the vector's L2 norm before scaling by 127, so on a well-spread vector every coordinate sits near 1/√d and the largest code it emits is 8 of 127 at d=768 — three of its eight bits are gone before any comparison happens. Beating it measures that defect, not the rotation.

Against max|v|, padded turbo is 1.4×/1.6×/1.75× worse, not 8× better.

But "turbo is worse" is not the honest story either

Turbo's error is ~distribution-invariant (0.0001–0.0004 everywhere); max-abs swings ~20× with the input's tail, because its divisor is a single order statistic. On Gaussian vectors with 2 dimensions at 20× — "massive activations", which real embedding models exhibit:

d	padded turbo	max-abs int8
768	0.00039	0.00174
3072	0.00020	0.00138

So the corrected guidance is: prefer padded turbo when the corpus may contain outlier dimensions or is not under your control; max-abs int8 is simpler, keeps the input length, and is slightly more accurate when it does not. What the rotation buys is independence from the input distribution, not a lower error floor.

Scope decision: quantizeToInt8 is deliberately NOT fixed here

Follow-up filed as #796.

quantizeToInt8 is private (packages/ai/src/task/VectorQuantizeTask.ts:322, called only from :286) — not an exported function. It is a pre-existing defect on main, unrelated to TurboQuant. Changing its divisor from ‖v‖₂ to max|v| rescales every stored coordinate ~16× at d=768:

• cosine is safe — a positive scalar multiple per vector, and cosine is scale-invariant.
• l2 and ip are not. VectorDistanceMetric includes both, distances under them scale with magnitude, and there is no version marker on a stored int8 vector. A corpus written partly before and partly after is incoherent under those metrics and undetectable.

It needs a migration note and probably a linearScale: "l2" | "max-abs" input rather than a silent flip. Landing that inside an "add TurboQuant" PR buries a migration-relevant change where no reviewer is looking.

The hard requirement that follows: the corrected claims and tests are worded and computed so that repairing quantizeToInt8 later cannot invalidate them. The max-abs baseline is the quoted reference, and the tests compute it inside the test file rather than calling the task's linear path.

Changes

packages/util/src/vector/TurboQuantize.ts

• Padding paragraph replaced with the three-column table, an explanation of why the middle column is not a yardstick, the outlier-dimension figures, and the actual recommendation. Cropping paragraph keeps its figures but is re-anchored: cropped turbo measures 0.0162 at d=768, ~50× padded turbo and ~70× max-abs, recorded only to explain why the variant is not offered.
• Rejection message rewritten: names the padded length, warns the output is LONGER so a fixed-width column must be sized to it, gives both baseline comparisons and calls distribution-independence "the trade padding actually buys".
• optimalLoadingFactor's "within 0.35%" quoted the 16-level figure as if it were the worst. Measured deviation is 4.03% / 1.64% / 0.86% / 0.35% / 0.14% / 0.00% / 0.13% / 0.12% at 2…256 levels. Replaced with the curve plus why it does not matter — the solver is only ever called at 255 and 65535 levels, past the fine end, and the tabulated widths never reach it.
• nextPowerOf2 → turboPaddedLength. The guard is right; the name promised a general-purpose helper while the function throws TurboQuant dimensions must be at most 1048576. New on this branch (git grep nextPowerOf2 origin/main is empty), so renaming is free.
• DEFAULT_SEED exported with JSDoc.
• turboQuantizeToTypedArray's third parameter drops its default and its number | Options union → options: TurboQuantizeToTypedArrayOptions | undefined. { seed: 42 } alone was a compile error, since the interface uses T | undefined rather than T?. Breaking change to a function that has not shipped — free now, expensive later.

packages/ai/src/task/VectorQuantizeTask.ts

• method description and the throw both rewritten with the honest accuracy story.
• DEFAULT_SEED imported and used for the schema default and the destructuring default (was a hardcoded 42 in both).
• New required output originalDimensions, from vectors[0].length — the output length alone cannot distinguish a 768-dim model widened to 1024 from a model that genuinely emits 1024 dimensions.

Tests (packages/test/src/test/rag/VectorQuantizeTask.test.ts)

• The accuracy test scores against a max-abs int8 quantizer written in the test file, asserting turboRmse < 0.001 and turboRmse < maxAbsRmse * 2. Not linearRmse / 4, and not the task's linear path at any ratio: repairing the quantizer would have turned the old assertion red for exactly the right reason, which is the definition of a test pinned to the wrong thing. Turbo is legitimately the worse of the two on this generator (i.i.d. uniform — the best possible case for max-abs); the bound admits that and pins the magnitude.
• A second case asserts turboRmse < maxAbsRmse / 3 with Gaussian + 2 dims at 20×. Together these two mean the new wording cannot silently rot — one pins that turbo is not dramatically better in the benign case, the other that it is better in the heavy-tailed case.
• A recorded-defect guard asserts the shipped linear int8 path emits a max absolute code of 8 at d=768, commented as a recorded defect, not a desired property: when quantizeToInt8 is repaired the expectation changes to 127 and the comment goes away. This is what makes the deferred fix safe.

Verification

All figures in this PR were re-measured independently against the branch module before any edit and matched the review's numbers exactly, including the solver-deviation curve and the cropped-turbo ratios.

• bunx vitest run --project test packages/test/src/test/rag/VectorQuantizeTask.test.ts → 24 passed
• bunx vitest run --project test packages/test/src/test/util/TurboQuantize.test.ts → 72 passed
• bun scripts/test.ts util rag vitest → full section run, result in a follow-up comment
• bunx eslint over the four changed files → exit 0
• bunx prettier "packages/{util,ai,test}/src/**/*.{ts,tsx}" --check → All matched files use Prettier code style!

Note: this repo has no lint script — the root script is format (eslint --fix && prettier --check --write), so eslint and prettier were run directly in check mode.

[sroussey]

Full section verification

$ bun scripts/test.ts util rag vitest

Running all tests in sections [util+rag] — 78 file(s)

 RUN  v4.1.10 /tmp/wt-libs-tq

 Test Files  78 passed (78)
      Tests  1065 passed | 10 skipped (1075)
   Start at  09:06:47
   Duration  503.81s (transform 10.29s, setup 147.54s, import 5.13s, tests 334.74s, environment 6ms)

exit=0

Targeted runs on the two changed test files:

$ bunx vitest run --config vitest.config.ts --project test packages/test/src/test/rag/VectorQuantizeTask.test.ts
 Test Files  1 passed (1)
      Tests  24 passed (24)

$ bunx vitest run --config vitest.config.ts --project test packages/test/src/test/util/TurboQuantize.test.ts
 Test Files  1 passed (1)
      Tests  72 passed (72)

Lint:

$ bunx eslint packages/util/src/vector/TurboQuantize.ts packages/ai/src/task/VectorQuantizeTask.ts \
    packages/test/src/test/rag/VectorQuantizeTask.test.ts packages/test/src/test/util/TurboQuantize.test.ts
eslint exit=0

$ bunx prettier "packages/{util,ai,test}/src/**/*.{ts,tsx}" --check
Checking formatting...
All matched files use Prettier code style!
prettier exit=0

This repo has no lint script (the root script is format, which is eslint --fix && prettier --check --write), so both were run directly in check mode rather than fix mode.

New named tests in this PR:

✓ turbo method > should match a max-abs int8 baseline on similarity error at d=768 when padded
✓ turbo method > should beat a max-abs int8 baseline when the input carries outlier dimensions
✓ turbo method > records the shipped linear int8 path's largest emitted code at d=768
✓ turbo method > should report originalDimensions alongside a widened turbo vector

---

Generated by Claude Code