feat(invariants): verified-fact elision markers for JSON and log compression - #82
Merged
Conversation
…ression
Silent elision forces readers to guess what was dropped, and agent
benchmarks show the guess is usually 'retrieve everything' — bare
'[N collapsed]' markers caused 11-97 recovery-call storms.
CompressJSON now replaces elided array items with a trailing sentinel
record ({"_tok_elided":{count,facts}}) keeping the array valid JSON,
and CompressLog's collapsed marker carries the level distribution of
the run. Facts state only what was VERIFIED across every elided unit:
- field constants: status=charged x15
- exact enumerations whose counts sum to the elided total:
state: pending x1 processing x2 shipped x3
- numeric ranges from original value strings: amount=5..199.99
(numerically ordered, integral floats without decimals)
- distinct-count coverage: order_id: 25 distinct, ord-1000..ord-1024
- dense-run upgrade when prefix+fixed-width+density all hold:
wh-5000..5008 all 9 present
Withhold rules are correctness, not tuning: credential-shaped field
names, values with spaces or >24 bytes, >5 buckets, units yielding no
fields -> that fact is withheld entirely, never shortened; runs under
3 units are not summarized at all; summaries cap at 160 bytes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Silent elision forces readers to guess what was dropped — and agent benchmarks (documented in the caveman project) show bare
[N collapsed]markers cause 11–97 recovery-call storms. This PR makes every elision state only verified facts about what was removed.Changes
invariants.go: computes facts that hold across every elided unit:status=charged ×15)state: pending×1 processing×2 shipped×3)amount=5..199.99)order_id: 25 distinct, ord-1000..ord-1024)wh-5000..5008 all 9 present) when prefix+fixed-width+density holdCompressJSON: elided items now replaced by a trailing sentinel{"_tok_elided":{count,facts}}— output stays a valid JSON array; silent drops eliminated.CompressLog: collapsed-run marker now carries the level distribution (info×3 debug×2).Verification
Full test suite green (
go test ./...), vet clean.