opencode-compaction-probe

A/B benchmarking harness for opencode's split-turn compaction context: does the "Compacted Turn Context" section, a second summary covering the split-turn prefix, retain the user's request measurably better than the pre-fix artifact (main summary only)?

Bottom line: redundant for a strong summarizer, measurably valuable for a weak one.

• deepseek-v4-flash (strong, 1M context), heads up to 300K: no measurable difference; the main summary keeps the request verbatim.
• Gemma 4 31B (weak, 262K context), heads at 80K and up: new over old improves recall-judge 0.67 to 0.96, artifacts 0.89 to 1.00, continuation 0.93 to 1.00.
• Scenario extensions (--scenario): the value is bottom-cut-specific. long-turn (agentic-scale turn) reproduces the baseline; mid-cut (cut mid-conversation) fires the section everywhere but it reframes attention without restoring facts the main summary lost, so the main summary is the load-bearing retention mechanism there.

Full methodology and tables: docs/writeup.md.

Requirements

• Bun 1.3+
• The opencode source as a submodule (opencode/), checked out on the split-turn-context branch. The packages are unpublished, so the harness runs against the submodule's code and dependencies, linked via tools/link.mjs. On the dev branch the mechanism is absent and runs degrade to old-vs-old.
• A configured LLM provider for the summarizer and for probes/judging.

Setup

bun run setup  # submodule init + opencode install + link

OPENCODE_DIR (default: the opencode/ submodule) can point at any other checkout. Re-run the link if the submodule changes: bun run setup again, or bun install (runs postinstall, which links without re-installing the submodule).

Usage

bun run bench                                            # 3 trajectories * 3 cycles * 2 variants
bun run bench --trajectory retry-logic --cycles 1        # cheap smoke run
bun run bench --model provider/model                     # summarizer under test
bun run bench --probe-model provider/model               # separate, stronger judge
bun run bench --head-tokens 30000,80000,150000 --cycles 1
bun run bench --scenario long-turn,mid-cut --cycles 1    # long-turn / mid-cut fixtures
bun run bench --sensitivity                              # probe controls
bun run bench --dry-run --head-tokens 100000             # estimates only, no LLM calls
bun run bench --jsonl out.jsonl --verbose
bun run bench --resolve-only

Typical weak-model run:

bun run bench --model openrouter/google/gemma-4-31b-it \
  --probe-model deepseek/deepseek-v4-flash --head-tokens 80000 --cycles 3

• --model provider/model: compaction/session model, the summarizer under test.
• --probe-model provider/model: model for every probe and judge completion. Defaults to --model; a weak summarizer needs a strong judge.
• --cycles N: compact-advance-recompact cycles per variant, default 3.
• --head-tokens N[,M,...]: synthetic head sizes. Deterministic filler turns, disjoint from every planted fact.
• --scenario split-turn|long-turn|mid-cut|all: which fixture to run, default split-turn (the baseline bottom cut).
  • long-turn: same bottom cut, but the last turn is a realistic agentic tool loop (--turn-calls N calls, default 60). The split-turn prefix summary has to absorb a large prefix.
  • mid-cut: the cut lands mid-conversation, on an older fact-bearing turn, while recent turns stay whole (--recent-turns N, default 5). Adds original-task probes (orig-recall, orig-recall-judge) for facts planted in turn 1, which only the main summary can carry.
• --turn-calls N: tool calls in the inflated split turn, default 60.
• --recent-turns N: recent turns kept whole in mid-cut, default 5.
• --trajectory name: run only that trajectory.
• --sensitivity: probe two controls, the raw retained tail and the artifact with facts redacted, to prove the probes detect absence.
• --dry-run: build fixtures and print estimates, no LLM calls.
• --jsonl path, --verbose, --resolve-only: machine output, artifact dumps, model check.

How it works

• Fixtures: three deterministic trajectories (retry-logic, yaml-migration, session-store-fix) with planted facts, a buried constraint, file paths, and a continuation anchor. The last turn is too big to keep whole, so compaction splits it mid-turn.
• Scenarios: split-turn (baseline bottom cut), long-turn (bottom cut on one inflated agentic turn), mid-cut (cut lands mid-conversation; original task in turn 1, recent turns kept whole with a larger preserve budget).
• Variants: new keeps the full artifact (main summary plus the turn-context section); old empties that section after each compaction, matching pre-fix behavior exactly.
• Probes: yes/no membership (request, constraint, artifacts, negative distractor), free-form recall scored by fragment containment plus a fidelity judge (3-sample median), and continuation judged 0-5. Mid-cut adds original-task recall probes (orig-recall, orig-recall-judge) for facts only the main summary can carry, and every new-variant run records whether the turn-context section was actually produced (tc).
• Drift: repeated compact-advance-recompact cycles measure retention decay across a compaction chain.

Layout

script/compaction-probe.ts  # the harness
tools/link.mjs              # links the opencode checkout
docs/writeup.md             # methodology and full results