fix(ci): run the fuzz corpus replay outside the coverage lane

[thymikee]

Summary

The Coverage job intermittently ends with no failing test and one file's results missing:

Test Files  1070 passed (1071)
Tests       7732 passed | 3 skipped
Errors      1 error
Error: [vitest-pool]: Worker forks emitted error.
Caused by: Error: Worker exited unexpectedly

This is shape (B) of #1824 — the half #1854 did not fix, and which that issue said to fix at the source if it recurred. It has.

scripts/fuzz/corpus-replay.test.ts moves into its own serialized project that the coverage run skips (--project=!fuzz-worker); a second, uninstrumented Vitest invocation in test:coverage:ci runs it, so the tests still execute on every PR.

The vanished file is the same one every time

Scanning every failed Coverage job across the 120 CI runs since #1854 merged (2026-08-21 → 08-24): the signature appears 5 times, and the vanished file is scripts/fuzz/corpus-replay.test.ts in all 5 — 23% of Coverage failures in that window, ~4% of all CI runs. With #1866's occurrence that is six for six.

run	job	vanished
32514899452	96874232543	corpus-replay.test.ts
32589357049	97070586970	corpus-replay.test.ts
32621245948	97149619907	corpus-replay.test.ts
32630591037	97172683670	corpus-replay.test.ts
32620910025	97148819646	corpus-replay.test.ts
(plus #1866's occurrence)		corpus-replay.test.ts

Identified by subtraction: the parenthesised count is what was collected, the first number is what reported. (gh run view --log emits ANSI as literal caret notation, so strip with (?:\x1b\[|\^\[\[)[0-9;]*m.) A run re-run to green reports conclusion: success, so this needs a per-attempt sweep — scanning --status failure alone misses most occurrences.

Every one has zero failing tests, exactly one file's results missing, and scripts/fuzz/harness.test.ts as the last file reported — i.e. corpus-replay was alone in flight.

What the evidence rules out

The ~40s gap before the error is coverage report generation, not test time. The pool surfaces its AggregateError only once every task settles, which on this job is after the report is written. Control, from a green attempt of run 32630591037:

09:37:31.5  ✓ subprocess-stub  scripts/fuzz/harness.test.ts     (11 tests) 11404ms
09:37:35.9  ✓ subprocess-stub  scripts/fuzz/corpus-replay.test.ts (11 tests) 3152ms
09:38:12.5    Test Files  1071 passed (1071)      ← 36.6s of report generation

So the file takes 3.1s in CI, not 40. That rules out:

• CPU contention — in the failing runs nothing else was executing; subprocess-stub is serialized and unit-core had drained ~40s earlier.
• A missed per-case budget producing a false hang and a terminate() — there is no 5s overrun in a 3s file, and a hang would have failed the test loudly rather than vanishing it.
• STARTUP_BUDGET_MS being too low — a 60s budget is never approached.

Partial test counts (3/11 and 9/11 reported) place the death mid-file, inside runCases.

Membership is by demonstrated failure, not by a property

An earlier revision of this PR claimed the criterion was "constructs a node:worker_threads Worker inside the Vitest worker". That rule is false, as review caught: src/daemon/handlers/__tests__/session-replay-runtime-maestro.test.ts constructs one and stays in unit-core, instrumented and green. It is the only other test file in the repo that does, and it has never died.

Since the proximate cause was never reproduced, the honest criterion is the observed record: a file is admitted here only when it has actually vanished from the instrumented Coverage lane, with run URLs. A theory about workers is not enough. The project and constant are named fuzz-worker / FUZZ_WORKER_TESTS so the name does not re-assert the false rule.

Measured: this costs zero coverage. Two full local runs, baseline vs. variant, differ by 0 lines — the cases execute in worker threads, a separate isolate the fork's inspector session never instruments, so the file never contributed measured coverage in the first place.

scripts/gate/model.ts learns Vitest's negated --project semantics, so unit-ci still provably owns every project instead of being credited with one it skips.

Validation

• pnpm gate unit-ci green end-to-end: 1094 passed (1094), then fuzz-worker 1 passed (1).
• typecheck, lint, format:check, check:fallow --base origin/main, check:layering, check:gate-manifest, check:agent-guidance, check:tmpdir-leaks, mutation:test, and the scripts/gate + scripts/check-affected node:test suites all pass.
• Coverage delta measured across two full instrumented runs (baseline 43,057 covered lines → variant 43,061; the only differing file moved up, unrelated).

Honest limits

• The proximate Node-level cause was not reproduced. 80 terminate-of-wedged-worker cycles under an active precise-coverage inspector session (Node 24.13, macOS/arm64) did not crash, nor did repeated instrumented runs under heavy artificial contention. This targets the factor that correlates with red/green across six occurrences, not a mechanism demonstrated in a debugger.
• Kill criterion, in the style Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824 used: if the signature reappears on a different file, this containment is aimed at the wrong thing and the next lever is a fork heap ceiling (poolOptions.forks.execArgv), which would at least turn a silent SIGKILL into a named JavaScript heap out of memory — emitUnexpectedExit discards the exit code and signal, so today CI cannot tell us which it was.
• docs/agents/testing.md is at 9,985 of its 10,000-byte ceiling on main, so a pointer to the new project does not fit. Per that gate's own rule, the rationale lives with its owner in vitest.config.ts.

Refs #1824

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-24 15:03 UTC

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.40 MB	2.40 MB	0 B
JS gzip	805.3 kB	805.3 kB	0 B
npm tarball	929.6 kB	929.6 kB	+54 B
npm unpacked	3.22 MB	3.22 MB	+236 B

npm unpacked components

Component	Base	Current	Diff
JS / dist source	2.55 MB	2.55 MB	0 B
Apple runner source/project	570.6 kB	570.6 kB	0 B
macOS helper source	54.5 kB	54.5 kB	0 B
Android helper artifacts	0 B	0 B	0 B
Other package files	44.5 kB	44.7 kB	+236 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	21.8 ms	22.0 ms	+0.1 ms
CLI --help	62.5 ms	62.4 ms	-0.1 ms

Top changed chunks: no changes in the largest emitted chunks.

Top changed packed files

Packed file	Base	Current	Diff
package.json	18.2 kB	18.5 kB	+236 B

[thymikee]

Request changes at 0ac0fcc. The coverage-lane split is a reasonable, narrowly scoped containment for the repeatedly vanishing fuzz corpus file, and the second uninstrumented invocation plus gate-model tests preserve execution ownership. Two corrections remain: (1) narrow the NESTED_WORKER_TESTS membership comment to the observed fuzz-worker/coverage condition—src/daemon/handlers/__tests__/session-replay-runtime-maestro.test.ts also imports node:worker_threads and constructs a Worker while correctly remaining in unit-core, so the current universal rule is false. (2) openSession() adds real failed-start cleanup by terminating a worker when the ready handshake rejects, but no focused test or planted-red evidence exercises that path. Add a regression proving the live worker terminates on startup rejection, or keep this PR strictly to the CI lane split and remove the unrelated cleanup. Current CI is still in progress.

[thymikee]

Both corrections applied at 8a496ea (rebased onto 8d280a2).

(1) The membership rule was false — confirmed and removed. src/daemon/handlers/__tests__/session-replay-runtime-maestro.test.ts:469 constructs a new Worker(...) and stays in unit-core, instrumented and green; it is in fact the only other test file in the repo that constructs one, so "nests a Worker" was not just imprecise, it was refuted by the single available counterexample.

Since the proximate cause was never reproduced, I replaced the mechanism-shaped rule with the observed one: a file is admitted only on a record of actually vanishing from the instrumented Coverage lane, with run URLs — a theory about workers is not enough. I also renamed the project and constant nested-worker/NESTED_WORKER_TESTS → fuzz-worker/FUZZ_WORKER_TESTS, since the old name re-asserted the false rule in the place people read first.

(2) openSession() removed — the PR is now strictly the lane split. scripts/fuzz/execute.ts is byte-identical to main.

I took the removal rather than the test because the genuinely leaky case is not reachable without a seam: the path that strands a live thread is the startup-budget miss, and STARTUP_BUDGET_MS is a hard-coded 60s constant. The reachable rejection — an unknown target name, which makes worker.ts throw at module top level — proves nothing, because that thread is already dying, so terminate() is a no-op and the test would pass with or without the fix. Planting it red needs either a 60s wait or a test-only budget seam, and neither belongs here. Happy to file it as a follow-up with that constraint written down if you want the finding kept.

Diff is now 6 files: vitest.config.ts, vitest.mutation.config.ts, package.json, scripts/gate/model.ts (+ its test), docs/agents/pull-requests.md.

Re-validated on the new base: pnpm gate unit-ci green end-to-end (1094 passed (1094), then fuzz-worker 1 passed (1)), plus typecheck, lint, format, fallow vs origin/main, layering, gate-manifest, agent-guidance, mutation:test, and the scripts/gate + scripts/check-affected suites.

One thing worth flagging from the sweep, since it changes how this class gets triaged: a run that is re-run to green reports conclusion: success, so scanning --status failure misses most occurrences. The five runs above only surface by walking each run's attempts and filtering name=="Coverage" and conclusion=="failure".

[thymikee]

This was generated by AI during triage.

Cross-PR review finding against #1996 (currently open, all-green): once both land, the coverage lane breaks.

unit-ci runs test:coverage:ci, which this PR turns into two chained invocations:

vitest run --coverage --project=!fuzz-worker && vitest run --project fuzz-worker

#1996 sets AGENT_DEVICE_COVERAGE_SHARD as a job-level env on the Coverage matrix jobs and wires blob output through it (vitest.config.ts: outputFile.blob = .vitest-reports/blob-<shard>.json). That env applies to both invocations of the chained script, so on each shard runner:

1. The instrumented run writes .vitest-reports/blob-1.json.
2. The uninstrumented fuzz-worker run then overwrites the same file (same shard id → same blob name), so the artifact uploaded by Upload coverage blob contains only fuzz-worker results. The Coverage Report job merges those and evaluates thresholds over a report missing the entire instrumented suite.

There is a second wrinkle in the other direction: #1996's Coverage Report job runs gate: unit-ci with AGENT_DEVICE_COVERAGE_MERGE=1, which would now execute the second invocation in merge mode too — vitest run --project fuzz-worker with mergeReports set re-merges the blobs instead of running the project.

Suggested reconciliation (either side can own it): make the shard env apply only to the instrumented leg (e.g. distinct blob names per leg, or move the fuzz-worker invocation out of the shared script into the workflow), and skip the second leg when AGENT_DEVICE_COVERAGE_MERGE=1.

Also a heads-up: this PR and #1997 both edit the readiness bullet in docs/agents/pull-requests.md; they textually conflict (verified via git merge-tree), so whoever merges second needs a one-line rebase.

[thymikee]

Verified the #1996 finding by merging the two branches locally and running the sharded lane. It is real, and the failure is harder than described — the shared leg doesn't silently produce a wrong report, it takes the shard down:

Error: --shard <count> must be a smaller than count of test files. Resolved 1 test files for --shard=1/2.
 Test Files  no tests
     Errors  1 error

AGENT_DEVICE_COVERAGE_SHARD=1/2 pnpm gate unit-ci on the merged tree exits 1, and blob-1.json is left at 1.4 kB containing {"fuzz-worker": …} plus that error — the instrumented shard's report overwritten on the way out, exactly as predicted. Both shards would fail this way, so the Coverage Report job never gets a usable blob.

Fixed at 1dd134f (rebased onto 8ceff28). The second leg now goes through its own script that blanks both switches:

"test:coverage:ci": "vitest run --coverage --project=!fuzz-worker && pnpm test:fuzz-worker",
"test:fuzz-worker": "AGENT_DEVICE_COVERAGE_SHARD= AGENT_DEVICE_COVERAGE_MERGE= vitest run --project fuzz-worker",

I put it on this side because the two-leg split is the only thing that knows there are two legs — moving the env from job-level to step-level in #1996 wouldn't help, since both legs live inside the single gate: unit-ci step. It is inert until #1996 lands and required the moment it does, so neither PR has to wait for the other.

Re-ran the whole sharded lane on the merged tree with the fix:

leg	result
AGENT_DEVICE_COVERAGE_SHARD=1/2	exit 0 — 547 files, blob-1.json 22.8 MB, all five instrumented projects
AGENT_DEVICE_COVERAGE_SHARD=2/2	exit 0 — 547 files, blob-2.json 22.9 MB
AGENT_DEVICE_COVERAGE_MERGE=1	exit 0 — merged 1094 passed (1094), 90.38% lines / 88.39% statements, thresholds pass

Two notes for whoever wires this up:

• In merge mode the second leg now runs the fuzz project rather than re-merging (7s on the Report job, and once per shard). That is duplicate work I chose over a conditional in the script; if you'd rather it ran exactly once, ci: consolidate CI workflow from 15 jobs to 8 #1996 can gate the step and I'll drop the blanking for MERGE.
• The blanking hard-codes ci: consolidate CI workflow from 15 jobs to 8 #1996's env names. If those get renamed before it lands, this needs the same rename — vitest.config.ts carries the reason next to the project so it is findable.

scripts/gate/model.test.ts now pins the real shape (nested script + env prefix), so the lane can't quietly stop owning the project it hands to that leg.

---

On the docs/agents/pull-requests.md overlap with #1997 — confirmed via git merge-tree, but it is semantic, not just textual. #1997 rewords that bullet while keeping vitest run --project unit-core --project subprocess-stub, so resolving it by taking #1997's side would silently drop --project fuzz-worker and leave the documented command missing a project. Whoever merges second should re-add the flag rather than pick a side. Happy to rebase onto #1997 instead if it lands first.

[thymikee]

Reviewed exact head 1dd134fbb. The fuzz-worker split is execution-correct: corpus replay stays in test:unit, leaves the instrumented coverage leg, and exact-head Coverage/gate ownership pass. Two follow-ups remain before readiness: update the stale “plain/bare vitest run --coverage” wording in scripts/gate/declarations.ts and scripts/gate/audit-coverage.test.ts; then re-run the sharded coverage lane on the combined tree after #1996 lands (the env reset is at the right boundary, but that integration is not present on this head). The current macOS red is a confirmed DEVICE_IN_USE retry-cleanup infrastructure bug shared with merged #1952, not this CI split; a focused fixer is now running.

[thymikee]

Review of exact head eb169d794b3e506819051a1bc081291e530c77ed: the coverage split is functionally sound and is the smallest owning containment. The disappearing corpus replay moves to one serialized fuzz-worker project, regular unit runs still execute it, coverage excludes it, and the gate model structurally parses both legs. Exact-head CI is fully green, including Coverage, Integration, and macOS smoke.

One P3 cleanup remains: scripts/gate/declarations.ts:4-5 and scripts/gate/audit-coverage.test.ts:98-99 still describe coverage as a plain/bare vitest run --coverage, which is no longer accurate after the excluded-project plus nested-script shape. Update that rationale.

Sequence is also required: land #1996 first, then rebase #1994, resolve the real vitest.config.ts conflict while preserving both the shard/merge switches and #1994's project/env reset, and rerun the combined coverage lane.

[thymikee]

#1996 is now merged and this branch is conflicting as anticipated. Rebase onto current main, preserve #1996's shard/merge switches plus #1994's fuzz-worker project/env reset, fix the stale bare-coverage rationale noted above, and rerun the combined coverage lane. No new code review is needed until the head changes.

[thymikee]

Rebased onto main with #1996 and #1997 landed — now at ca3dd64. Both follow-ups done.

The vitest.config.ts merge came out clean. #1996's shard/merge switches and this PR's fuzz-worker project + env reset auto-merged without a conflict; verified both survive. The real conflict was the one I flagged: docs/agents/pull-requests.md, and it was semantic as predicted — #1997's reworded bullet still said --project unit-core --project subprocess-stub. Resolved by keeping #1997's terser wording and re-adding --project fuzz-worker, rather than taking a side.

Combined coverage lane, re-run on the real merged tree (not the synthetic merge from before):

leg	result
AGENT_DEVICE_COVERAGE_SHARD=1/2	exit 0 — 549 files, blob-1.json 23.0 MB
AGENT_DEVICE_COVERAGE_SHARD=2/2	exit 0 — 548 files
AGENT_DEVICE_COVERAGE_MERGE=1	exit 0 — 1097 passed (1097), 8136 tests, 90.38% lines / 88.37% statements, thresholds pass

549 + 548 = 1097, and the fuzz leg runs on each. Negative control on the same tree, so the guard is demonstrably load-bearing rather than defensive:

$ AGENT_DEVICE_COVERAGE_SHARD=1/2 npx vitest run --project fuzz-worker
Error: --shard <count> must be a smaller than count of test files. Resolved 1 test files for --shard=1/2.
 Test Files  no tests
     Errors  1 error

$ AGENT_DEVICE_COVERAGE_SHARD=1/2 pnpm test:fuzz-worker
 Test Files  1 passed (1)

P3 rationale cleanup. Both sites updated:

• scripts/gate/declarations.ts — OPAQUE_RUNNERS stays empty and the conclusion is unchanged; only the premise had rotted. It now says the script is no longer a single bare run (negated --project leg plus a nested script carrying an env prefix) and that model.ts resolves both, which is why the units still come from the script itself.
• scripts/gate/audit-coverage.test.ts — the helper is still needed for the same reason, so I kept the conclusion and fixed the premise: the two legs together run every declared project, so an unrun project is still only representable once the script names them positively.

Local: typecheck, lint, format:check, check:fallow --base origin/main, check:layering, check:gate-manifest, check:agent-guidance, mutation:test, and the scripts/gate + scripts/check-affected suites all pass.

Thanks for the DEVICE_IN_USE call — that matches what I found from the artifacts. All three macOS reds had the same attempt-1 cause (swift timed out after 10000ms, the 10s swift --version budget on a cold runner) with the retained claim cascading into attempts 2 and 3. One thing worth recording for that fixer: gh run rerun --failed re-samples nothing here — three reruns of the same run failed identically, while amending to a fresh SHA with an identical tree passed first try.

[thymikee]

Re-reviewed ca3dd644b0e4287a92080d78b3f61beb51d6c0ba after #1996 merged: the branch is rebased and mergeable, preserves the shard/merge switches while isolating the fuzz-worker leg, and fixes the stale coverage rationale. Coverage, integration, typecheck, and repository guards are green; iOS and Android smoke are still pending. Code review is clean, so I’m marking it ready for human review while those device lanes finish.