test: refuse foreign-pid signals from unit-test workers (Coverage fork death, #1824)

[thymikee]

Summary

Refs #1824 — this closes one of two causes behind that signature; see "A second, independent cause" below. The issue stays open on its 20-run criterion.

The Coverage job's "Worker exited unexpectedly" was not OOM. runner-session.test.ts and runner-request-cancellation.test.ts fabricate runner child pids (makeBackgroundRunner(4242), plus 4141/4343/4444) and mock the liveness reads in src/utils/host-process.ts but not the signal writes, so killRunnerProcessTree delivered real SIGINT/SIGTERM/SIGKILL to those pids and to their process groups (process.kill(-4242, …)) — 146 signals per run of runner-session.test.ts. On the ubuntu runner the sibling vitest forks live in exactly that pid band, so the job periodically killed one of its own forks: one file's results vanish, no test fails, and the pool reports an unhandled error 30–40 s later when the run settles.

The negative-pid half is the sharper edge and the reason this is fixed at the seam rather than by renumbering the fake pids: -4242 addresses a whole process group, so a local pnpm test can deliver SIGKILL to an unrelated group leader's children on a developer machine. Any constant a test invents is someone's live pid somewhere.

Two changes, both at the owning interface:

• signalProcessGroupBestEffort joins signalPidsBestEffort in src/utils/host-process.ts, and runner-disposal.ts's raw process.kill(-pid, …) moves behind it. Signal writes now live in the same module as the liveness reads, so a test mocks both in one place — which runner-session.test.ts, runner-request-cancellation.test.ts, and runner-disposal.test.ts now do.

• src/__tests__/hermetic-signal-setup.ts (new, in the shared SETUP_FILES — now exported from vitest.config.ts and imported by vitest.mutation.config.ts, so the Stryker lane cannot drift off it) makes the class impossible rather than trusting each test: a worker may signal only itself and its own direct children. It closes both exits, because the runner-disposal family uses both — process.kill (answered ESRCH) and a spawned kill/pkill/killall (answered ENOENT), whose -P and -f forms reach processes the worker never spawned at all. Best-effort kill paths therefore proceed as if nothing matched, no write leaves the worker, and the sending test fails by name in afterEach/afterAll. Signal 0, the liveness probe, stays free; killing a daemon or Metro fixture the test itself spawned stays allowed.

  The spawned half is not hypothetical: request-router-open.test.ts ran a real pkill -f 'xcodebuild.*AgentDeviceRunner\.env\.session-…' twice per suite run, so on a developer's Mac pnpm test could reach a live runner. It now stubs the Apple tool seam, the way the runner tests stub the signal seam.

runner-session.test.ts was over the 1,000-line tripwire, so its fabricated session/lease/child fixtures moved to runner-session-fixtures.ts (the sibling-fixtures pattern AGENTS.md prescribes) instead of growing the file; the size-ratchet pin ratchets down, 2083 → 2001. docs/agents/testing.md documents the invariant.

Validation

Cause confirmed on CI, before the fix — run 32165517507, Coverage job 95804064013, instrumentation-only head 2711b116f. Sentinel processes were parked on 4141/4242/4343/4444 before pnpm test:coverage:ci:

pid 4242: DEAD   received SIGTERM 17:27:01.909
pid 4141: DEAD   received SIGTERM 17:28:10.302
pid 4343: DEAD   received SIGTERM 17:28:10.387
pid 4444: DEAD   received SIGTERM 17:28:10.407
kernel OOM / kill lines: (none)          # 16 GB runner
runner-session.test.ts: 26x kill -4242 SIGTERM | 27x kill -4242 SIGKILL | 20x kill -4242 SIGINT (+ same to 4242)

The split is itself corroboration: 4242 dies 223 ms after the first unit-core file starts — runner-session.test.ts, which signals only 4242 — and the other three die together 69 s later, which is where runner-request-cancellation.test.ts (4141/4242/4343/4444) runs.

Fix verified with the same probe — run 32222448997, Coverage job 95975387817. Note this is a different base from the before-run (2711b116f → 84ffcc744, 71 files and +3272 lines of unrelated main between them), so it is the probe that is held constant, not the tree:

pid 4141: ALIVE   pid 4242: ALIVE   pid 4343: ALIVE   pid 4444: ALIVE
Test Files  983 passed (983)    Tests  7238 passed | 3 skipped

Neither runner test appears in the trace at all any more; the only signals left in the whole run are tests killing daemon/Metro children they spawned themselves. (The report line signals aimed at a pid that was also a vitest fork: 0 printed in both runs and proves nothing — planting the sentinels burns ~1977 throwaway pids, which pushes every fork in the instrumented run above 4444. I am not offering it as a contrast.)

What is observed vs inferred. Observed: the tests deliver real signals to those pids, the signals kill whatever holds them, and the fix stops both. Inferred: that the victim in the six catalogued deaths was a sibling fork. The arithmetic behind the inference — in an uninstrumented run the ~1977 planting pids do not exist, so the first src/ file's worker (pid 6128 before, 6121 after) would land near 6128 − 1977 ≈ 4151, inside 4141–4444 — plus the victims being exactly the files alive in that window. No run captures a fork's death certificate, so the mechanism evidence carries this, not a single smoking-gun frame.

Vanished-file forensics — in all six recent deaths (runs 32041468051, 32044136488, 32047734673, 32126085337, 32151259761, 31783654175) the file whose results disappear is one of the three largest unit-core files (snapshot-handler ×3, runner-session ×2, daemon-client ×1) — exactly the files alive in the same scheduling window as runner-session.test.ts, ~50–60 s into the run. The issue's "last ~40 s" was the report-print time, not the death time.

Red proof, direct writes — guard on, tests unfixed: runner-session.test.ts + runner-request-cancellation.test.ts fail 28 of 76, each naming the refused signal, the pid, and the delivery path:

Error: This test tried to send 4 real signal(s) to a pid this vitest worker did not spawn. …
SIGTERM -> pid -4242 (runner session invalidates after runner-fatal ok payloads)
    at signalProcessGroupBestEffort (src/utils/host-process.ts:165:11)
    at killRunnerProcessTree (src/platforms/apple/core/runner/runner-disposal.ts:250:3)
    at disposeRunnerSession (src/platforms/apple/core/runner/runner-disposal.ts:50:11)

With the test-side mocks restored: 76/76 pass.

Red proof, the guard's own holes — hermetic-signal-setup.test.ts covers the two ownership gaps re-review found. Reverting either fix reds three of its seven cases:

× authority ends when a child is reaped, so its recycled pid is refused
× a promisified execFile cannot smuggle a pkill past the guard
× a promisified execFile still tracks the child it starts

signalProcessGroupBestEffort also carries its own behavioral tests (delivery to -pid, ESRCH → false, and a refusal to touch process.kill at all for pid ≤ 0, which would otherwise address the caller's own group).

Green — pnpm check:affected --run green on the final head (unit lane 2807/2807 with the guard active). Local full-suite runs under 100–280 host load flaked only on the documented contention files (ios-lifecycle, interaction-response-shape, fuzz/harness), which fail identically on plain origin/main under the same load, and pass in isolation.

Four lines

• Catches: the silent one-fork death ([vitest-pool]: Worker forks emitted error) behind 6 of the last ~40 red CI runs — and any future test that signals a pid it did not spawn, which now fails by name instead of killing an unrelated process or process group on the host.
• Evidence: sentinel + kill-trace CI experiment, dead before / alive after (runs 32165517507 → 32222448997); six-run vanished-file table on Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824; 28-test planted red.
• Cost: a 170-LOC setup file with 94 LOC of tests, plus one host-process helper; no measurable wall clock (per-fork setup, microseconds), no macOS runner time, no new lane, no red-lane attention beyond this PR.
• Kill-criterion: not a lane. Done when 20 consecutive Coverage runs show no fork death — checklist on Coverage job: "Worker exited unexpectedly" kills one fork in the last ~40s of the run (4 of 17 recent CI failures) #1824, which stays open until then (1/20 recorded). A recurrence of the tail shape below does not reset that count; it belongs to the other cause.

A second, independent cause (not fixed here)

PR #1866 hit the identical pool error on run 32223038279 after making scripts/fuzz/corpus-replay.test.ts heavier (a large module graph pulled into the V8-instrumented run), and fixed it by shrinking that graph. I checked whether it was my mechanism with the timing shifted. It is not:

• The vanished file is corpus-replay.test.ts, a subprocess-stub member. That project runs fileParallelism: false, maxWorkers: 1 in a group that starts only after unit-core drains: last unit-core file ✓ 06:27:32, stub files 06:27:38 → 06:27:49, death reported 06:28:43. It was the only file executing.
• The pid-signalling files finish ~3.5 minutes earlier — in my own instrumented run, runner-session.test.ts ✓ 17:27:02 vs corpus-replay.test.ts 17:30:33. Their forks are gone, and late forks carry pids ~14 000, nowhere near 4141–4444.
• My kill tracer logged every foreign-pid signal across a whole 983-file run: only those two files, only in the early window. It would have recorded nothing at corpus-replay's death.
• The log carries [vitest-pool]: Timeout terminating forks worker for … corpus-replay.test.ts and no JavaScript heap out of memory — a silent death of a bloated fork, i.e. an external SIGKILL such as the kernel OOM killer (which prints only to dmesg), not an in-process V8 heap abort.

So there are two death shapes behind #1824: (A) an early unit-core file dies while other files run and the fabricated-pid signals are in flight — that is what the sentinels proved, dead before / alive after, and what this PR fixes; (B) a single heavy instrumented file dies alone in the tail under memory pressure — #1866's, unaffected by this PR. Five of the six deaths I catalogued have shape (A); #1866's has shape (B). I am not claiming this PR closes (B), and the 20-run criterion is what distinguishes a real recurrence from a fluke either way.

Residual risk

The guard covers process.kill and spawned kill/pkill/killall from a test worker. What it cannot see is a signal sent from inside a subprocess a test spawned (that process's own process.kill runs in another runtime); no such sender exists in the repo today. And shape (B) above remains open: if the tail deaths continue, the follow-up is a fork heap ceiling or a coverage shard, not this seam.

Touched files: 14 across the three commits (the earlier "11" counted only the first commit, not the review follow-ups). Scope stayed inside the test harness plus the Apple runner disposal seam.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.31 MB	2.31 MB	+97 B
JS gzip	759.3 kB	759.3 kB	+12 B
npm tarball	882.6 kB	882.6 kB	+10 B
npm unpacked	3.08 MB	3.08 MB	+97 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.2 ms	28.1 ms	+0.9 ms
CLI --help	69.4 ms	71.4 ms	+2.0 ms

Top changed chunks:

Chunk	Raw diff	Gzip diff
dist/src/runner-disposal.js	-14 B	-4 B

[thymikee]

Reviewed exact 2711b116: the Coverage experiment confirms the unsafe collision class. All four fabricated-PID sentinels (4141/4242/4343/4444) died from SIGTERM, and the trace attributes the foreign signals to runner-session / runner-request-cancellation tests. No actual Vitest fork overlapped those PIDs in this run, so this proves the unsafe signaling behavior rather than reproducing the fork death itself. Intentionally not ready: keep this draft and unlabeled until the real fix and regressions replace all temporary global instrumentation, the body is updated, and exact-head CI completes. No device evidence applies.

[thymikee]

CI note (head 3e7e22724, the instrumentation-free head): 27/28 checks green, including the lane this PR is about — Coverage passed 983 files / 7238 tests with no fork death.

The one outstanding check is Smoke Tests (Linux), which is stuck in Install Linux desktop dependencies (apt) and hit the job's 30-minute timeout; a gh run rerun --failed is stuck in the same step. It is a fleet-wide condition today, not this branch: #1866's Linux lane was cancelled identically, and three other branches' lanes were in the same step at the same time. Nothing in this diff reaches that lane — it never gets past apt to run any repo code.

Leaving this in draft until that lane is green rather than undrafting on a partial signal. It needs one more rerun once the mirror recovers; no code change is pending.

[thymikee]

Re-reviewed exact 090cfb9bb: temporary instrumentation is gone, the direct runner signal seam is fixed, and direct spawned kill binaries are now intercepted. Two P1 ownership holes remain:

1. Async child PIDs stay authorized for the worker lifetime after exit/close. Once reaped, that numeric PID can belong to an unrelated process, recreating the PID-reuse hazard this guard exists to prevent. Remove exited children from authority (or prove process identity/start time) and add a post-exit reuse regression; idempotent cleanup must treat exit as dead.
2. wrapSpawner copies the original util.promisify.custom from execFile, so promisify(execFile) invokes Node’s original implementation and bypasses both kill-binary refusal and child tracking. Wrap that custom path too and add a promisified-execFile("pkill", …) refusal regression.

Fresh exact-head CI is still pending. The body also needs updating: effective scope is 13 files, not 11, and the setup is 140 LOC, not ~85.

[thymikee]

CI note: the two iOS reds on this PR are not from this diff — filed as #1874. Run 32229709270 failed RunnerTests.testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden, which failed identically on the unrelated agent/fix-install-source-network-cause (#1863) in run 32176591415 yesterday; the earlier red on this PR was typed email suffix should be observable, the same bare-type-with-hidden-keyboard behaviour one layer up. This branch touches no Swift and no iOS production code — the diff between its last two heads is test setup, docs and two vitest configs. Rerunning.

[thymikee]

Undrafted on green CI. 28/28 checks pass on 090cfb9bb after rerun. The two earlier iOS reds were not from this diff — testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden failed identically on the unrelated #1863 (run 32176591415), now filed as #1874; this branch touches no Swift and no iOS production code. All four adversarial-review findings are addressed: the guard now refuses spawned kill/pkill/killall as well as process.kill (reproduced red first — the daemon test failed by name on both its pkill -f spawns — then clean across a 983-file/7241-test sweep), SETUP_FILES is exported and imported by vitest.mutation.config.ts so the lists cannot drift, the docs say direct children, and the evidence section now states that signals aimed at a fork: 0 prints in both runs and that the fork-was-victim step is an inference with the arithmetic shown.

[thymikee]

Both P1s were real, and both were the guard failing at its own premise. Fixed in 10c4db829.

1. Authority now ends at child exit. You're right and my earlier reasoning was backwards: a pid is a claim on a process-table slot, not on a process, and the kernel reissues that slot once it's free — so "I spawned this pid once" was licensing exactly the reuse hazard the file exists to close. rememberChild now drops the pid on exit and close (a child that fails to spawn emits only the latter). I checked the objection I'd raised: the cleanup paths that motivated it — stopProcess in client-metro.test.ts:1091, the finally blocks in daemon-client.test.ts — already gate on isProcessAlive, so treating exit as dead costs nothing and the false-positive I was protecting against doesn't exist in the tree. A full 984-file sweep produced zero refusals.

2. The promisified path is wrapped. promisify(execFile) never calls the function, so copying the original util.promisify.custom across handed every promisified caller an unguarded execFile — both refusal and child tracking bypassed. There are two live callers (help-conformance-bench.test.ts, size-report-post-comment.test.ts), which is why the copy was there in the first place. The wrapper now defines its own custom implementation that refuses kill binaries (rejecting, since that's what an async ENOENT looks like), remembers promise.child, then delegates.

Regressions — new src/__tests__/hermetic-signal-setup.test.ts, 7 cases, both of the ones you asked for plus the allowed cases they could regress into. It drains the guard's refusal buffer through a drainRefusedWritesForTest export so it can drive refusal paths without the afterEach hook failing it. Reverting either fix reds three:

× authority ends when a child is reaped, so its recycled pid is refused
× a promisified execFile cannot smuggle a pkill past the guard
× a promisified execFile still tracks the child it starts

Body facts corrected: 14 files (not 11 — I'd counted only the first commit; 13 was right before this commit added the test file), and the setup file is 170 LOC with 94 LOC of tests, not ~85.

CI: pnpm check:affected --run green on 10c4db829; run 32248983600 is in flight. Two notes on what I saw while validating, neither a code issue: the first gate attempt failed on runner-client.test.ts > ensureXctestrunArtifact aborts only the disconnected request build — a 5 s timeout with zero guard refusals in the whole run; it passes 3/3 in isolation and the rerun was clean. And ios-lifecycle.test.ts flakes on this host regardless of the guard: same tree, guard on → 1 pass / 1 fail, guard off → 2 fails.

[thymikee]

Re-reviewed exact 10c4db829: both P1 ownership holes are closed. Child PID authority is revoked on exit/close, and the custom promisified execFile path now enforces kill-binary refusal while tracking promise.child. The focused regressions are non-vacuous: reverting either fix reds the named ownership/promisify cases, and the body’s scope/evidence facts are current.

Code review is clean. Exact-head CI is 27/28 green with iOS Smoke still pending; no device evidence applies to this test-harness/host-process change.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-19 12:00 UTC