Skip to content

ci: retry once when vitest loses its browser page - #1615

Merged
obiot merged 1 commit into
masterfrom
fix-browser-flake
Aug 29, 2026
Merged

ci: retry once when vitest loses its browser page#1615
obiot merged 1 commit into
masterfrom
fix-browser-flake

Conversation

@obiot

@obiot obiot commented Aug 29, 2026

Copy link
Copy Markdown
Member

The browser suite aborts at a random point with:

Error: Failed to run the test <spec>.
Caused by: [vitest] Browser connection was closed while running tests.
Caused by: [birpc] rpc is closed, cannot call "createTesters"
 Test Files  <N> passed (263)

A different victim spec every time — six distinct ones observed — and zero
failed tests
in every occurrence. Measured at 4 deaths in 21 local runs; on CI
it took three attempts to land a PR that changed nothing but Markdown.

Cause

Vitest 4.1.11 runs each worker as one long-lived orchestrator page and dispatches
every spec over a single websocket. Its ws.on("close") handler destroys the
session and rejects the pending createTesters, so losing that socket kills the
whole run — the in-page client does attempt to reconnect, but the session is
already gone. That is why no test ever fails and the named spec is just whichever
was in flight.

Underneath, the browser intermittently severs a renderer's network-service
channels: every websocket belonging to that page dies within ~3 ms, the server
receives a graceful 1001 while the page sees 1006 wasClean=false, and both
processes stay alive with healthy memory.

Ruled out by direct measurement:

hypothesis result
memory / OOM 2.46 GB peak against 64 GB
CPU starvation 5/5 clean under 14 competing hogs
browser crash no crash event, no signal in stderr
page navigation / reload zero in ~90 instrumented runs
worker concurrency single worker was worse, 2/6
drawmesh_bench (#1612) skips in ~30 ms, no stall in any log
CI-specific reproduces locally

Why a retry rather than a fix

Launching the full Chromium build instead of the headless shell fixes it
locally — 20 consecutive clean runs against a baseline that died 4 times in
21. It does nothing here: 0 of 4 CI attempts passed with it applied. So it
was dropped, and this retries instead.

The condition is deliberately narrow — the page must have been lost and no
test may have failed:

grep "Browser connection was closed"  &&  ! grep "Test Files.*N failed"

A genuine failure prints N failed in the summary and exits immediately with no
second attempt. Both branches were verified against real captured logs from this
investigation: the flake log retries, a 1 failed | 262 passed log does not.

texture.spec.js

A latent bug found on the way, independent of the retry. The app is built with
video.AUTO while the test asserts boundTextures and currentTextureUnit
WebGL texture-unit bookkeeping that no WebGPU batcher keeps. It passes only
because the headless browser has no GPU and falls back to WebGL; on a machine
where WebGPU is available it fails. Now gated on the renderer rather than on the
fields being present, so a WebGL regression that drops them still fails rather
than silently skipping.

Worth reporting upstream

Vitest destroying the session on the first socket close while its own client is
still retrying, and @vitest/browser-playwright silently ignoring an
instance-level launch: key (it only reads launchOptions from the
playwright({...}) factory call — setting the channel on the instance looks
correct, does nothing, and quietly re-measures the default).

Copilot AI lite review requested due to automatic review settings August 29, 2026 07:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@obiot
obiot force-pushed the fix-browser-flake branch from 884e673 to 43af892 Compare August 29, 2026 07:55
Copilot AI review requested due to automatic review settings August 29, 2026 07:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@obiot obiot changed the title test: stop the browser suite losing its page mid-run ci: retry once when vitest loses its browser page Aug 29, 2026
The browser suite aborts at a random point with "Browser connection was closed
while running tests", naming a different victim spec every time and reporting
zero failed tests. Measured at 4 deaths in 21 local runs; landing a PR that
changed nothing but Markdown took three attempts.

Vitest 4.1.11 runs each worker as one long-lived orchestrator page and dispatches
every spec over a single websocket. Its ws close handler destroys the session and
rejects the pending `createTesters`, so losing that socket kills the whole run —
the in-page client does retry, but the session is already gone. Underneath, the
browser intermittently severs a renderer's network-service channels: every
websocket for that page dies at once, the server sees a graceful 1001 while the
page sees 1006, and nothing crashes. Ruled out by measurement, not argument:
memory (2.46 GB peak of 64 GB), CPU starvation (5/5 clean under load), browser
crash (no crash event or signal), page navigation (zero in ~90 instrumented
runs), worker concurrency (single worker was worse), and the earlier
drawmesh_bench fix (skips in ~30 ms, no stall in any log).

Launching the full Chromium build rather than the headless shell fixes it
locally — 20 consecutive clean runs — but does nothing on CI: 0 of 4 attempts
passed. So that is not the fix, and this retries instead.

The retry condition is deliberately narrow: the page must have been lost AND no
test may have failed. A genuine failure prints "N failed" in the summary and
exits immediately without a second attempt. Both branches were checked against
real captured logs from this investigation.

texture.spec.js is a latent bug found on the way, unrelated to the retry. Its app
is built with `video.AUTO` while the test asserts `boundTextures` and
`currentTextureUnit` — WebGL texture-unit bookkeeping that no WebGPU batcher
keeps. It passes only because the headless browser has no GPU and falls back to
WebGL; anyone running the suite where WebGPU is available fails on it. Now gated
on the renderer rather than on the fields being present, so a WebGL regression
that drops them still fails instead of silently skipping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Aa37KGXZcnVrbn1yG4j1N
@obiot
obiot force-pushed the fix-browser-flake branch from 43af892 to e3dff15 Compare August 29, 2026 09:43
Copilot AI review requested due to automatic review settings August 29, 2026 09:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@obiot
obiot merged commit 8f05ba0 into master Aug 29, 2026
6 checks passed
@obiot
obiot deleted the fix-browser-flake branch August 29, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants