fix(ci): publish the bundle-size baseline instead of tracking it in git

[RaananW]

Problem

The PR Validation → Bundle Size job fails on PRs that have nothing to do with bundle size, and the same mechanism is the repo's dominant source of merge conflicts.

Both symptoms have one root cause: the bundle-size baseline was ~227 generated JSON files tracked in git and refreshed by PR authors. Nearly every change to a shared module moves bytes in most of the ~230 scenes, so:

• Red CI — the moment any shared-code PR merged, every other open PR's committed manifest was stale and its Bundle Size job went red for reasons unrelated to that PR. The only cure was a rebase plus a very slow full local rebuild — and then again on the next merge.
• Merge conflicts — two branches that both regenerated the manifest rewrote the same ~200 files and collided. In PR feat(texture): add opt-in HTML-in-Canvas texture (createHtmlTexture) #523, 168 of the 169 conflicting paths were manifest files and zero were source files.

Fix

Stop putting a generated file in git. The baseline is measured once per master build and published to public storage; PR builds and local runs fetch it over HTTPS.

There is no tracked file to go stale, and nothing for two branches to collide on. Both failure modes become structurally impossible rather than merely less frequent.

Publishing rather than pushing to master also keeps CI out of the repository's write path — which matters, because master is protected (required review + a ruleset with required status checks) and the CI identity deliberately holds no bypass.

Why this is safe

The baseline gates nothing. Every consumer was audited:

Consumer	Behaviour without a baseline
bundle-size.spec.ts:139	console.warn only. The real gate is line 158, rawKB <= scene.maxRawKB from scene-config.json.
report-bundle-size-deltas.ts:133	Sets POST_BUNDLE_COMMENT=false, exits 0.
lab/vite.config.ts:382	mtime for dev-server cache busting.

Size regressions are still gated by the absolute per-scene maxRawKB ceilings, which build:bundle-scenes enforces byte-exactly and which this PR does not touch. A missing or unreachable baseline degrades to "no delta report" and can never fail a build.

Reporting is unchanged

The PR diff contains zero changes to any reporting step. Reviewers still get:

1. the bundle-size delta comment (report-bundle-size-deltas.ts + GitHubComment@0),
2. the ceiling gate, which still fails the job on a real regression.

Changes

File	Change
scripts/bundle-scenes-core.ts	Resolve the baseline from file → HTTPS → git ref (the last only for pre-migration refs). Shape guard so a CDN serving HTML or wrong-shaped JSON is never adopted as a baseline. BUNDLE_MASTER_MANIFEST_FILE / BUNDLE_MASTER_MANIFEST_URL overrides.
azure-pipelines-bundle-manifest.yml	New. Master-triggered. Enforces ceilings first, then uploads manifest.json and purges the CDN. Refuses to publish an empty manifest, so a broken build cannot blank the baseline.
azure-pipelines.yml	Drop the git baseline fetch and the drift gate. Ceiling checks and the delta comment untouched.
scripts/validate-bundle-manifest.ts, scripts/commit-bundle-manifest.ts	Deleted.
lab/public/bundle/manifest/	Untracked (228 files).
bundle-scenes-core.ts	Dropped the device-dependent write suppression — it existed only to protect tracked files.
Docs	GUIDANCE.md, TESTING.md, docs/lite/architecture/38-bundle-size-tooling.md.

Validation

• 6 new unit tests for baseline resolution (404, malformed response, file precedence, unreadable file fallback, blank-URL skip). Mutation-tested: disabling the shape guard and removing the blank-URL guard each fail exactly one test, so neither is vacuous. An earlier draft of these tests was vacuous — mutation testing is what caught it.
• tsc, eslint, prettier clean; all four pipeline YAMLs parse.

⚠️ One-time ADO setup required before merge

1. Register azure-pipelines-bundle-manifest.yml as a pipeline.
2. Authorize the BabylonJS-Deployment and BabylonJS-CI-Infrastructure variable groups for it.

Until then the baseline is simply never published — PR builds skip the delta comment and keep gating on ceilings, so nothing breaks.

Accepted trade-off

Per-scene size history is no longer in git — you can no longer git log a single scene's size over time. In exchange, the two failure modes above disappear entirely.

[Copilot]

Pull request overview

This PR stabilizes CI bundle-size validation by making the per-scene bundle-size manifest baseline CI-owned (auto-refreshed by the Bundle Size job) instead of author-maintained, while keeping size regressions gated by scene-config.json ceilings.

Changes:

• Add a CI script (commit-bundle-manifest.ts) that pushes refreshed per-scene manifest files back to the PR branch (with safeguards for forks, incomplete builds, and non-converging measurements).
• Update validate-bundle-manifest to report drift by default (exit 0) and add a strict mode that preserves the old hard-fail behavior.
• Update docs/tests/pipeline wiring to reflect the new CI-owned baseline workflow and hash-stripped chunk-set comparisons.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/lite/unit/validate-bundle-manifest.test.ts	Adds unit + CLI coverage for drift reporting, strict mode, and hash-stripped chunk comparisons.
tests/lite/unit/commit-bundle-manifest.test.ts	Adds end-to-end tests covering the worktree-based commit/push behavior and safety rails.
TESTING.md	Documents that CI refreshes the per-scene manifest and how to validate drift locally (report vs strict).
scripts/validate-bundle-manifest.ts	Converts validation into report-by-default, adds strict mode, and ignores chunk content hashes when comparing chunk sets.
scripts/commit-bundle-manifest.ts	New script to refresh/push per-scene manifest files from CI without pushing the PR merge commit.
package.json	Adds validate:bundle-manifest:strict and commit:bundle-manifest scripts.
GUIDANCE.md	Updates the workflow guidance to reflect CI-owned manifest refresh and non-gating drift.
docs/lite/architecture/38-bundle-size-tooling.md	Updates architecture documentation to describe the CI-owned baseline and the new scripts.
azure-pipelines.yml	Wires the pipeline to report drift, then refresh the manifest on the PR branch (continueOnError).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260804.4 - merge @ 4dd5ad8}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260804.5 - merge @ 34ae187}

[bjsplat]

Lab - Static Site

Open deployed site

_{Build 20260804.5 - merge @ 34ae187}

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260804.7 - merge @ 0a589ae}

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

scripts/validate-bundle-manifest.ts:298

• This log message promises that a CI auto-commit step will refresh the manifest, but in some CI contexts (e.g. fork PRs or missing GITHUB_TOKEN) the refresh step cannot push and the manifest will remain stale. That can mislead readers of the build log and complicate debugging.

Consider wording this as an attempt/conditional refresh rather than a guarantee.

    console.log(`Bundle manifest drift detected — the CI auto-commit step will refresh it.\n${detail}`);

scripts/commit-bundle-manifest.ts:138

• countLeadingAutocommits() treats any commit subject that starts with the bot subject as an autocommit. That can misclassify a human commit like "chore(bundle): refresh per-scene bundle-size manifest (manual)" and trigger the non-convergence loop guard earlier than intended.

Since this script controls the exact subject it emits, an exact equality check is safer and clearer.

        if (!subject.startsWith(COMMIT_SUBJECT)) break;

scripts/commit-bundle-manifest.ts:197

• This warning tells fork-PR authors to run a full pnpm build:bundle-scenes and commit the entire manifest directory. That’s both heavier than necessary and inconsistent with the updated workflow where the manifest is generally CI-owned.

Since fork PRs are the special case where CI cannot push, it’s clearer to say CI cannot refresh it and that the author should regenerate (filtered scenes is fine) and commit the affected per-scene manifest files if they want the baseline updated in the diff.

        warn("Fork pull request — cannot push the refreshed bundle-size manifest. Run 'pnpm build:bundle-scenes' and commit lab/public/bundle/manifest/ yourself.");

[bjsplat]

Lab - Static Site

Open deployed site

_{Build 20260804.7 - merge @ 0a589ae}

[RaananW]

Leaving a comment here to avoid unwanted merging. This is a change we can discuss!

[deltakosh]

no problem for me

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (4)

azure-pipelines-bundle-manifest.yml:40

• The path filter excludes lab/public/bundle/manifest as a literal path, which does not reliably exclude files under that directory in Azure Pipelines. That can let the bot’s commit (touching lab/public/bundle/manifest/*.json) retrigger this pipeline, defeating the “single writer / no self-trigger” safety property.

    paths:
        exclude:
            - lab/public/bundle/manifest

azure-pipelines-demos.yml:16

• This trigger exclusion has the same issue as the bundle-manifest pipeline: excluding lab/public/bundle/manifest may not exclude changes to files inside that folder, so the manifest bot commit can still trigger a demos deploy.

        exclude:
            - lab/public/bundle/manifest

azure-pipelines-playground.yml:29

• This trigger exclusion may not match files under lab/public/bundle/manifest/ unless it’s written as a folder glob. If it doesn’t, the manifest bot commit can still trigger the playground deploy/purge pipeline.

        exclude:
            - lab/public/bundle/manifest

scripts/validate-bundle-manifest.ts:271

• This error message still instructs contributors to regenerate + commit the per-scene manifests. With the new master-owned baseline design, that guidance is misleading: PR branches should restore the baseline from master, not commit regenerated manifests from the branch.

    if (committed === null) {
        console.error(`No committed manifest found under ${MANIFEST_DIR_REL_PATH}/ at HEAD.\nRun 'pnpm build:bundle-scenes' and commit the generated per-scene manifest files.`);
        process.exit(1);
    }

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260804.12 - merge @ 81d65fe}

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (3)

azure-pipelines-playground.yml:29

• Same as the demos pipeline: exclude lab/public/bundle/manifest/** (recursive) so the manifest-refresh bot commit does not trigger a full playground build + deploy on every merge.

    paths:
        # The bundle-manifest bot (azure-pipelines-bundle-manifest.yml) pushes a
        # commit touching only this directory after every merge. It cannot change
        # the playground, so exclude it rather than paying for a full build and
        # CDN deploy/purge on each one.
        exclude:
            - lab/public/bundle/manifest

azure-pipelines-demos.yml:16

• Same as the bundle-manifest pipeline: the trigger path exclusion should exclude files under lab/public/bundle/manifest/, not just the directory path. Otherwise the manifest-refresh bot commit can still trigger an expensive demos deploy.

    paths:
        # The bundle-manifest bot (azure-pipelines-bundle-manifest.yml) pushes a
        # commit touching only this directory after every merge. Nothing here
        # affects the demos, so exclude it rather than paying for a full build
        # and CDN deploy on each one.
        exclude:
            - lab/public/bundle/manifest

azure-pipelines-bundle-manifest.yml:40

• Azure Pipelines path filters are glob-based; excluding lab/public/bundle/manifest may not exclude changes under that directory. If the bot commit modifies lab/public/bundle/manifest/<scene>.json, this pipeline can still be retriggered on every refresh commit, defeating the “single writer” loop-safety goal. Use a recursive glob to exclude all files under the directory.

    paths:
        exclude:
            - lab/public/bundle/manifest

[bjsplat]

Lab - Static Site

Open deployed site

_{Build 20260804.12 - merge @ 81d65fe}

[Copilot]

Pull request overview

Copilot reviewed 240 out of 241 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

lab/vite.config.ts:244

• This comment still claims a “fresh checkout” can populate the Bundle tab without a full build by synthesizing from per-scene files. After this PR, the per-scene files are also gitignored, so on a fresh checkout neither aggregate nor per-scene manifests exist. Consider rewording to the narrower (still true) case: synthesize only when per-scene files exist but the aggregate hasn’t been generated (e.g. partial/failed build).
  azure-pipelines-bundle-manifest.yml:145
• The script says it “refuses to publish an empty baseline”, but test -s only checks non-zero file size. A broken build could still produce {} (non-empty file, 0 scenes) and this step would publish it, causing all consumers to treat the baseline as missing (shape guard rejects empty manifests) until the next successful run. Add a SCENES==0 guard before uploading.

                      if ! test -s "$MANIFEST"; then
                        echo "No aggregate manifest at $MANIFEST — refusing to publish an empty baseline."
                        exit 1
                      fi

                      SCENES="$(node -e 'process.stdout.write(String(Object.keys(require(process.argv[1])).length))' "$MANIFEST")"
                      echo "Publishing bundle-size baseline for $SCENES scene(s)."

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260805.15 - merge @ 0ab9381}

[Copilot]

Pull request overview

Copilot reviewed 243 out of 244 changed files in this pull request and generated no new comments.

Suppressed comments (2)

tests/lite/unit/bundle-master-baseline.test.ts:75

• expect(promise).resolves.not.toThrow() is not a valid Vitest/Jest assertion because toThrow() expects a function, not a resolved value. This will error even when the promise resolves. If the intent is just “promise does not reject”, assert that it resolves (e.g. toBeDefined()), while still allowing a null baseline.
  tests/lite/unit/bundle-content-no-f64.test.ts:86
• HAS_BUILT_CHUNKS is documented as implying HAS_MANIFEST, but the boolean expression doesn’t enforce that. If chunk files exist while manifest/ is missing (e.g. partial cleanup / stale local state), tests guarded by HAS_BUILT_CHUNKS will still run and then throw when loadManifest() reads MANIFEST_DIR. Make HAS_BUILT_CHUNKS depend on HAS_MANIFEST (or guard the relevant tests with both).

[Copilot]

Pull request overview

Copilot reviewed 243 out of 244 changed files in this pull request and generated no new comments.

Suppressed comments (3)

tests/lite/unit/bundle-master-baseline.test.ts:76

• await expect(resolveMasterBundleManifest()).resolves.not.toThrow() is not a valid Vitest/Jest pattern: toThrow expects a function, so this assertion can fail even when the promise resolves successfully. This test should simply await the promise (and then assert on observable side effects like requestCount).
  tests/lite/unit/bundle-content-no-f64.test.ts:81
• HAS_MANIFEST is true if any JSON exists in the manifest directory, but the manifest assertions assume at least scene2.json exists. If someone runs this after a filtered bundle build that didn’t include scene2, the test will run and fail with a confusing "scene2 missing" error instead of self-skipping as intended.
  docs/lite/architecture/38-bundle-size-tooling.md:92
• This section says the new baseline pipeline "purges the CDN", but azure-pipelines-bundle-manifest.yml does not actually run a purge step (unlike the playground/demos pipelines). Either add the purge step or adjust this documentation so it doesn’t promise behavior the pipeline doesn’t implement.

`azure-pipelines-bundle-manifest.yml` re-measures on each push to master and
uploads `manifest.json` to the deployment server, then purges the CDN so the new
baseline is visible promptly. It refuses to publish an empty or missing manifest,
so a broken build cannot blank the baseline, and the ceiling check runs first so a
breach can never be published as the new normal.

[bjsplat]

Lite Playground - Static Site

Open deployed site

_{Build 20260805.17 - merge @ 0b9464e}

[Copilot]

Pull request overview

Copilot reviewed 243 out of 244 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/lite/unit/bundle-content-no-f64.test.ts:86

• HAS_BUILT_CHUNKS is documented as implying HAS_MANIFEST, but the current definition does not include HAS_MANIFEST. If a developer has stale chunk files under lab/public/bundle/ but the manifest directory is missing/cleaned, HAS_BUILT_CHUNKS can be true and tests like ${HPM_OFF_SCENE}: no runtime chunk contains the F64 build tag will call loadManifest() and throw on readdirSync(MANIFEST_DIR). Tie HAS_BUILT_CHUNKS to HAS_MANIFEST (or guard the tests that call loadManifest) to avoid this crash in partial-build states.

[bjsplat]

Lab - Static Site

Open deployed site

_{Build 20260805.17 - merge @ 0b9464e}