Cache Azure Linux packages in CI

[Amaury Chamayou (achamayou)]

Summary

• cache Azure Linux 3 and 4 RPMs through one local composite action
• rotate the immutable RPM cache every Sunday at midnight UTC, warm-seeding each new cache from the latest compatible entry
• always refresh repository metadata and keep TDNF/DNF online so stale or incomplete caches cannot block CI
• treat Actions cache failures as non-fatal and fall back to the package repositories
• remove cache-only installs, checked-in snapshot/cache-generation files, and version-specific actions
• use TDNF snapshots only when the release workflow supplies SOURCE_DATE_EPOCH for reproducible builds

Motivation

CI repeatedly downloads roughly 800 MB while installing dependencies from packages.microsoft.com. Those downloads have produced intermittent timeouts. The cache is an optional reliability layer, not a reproducibility mechanism: normal CI should use current repository metadata and remain able to install online.

GitHub Actions caches are immutable, so an unchanged key would never absorb newer RPMs downloaded by later jobs. Including the date of the most recent Sunday in the key creates a refreshed cache once a week without requiring a repository change.

Download impact

The fixed-image Azure Linux 3 experiment measured:

Measurement	Uncached	Warm RPM cache
Dependency-install ingress	802,196,129 B	31,494,860 B
Repository metadata on disk	63,956,932 B	restored in the earlier offline experiment
Full Actions cache archive	N/A	752,090,120 B

This version deliberately discards restored metadata and downloads current metadata before installing. Conservatively treating the entire 63,956,932-byte on-disk metadata footprint as additional network transfer gives a warm-path upper bound of 95,451,792 B. That avoids at least 706,744,337 B per Azure Linux 3 job, an 88.1% reduction in dependency-install traffic. The actual reduction should be larger because repository metadata is transported compressed.

The Actions cache archive itself is approximately 752 MB, so this is not a claim of a comparable reduction in total runner ingress. It shifts more than 700 MB per warm job away from packages.microsoft.com to the Actions cache service; if that service is unavailable, CI continues with the normal online installation.

Design

The shared action detects Azure Linux 3 or 4 and configures the native package manager to retain RPMs in a persistent directory outside /github/home/.cache, which CI tests clear later in the job.

The primary cache key contains the package manager, runner architecture, dependency-script hash, and the date of the most recent Sunday at midnight UTC. On an exact hit, the immutable weekly cache is reused. At the next weekly rollover, ordered restore prefixes first select the newest cache for the same dependency scripts and then fall back to any cache for the same package manager and architecture. This warm-seeds the new cache, after which the package manager downloads only missing or updated RPMs. actions/cache saves the populated directory under the new weekly key in its automatic post-job phase.

Before installation, the action removes restored repository metadata and runs the existing setup script online. This ensures dependency resolution uses current repository metadata even when the RPM cache was populated earlier.

scripts/setup-ci.sh no longer invents a snapshot timestamp. It adds --snapshottime only when SOURCE_DATE_EPOCH is explicitly set by the release workflow, preserving release reproducibility without pinning ordinary CI.

Validation

• the earlier cold/warm Azure Linux 3 experiment produced identical installed RPM manifests
• Azure Linux 3 and 4 CI jobs successfully created dated weekly caches
• Sunday boundary checks cover the rollover at midnight UTC
• changed workflow and action YAML formatted with Prettier
• Bash syntax checks for both package setup scripts and the composite action blocks
• changed-file ASCII and whitespace checks

[Copilot]

Pull request overview

Caches Azure Linux 3 and 4 CI packages and centralizes dependency installation through a shared composite action.

Changes:

• Adds TDNF/DNF5 caching and offline installation.
• Migrates Azure Linux workflows to the shared action.
• Adds snapshot and cache-generation controls.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.

Show a summary per file

File	Change
scripts/setup-ci.sh	Adds TDNF cache-only support.
scripts/setup-ci-al4.sh	Adds DNF5 cache-only support and compatibility RPM reuse.
.github/workflows/long-test.yml	Uses the shared dependency action.
.github/workflows/doc.yml	Uses the shared dependency action.
.github/workflows/coverage.yml	Uses the shared dependency action.
.github/workflows/codeql-analysis.yml	Uses the shared dependency action.
.github/workflows/ci.yml	Uses the shared dependency action.
.github/workflows/ci-verification.yml	Uses the shared dependency action.
.github/workflows/ci-al4.yml	Uses the shared Azure Linux 4 action.
.github/workflows/bencher.yml	Uses the shared dependency action.
.github/workflows/bencher-ab.yml	Uses the shared dependency action.
.github/azure-linux-4-cache-version	Versions the Azure Linux 4 cache.
.github/azure-linux-3-snapshot	Pins the Azure Linux 3 snapshot.
.github/actions/install-ci-dependencies/action.yml	Configures, restores, installs, and saves package caches.

Suppressed comments (4)

scripts/setup-ci-al4.sh:61

• This function is called from retry as the condition of an if, so Bash's errexit is disabled inside it. If this multi-package install fails, execution continues into the compatibility-RPM install; if that later command succeeds, the function reports success and the composite action saves an incomplete exact cache. Propagate the package-install failure before continuing.

    dnf "${DNF_OPTIONS[@]}" install  \

scripts/setup-ci-al4.sh:134

• Because this function is evaluated by retry inside an if, a failed DNF install does not trigger errexit. If Node.js from the base image is already at a supported version, the subsequent version check can succeed and the new cache step can persist a cache that never installed nodejs-npm. Return immediately when this DNF command fails.

        return 1

scripts/setup-ci-al4.sh:154

• This standalone DNF command is followed by the pip installation, and retry invokes the function in an if condition, so a DNF failure can be masked when pip succeeds. On a cold run the composite action would then save an incomplete package cache, causing later exact-cache/offline runs to fail. Propagate the DNF status before running pip.

        return 1

scripts/setup-ci-al4.sh:98

• This install is inside the outer retry, but a failed validation/install leaves the downloaded RPM at $libbacktrace_path. Subsequent retries therefore skip curl and reuse the same corrupt or truncated file, turning a transient download problem into three guaranteed failures. Remove or revalidate the cached file when the local install fails so the retry can redownload it.

    if [[ ! -f "$libbacktrace_path" ]]; then

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.