Dockerfile - Add cuda13.3.dockerfile on pytorch:26.07-py3

[gusui-msft]

Description

Adds a cuda13.3 image built on nvcr.io/nvidia/pytorch:26.07-py3, replacing cuda13.0 (25.08-py3) in the CI matrix.

Motivation is security. A Trivy 0.72.0 scan of main-cuda13.0 (2026-07-15) reports 58 Critical occurrences from exactly two sources. This PR closes both:

Source	Occurrences	Fixed by
NVIDIA base image (25.08-py3)	18	FROM nvcr.io/nvidia/pytorch:26.07-py3
Docker static bundle we install	40	DOCKER_VERSION=29.7.2
Total	58

Both bumps land together on purpose. Following the one-Dockerfile-per-CUDA-version convention this is a new file rather than an edit to cuda13.0.dockerfile, and a new file should not be born carrying a known-vulnerable Docker bundle. One CUDA version then maps to exactly one image that gets built, benchmarked, and scanned once. Supersedes #845, which is closed.

Major Revision

• Add dockerfile/cuda13.3.dockerfile based on nvcr.io/nvidia/pytorch:26.07-py3.
• Set DOCKER_VERSION=29.7.2 instead of inheriting 20.10.8 from cuda13.0.dockerfile.
• Point the docker-build and docker-merge matrices at cuda13.3 instead of cuda13.0.
• Add a CUDA_VER >= 13.3 branch to third_party/Makefile selecting CUTLASS v4.7.0, whose README states "Optimal code generation with CUDA toolkit versions 13.3". The existing >= 12.9 branch would otherwise still pick v4.1.0, which predates CUDA 13.3.

Minor Revision

• HPCX_VERSION v2.24.1 -> v2.50 and UCX_VERSION 1.18.0 -> 1.21.0, both matching the base image. Neither is cosmetic: the HPC-X block does rm -rf /opt/hpcx, and LD_LIBRARY_PATH puts /usr/local/lib ahead of the base libraries, so a lower version here would replace part of the stack PyTorch was built against with an older one. The UCX URL is also corrected from the -rc1 tag to the final release tag.
• Removed the setuptools pin entirely. cuda13.0 pins 78.1.0, which against this base is both a downgrade and CVE-2025-47273 (HIGH). Re-pinning to 81.0.0 was the obvious fix, but the base already ships exactly 81.0.0, so the line is a no-op that would silently downgrade a future base image. setup.py's setuptools>=45, <66 guard only runs on Python < 3.11 and this image is Python 3.12, so nothing else depends on the pin.
• Version comment header rewritten to the real 26.07-py3 components. Every value is read from the image config blob rather than release notes. The Docker line reads full static bundle, because the RUN unpacks all eight binaries, not only the client. The MOFED field now names the version it installs instead of being left blank.
• Replaced the stale comment above cuda_cutlass in third_party/Makefile. It claimed 12.9+ builds v3.9 while the recipe already cloned v4.1.0. The replacement states the actual selection rule: the tag tracks CUDA toolkit codegen support. It is deliberately not phrased in terms of SM archs, because the 13.3 and 12.9 branches set the same ARCHS.

---

1. Base image findings cleared, 18 occurrences

These are inherited and cannot be fixed in our own layers.

Package	Occurrences	In 25.08-py3	Required fix	26.07-py3 ships
linux-libc-dev	16	6.8.0-71.71	6.8.0-134.134	6.8.0-136.136
jupyter_server	1	2.16.0	2.20.0	2.20.0
Nsight efa_metrics/nic_sampler	1	Go v1.23.4	1.26.0-rc.3	go1.26.1

Each version was read directly out of the 26.07-py3 registry layers, not taken from release notes. The upgrade additionally brings wheel 0.45.1 -> 0.46.3 (CVE-2026-24049, HIGH) and pip 25.2 -> 26.1.2 (4 Medium, 1 Low).

2. Docker bundle findings cleared, 40 occurrences

cuda13.0.dockerfile pins the Docker static distribution at 20.10.8, built with Go 1.16.6. Trivy reports 40 Critical occurrences against the eight binaries the RUN unpacks into /usr/local/bin/. These are compiled-in Go dependencies, not installed packages, so apt and pip cannot touch them; the bundle itself has to be replaced.

Docker 29.7.2 (released 2026-08-05) is built with Go 1.26.5 and carries gRPC v1.82.1:

CVE	Component	Occurrences	Required fix	29.7.2 ships
CVE-2025-68121	Go stdlib	8	1.24.13, 1.25.7, 1.26.0-rc.3	go1.26.5
CVE-2022-23806	Go stdlib	8	1.16.14, 1.17.7	go1.26.5
CVE-2023-24538	Go stdlib	8	1.19.8, 1.20.3	go1.26.5
CVE-2023-24540	Go stdlib	8	1.19.9, 1.20.4	go1.26.5
CVE-2024-24790	Go stdlib	8	1.21.11, 1.22.4	go1.26.5

Verified before opening, carried over from #845:

• x86_64/docker-29.7.2.tgz and aarch64/docker-29.7.2.tgz both return 200. ${TARGETARCH_HW} already resolves correctly for both.
• Every binary in the bundle reports go1.26.5, on both architectures.
• dockerd embeds google.golang.org/grpc v1.82.1.
• The extracted client is statically linked and reports Docker version 29.7.2, build a7dcaa6.

Host compatibility. SuperBench only uses the Docker client against the host daemon socket (monitor.py, docker_base.py, runner.py, system_info.py, and deploy.yaml, which mounts /var/run/docker.sock). There is no reference to dockerd, containerd, runc, ctr, or docker-proxy anywhere in the tree. Docker 29.3.0 lowered the minimum daemon API version from v1.44 back to v1.40 (Docker 19.03), so the newer client still talks to older host daemons. docker login --username/--password and every subcommand we invoke are still present in 29.7.2. One note for reviewers: the v1 containerd-shim binary no longer ships in the 29.x bundle, and nothing here references it.

3. What changed in the base image

Unchanged, so no migration work needed: Python 3.12, Ubuntu 24.04, CUDA_HOME, _CUDA_COMPAT_PATH.

Both columns are read from the respective image config blobs.

Component	25.08-py3	26.07-py3
CUDA	13.0.0.044	13.3.1.008
PyTorch	2.8.0a0+34c6371	2.13.0a0+9186a08
cuDNN	9.12.0.46	9.24.0.43
cuBLAS	13.0.0.19	13.6.0.2
NCCL	2.27.7	2.30.7
TransformerEngine	2.5	2.17
OpenMPI	4.1.7	5.0.10
HPC-X	2.24	2.50
UCX	1.19.0	1.21.0
RDMA-core	56.0	63.0
Nsight Systems	2025.4.1.136	2026.3.1.117

4. On the two automated review comments

Both are real, and both are pre-existing repository-wide patterns rather than regressions introduced here. Counted across the 14 Dockerfiles on main:

Pattern	Files affected
Docker static bundle fetched over HTTPS with no checksum or pinned digest	14 of 14
PermitRootLogin yes and PermitUserEnvironment yes	14 of 14

No Dockerfile in the tree performs any sha256 verification today. I would rather not have cuda13.3 diverge from the other thirteen inside a PR whose claim is a specific, measured CVE closure:

• Checksum verification is worth doing as one sweep over dockerfile/*.dockerfile, so every image gains it at once and the pattern stays uniform. Happy to open that PR.
• The SSH settings need a decision from the PerfGate side on whether root login and user-environment injection are actually required, since tightening them could break existing deployments. This PR should not decide that unilaterally.

If reviewers would rather see either one fixed here, say so and I will fold it in.

Artifact availability

Checked 2026-08-18; HPC-X re-checked 2026-08-31.

• hpcx-v2.50-gcc-doca_ofed-ubuntu24.04-cuda13-{x86_64,aarch64}.tbz: both 200
• ucx-1.21.0.tar.gz at tag v1.21.0: 200
• MLNX_OFED_LINUX-24.10-1.1.4.0-ubuntu24.04-{x86_64,aarch64}.tgz: both 200
• NVIDIA/cuda-samples tag v13.3 exists, so the unconditional clone in cuda_bandwidthTest will not break
• NVIDIA/cutlass tag v4.7.0 exists

Draft, because this is unvalidated

This replaces CUDA, cuDNN, NCCL, PyTorch, HPC-X, UCX, and Docker at once, so it cannot be accepted on a scan result alone.

• Builds on arm64
• Builds on amd64
• docker --version inside the built image reports 29.7.2
• Representative benchmark run on both architectures
• PerfGate thresholds re-baselined for affected scenarios
• Rescan the pushed digests per architecture, confirm 0 Critical

Risks, in the order I expect them to bite:

1. PyTorch 2.8.0a0 -> 2.13.0a0, five minor releases. Affects megatron_lm, megatron_deepspeed, Apex, and TransformerEngine. There is no torch.__version__ guard anywhere in superbench/, so pip will not surface a break; it has to be found by building and running.
2. OpenMPI 4.1 -> 5.0. python3-mpi4py comes from apt and links the distro MPI, while nccl-tests and perftest build against MPI_HOME=/usr/local/mpi.
3. CUTLASS v4.7.0 has not been built here against CUDA 13.3 yet.
4. Performance re-baselining, since the whole optimized stack moves together.
5. MLNX_OFED 24.10-1.1.4.0 user-space now sits on top of rdma-core 63.0 rather than 56.0. Pre-existing pattern, wider gap.
6. rm -rf /opt/hpcx discards NVIDIA's own HPC-X wiring. The base symlinks /usr/local/mpi -> /opt/hpcx/ompi5 and /usr/local/ucx -> /opt/hpcx/ucx, and patches openmpi-mca-params.conf (hwloc_base_binding_policy = none, btl = self commented out). The symlinks survive, because the HPC-X 2.50 tarball provides both ompi5 and ucx, but the MCA tuning does not. Pre-existing behaviour, unchanged here, and worth a follow-up.

Docker is deliberately low on that list: the client-side surface is small and the compatibility floor was checked above. It is bundled here because splitting it out produces a cuda13.0 image variant that CI would no longer build once this PR lands.

Questions for reviewers

• Should dockerfile/cuda13.0.dockerfile be deleted in this PR? CI/CD - Clean up image builds, remove cuda 12.8/12.4/12.2 and add cuda13.0 merge #819 kept superseded Dockerfiles and only changed the matrix, so this PR follows that precedent and leaves the file in place. If it stays it keeps Docker 20.10.8, which is intentional now that CI will not build it.
• Should the HPC-X and UCX install blocks be dropped entirely rather than bumped, given the base already ships HPC-X 2.50 and UCX 1.21.0? Not in this PR, because the reason to keep them turns out to be path rather than version. HPC-X does ship a multi-threaded UCX at /opt/hpcx/ucx/mt, which is what our configure-release-mt build exists to provide, but it only reaches LD_LIBRARY_PATH through hpcx_load, sourced from /etc/bash.bashrc and /etc/profile.d/. A benchmark launched in a non-interactive shell would lose --mca pml ucx. Dropping the source build therefore means moving /opt/hpcx/ucx/mt/lib into the image ENV and re-validating IB, which deserves its own PR and a multi-node run.
• cuda12.9, cuda11.1.1 and rocm6.3.x stay in the matrix at Docker 20.10.8/27.5.1, roughly 40 and 8 Critical occurrences respectively. Each is a one-line change. Should that be one follow-up PR, or folded in here?

[Copilot]

Pull request overview

This PR adds a new CUDA 13.3 container image definition based on nvcr.io/nvidia/pytorch:26.07-py3, updates the GitHub Actions image build/merge matrices to build/publish cuda13.3 instead of cuda13.0, and adjusts third-party build logic to select a newer CUTLASS for CUDA 13.3+.

Changes:

• Add dockerfile/cuda13.3.dockerfile (new base image + updated HPC-X/UCX/Docker client handling).
• Update .github/workflows/build-image.yml matrix entries to build/merge/publish cuda13.3 tags.
• Update third_party/Makefile CUTLASS selection to use v4.7.0 when CUDA_VER >= 13.3.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
third_party/Makefile	Adds a CUDA 13.3+ conditional branch to select CUTLASS v4.7.0.
dockerfile/cuda13.3.dockerfile	Introduces the CUDA 13.3 image based on nvcr.io/nvidia/pytorch:26.07-py3 with updated dependency versions.
.github/workflows/build-image.yml	Switches CI build/merge matrices from cuda13.0 to cuda13.3 tags/dockerfile.

Suppressed comments (1)

third_party/Makefile:47

• git clone ... && cd cutlass is misleading/no-op here: each recipe line runs in its own shell, and the later build uses -S ./cutlass anyway, so the cd cutlass has no effect on subsequent steps. Consider removing the trailing && cd cutlass to avoid implying later commands run from inside the repo.

	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (2)

third_party/Makefile:48

• git clone ... && cd cutlass is misleading here: Make runs each recipe line in its own shell, so the cd cutlass does not affect subsequent commands (which already reference ./cutlass). Consider dropping the && cd cutlass suffix to avoid implying that later steps run inside the repo directory.

ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)

dockerfile/cuda13.3.dockerfile:162

• This Dockerfile changes the established pattern of pinning setuptools to an exact version (e.g., cuda13.0.dockerfile and cuda12.9.dockerfile use setuptools==...). Using a lower-bound specifier (setuptools>=...) makes image builds non-reproducible because the resolved version can change over time. Consider pinning to a specific known-good version (e.g., the base image’s current setuptools) and bump it deliberately when needed.

RUN python3 -m pip install --upgrade "setuptools>=78.1.1" && \
    python3 -m pip install --no-cache-dir .[nvworker] && \

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.02%. Comparing base (28fe025) to head (01ca815).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #846   +/-   ##
=======================================
  Coverage   86.02%   86.02%           
=======================================
  Files         103      103           
  Lines        7950     7950           
=======================================
  Hits         6839     6839           
  Misses       1111     1111           

Flag	Coverage Δ
cpu-python3.10-unit-test	70.88% <ø> (ø)
cpu-python3.12-unit-test	70.88% <ø> (ø)
cpu-python3.7-unit-test	70.31% <ø> (ø)
cuda-unit-test	83.95% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

third_party/Makefile:48

• The new CUDA_VER>=13.3 branch duplicates the same ARCHS/rm/clone recipe used for the >=12.9 branch, which increases maintenance cost and makes future tag/arch updates easy to miss. Consider setting CUTLASS_TAG (and ARCHS) inside the conditional branches, then running the shared rm/clone logic once after the conditional (or factoring a small helper target).

# The CUTLASS tag is picked per CUDA_VER because each release only supports the SM archs of its own CUDA generation.
cuda_cutlass:
ifeq ($(shell echo $(CUDA_VER)">=13.3" | bc -l), 1)
	$(eval ARCHS := "100;103")
	if [ -d cutlass ]; then rm -rf cutlass; fi
	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass
else ifeq ($(shell echo $(CUDA_VER)">=12.9" | bc -l), 1)

[gusui-msft]

@microsoft-github-policy-service agree company="microsoft"

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

third_party/Makefile:47

• The && cd cutlass at the end of this recipe line is misleading/no-op: each Make recipe line runs in its own shell (no .ONESHELL here), and the later build already uses -S ./cutlass. Dropping the cd avoids implying subsequent lines run inside the repo directory.

	git clone --branch v4.7.0 --depth 1 https://github.com/NVIDIA/cutlass.git && cd cutlass

dockerfile/cuda13.3.dockerfile:138

• This UCX build leaves the downloaded tarball and extracted source tree under /tmp, which increases final image size and can make layer caching less effective. Consider removing the UCX tarball/source directory in the same RUN layer after make install.

    tar xzf ucx-${UCX_VERSION}.tar.gz && \
    cd ucx-${UCX_VERSION} && \
    ./contrib/configure-release-mt --prefix=/usr/local && \
    make -j ${NUM_MAKE_JOBS} && \
    make install

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/workflows/build-image.yml:34

• These entries remove the last CUDA 13.0 build and make the unvalidated CUDA 13.3 image the only CUDA 13 artifact, while the PR's own checklist still leaves both architecture builds, representative runs, and rescans unchecked. If the simultaneous CUDA/PyTorch/OpenMPI/UCX changes fail, this replaces the known-good CI artifact rather than introducing a canary. Please keep cuda13.0 in the matrix or gate this replacement on successful validation of both architectures and representative scans/runs.

        - name: cuda13.3-arm64
          dockerfile: cuda13.3
          tags: superbench/main:cuda13.3-arm64

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.