feat(dstack-util): warn when the NVLink fabric never comes up

[kvinwang]

The failure this reports

A multi-GPU CVM boots. Every surface signal is green:

• nvidia-smi lists all GPUs
• nvidia-smi nvlink -s shows all 18 links Active at 53.125 GB/s — because ALI trains links in hardware regardless of fabric state
• Fabric Manager logs Successfully configured all the available GPUs and NVSwitches
• Containers start, CUDA runs, results are correct

And NVLink P2P is dead. Collectives fall back to PCIe. The workload is 10–100× slower, nothing errors, and the tenant pays full price for a fraction of the bandwidth.

Not hypothetical:

• B300 SXM6 forum thread — exactly this shape, unresolved
• NVIDIA/NVSentinel#883 — a node with State: Completed and Status mixing Success and Unknown Error across GPUs, undetected by health monitoring
• arXiv 2606.23969 §7.2, on dstack-driven B300: "The operational failure modes we hit — stale FM partition state surfacing as guest FLA remap validation errors — argue for fabric-state health checks as a scheduling precondition."

What it checks

At the point that already gates GPUs — after attestation passes, before the ready state is set.

field	requirement	why
state	NVML_GPU_FABRIC_STATE_COMPLETED	baseline
status	NVML_SUCCESS	the reason this is worth doing — Completed + failed status is a real observed condition, and NCCL logs state and never status, so it is invisible in every NCCL log while P2P capability is gone
cliqueId	identical across GPUs	two GPUs can both register successfully and still land in different partitions, unable to reach each other

Then nvmlDeviceGetP2PStatus for both directions of every pair. That is the call behind NCCL's own "P2P is disabled between NVLINK connected GPUs" warning (src/graph/paths.cc), and it is what catches a fabric that registered cleanly but still cannot move traffic — the B300 case above.

Advisory, not a gate

Every one of these failures is host-side: a partition that was never activated, a stale Fabric Manager, GPUs landing in different cliques. None of it is something the guest can fix or should refuse to boot over.

So it warns, and the message points where the problem actually is:

NVLink fabric not ready after 30s; NCCL will fall back to PCIe.
This is a host-side condition: check that the partition is active (`fmpm -l`)
and that /var/log/fabricmanager.log reports every GPU notified

It runs after attestation so a fabric problem can never be mistaken for a security one, and on a blocking thread since it polls.

Polled, not sampled once

Registration is asynchronous — the GPU probes Fabric Manager over NVLink inband once the driver initialises it — so a single read right after boot legitimately catches IN_PROGRESS. A check that fails intermittently is worse than no check.

The elapsed time is logged, so the 30 s bound can be tuned from real deployments instead of guessed at a second time.

Zero configuration for the common case

Skipped entirely when the driver reports no fabric (NVML_GPU_FABRIC_STATE_NOT_SUPPORTED), which covers single-GPU tenants and every host without NVSwitch. The driver decides, not a device table or a GPU count — same approach as the module-option generator in #1157.

Implementation note

nvml-wrapper 0.12 binds neither nvmlDeviceGetGpuFabricInfo nor nvmlDeviceGetP2PStatus, so both go through nvml-wrapper-sys using the handle and loaded library the safe wrapper already owns (Nvml::lib(), Device::handle()). This is the first raw NVML call in the tree; the unsafe blocks carry their justification.

Testing

cargo test -p dstack-util — 109 passed, including two new tests over the pure predicates:

• registered() accepts only Completed and SUCCESS together, rejecting the Completed + failed-status pairing that NCCL cannot see
• shared_clique() returns None when two successfully-registered GPUs report different cliques

clippy -D warnings and fmt --check clean.

Not verified on hardware. No GPU was available, so the NVML paths are unexercised. The one thing worth confirming before relying on this is that a guest actually sees fabric state when NVSwitches stay on the host (Blackwell MPT CC) — the mechanism says yes, since nvmlDeviceGetGpuFabricInfo is a per-GPU query and the shared-NVSwitch model requires guest P2P to work, but NVIDIA's integration guide never shows a guest-side example. One command in any multi-GPU CVM settles it:

nvidia-smi -q | grep -A5 -i '^ *Fabric'

If that section is absent or reports Not Supported on a multi-GPU tenant, this check will simply skip and the design needs to move host-side.

[Copilot]

🟡 Changes recommended

There are correctness/robustness issues in the new fabric polling path (early skip on partial NOT_SUPPORTED and silently ignored JoinError) that can hide failures or drop diagnostics.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds an advisory NVLink fabric health check to dstack-util’s GPU attestation/readying path to surface cases where GPUs appear healthy but NVLink P2P is effectively unavailable (causing NCCL to fall back to PCIe with no hard error).

Changes:

• Introduces a polling NVML-based fabric readiness check (fabric state/status + clique consistency + P2P status per GPU pair) and logs actionable warnings when the fabric never becomes usable.
• Hooks the advisory check into system setup after GPU attestation and before setting the GPU “ready” state (runs in spawn_blocking due to polling).
• Adds nvml-wrapper-sys as a workspace dependency to access raw NVML entry points not exposed by nvml-wrapper.

File summaries

File	Description
dstack/dstack-util/src/system_setup.rs	Implements fabric + P2P checks via raw NVML calls and adds unit tests for the pure predicates.
dstack/dstack-util/Cargo.toml	Adds nvml-wrapper-sys dependency for the util crate.
dstack/Cargo.toml	Adds workspace dependency pin for nvml-wrapper-sys.
dstack/Cargo.lock	Locks the new dependency in the workspace.

Review details

• Files reviewed: 3/4 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.