[Bug] init_quantized_weights / --low_memory_mode exports numerically broken NVFP4 (quantizes meta tensors before weights load)

[spped2000]

Summary

init_quantized_weights (public API, exported in modelopt/torch/quantization/plugins/accelerate.py:__all__, and the mechanism behind hf_ptq.py --low_memory_mode) produces an NVFP4 checkpoint that is silently wrong: half-sized weight_scale tensors, dequantization cosine similarity 0.756 against the source BF16 (healthy NVFP4 is >0.99), and a checkpoint vLLM refuses to load.

There is no error at export time. The run completes and prints Quantized model exported to: ....

Root cause

In patched_from_pretrained (accelerate.py, current main a21173a):

with init_empty_weights():
    model = cls.from_config(config, dtype=torch_dtype)   # L230 — meta tensors

mtq.quantize(model, quant_cfg)                            # L232 — quantizes META
mtq.compress(model, config=mtq.CompressConfig(...))       # L233 — packs META
_device_map = get_model_device_map(model, gpu_mem_percentage)

return load_checkpoint_and_dispatch(                      # L236 — real weights load
    model, checkpoint=..., device_map=_device_map, *args, **kwargs,
)

Quantization and compression run against init_empty_weights() meta tensors, i.e. before any real weight values exist. Calibration/absmax has nothing to measure, and mtq.compress has already replaced each Linear.weight with a packed uint8 buffer — so load_checkpoint_and_dispatch then loads real BF16 values into buffers whose shape and scale metadata describe the packed view.

Observable consequence on a Qwen2 model: k_proj.weight_scale is exported as [128, 28] where NVFP4 block-16 over 896 in-features requires [128, 56] — the scale grid was computed over 448 packed bytes instead of 896 logical values. vLLM aborts at load:

RuntimeError: start (0) + length (304) exceeds dimension size (152).

Dequantizing the export by hand and comparing to the BF16 source gives cosine 0.756 (lo/hi nibble order; 0.015 with nibbles swapped, 0.758 assuming block-16-truncated) — the numbers are damaged at quantization time, not at load time, so no loader-side fix helps.

Reproduction (one command, 0.5B model — no large hardware needed)

# inside nvcr.io/nvidia/vllm:26.05.post1-py3, nvidia-modelopt[hf]==0.43.0
python hf_ptq.py \
  --pyt_ckpt_path ./Qwen2.5-0.5B-Instruct \   # local dir; repo ids are rejected on this path
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 1 \
  --dataset ./calib.jsonl --attn_implementation sdpa \
  --skip_generate --low_memory_mode \
  --export_path ./out-NVFP4

Then check the export: weight_scale second dimension is half of in_features/16, and ~168 runtime-state tensors named *.weight_quantizer._double_scale leak into the state dict (vLLM: no module or parameter named ... weight_quantizer).

Three smaller defects sit on the same path and block reaching the export at all; I have opened separate small PRs for the first two:

1. patched_from_pretrained forwards **kwargs (including attn_implementation) into load_checkpoint_and_dispatch(), which does not accept it → TypeError.
2. No model.tie_weights() before quantization, so tied lm_head.weight stays on meta and dispatch_model's .to() raises Cannot copy out of meta tensor (accelerate's docs require tie_weights() before load_checkpoint_and_dispatch).
3. The path accepts only local checkpoint directories, not HF repo ids (ValueError from load_checkpoint_in_model) — worth documenting if intentional.

Environment

modelopt 0.43.0, NGC nvcr.io/nvidia/vllm:26.05.post1-py3, NVIDIA GB10 (SM121, aarch64), 121 GB unified memory. Verified the same code path is unchanged on current main (a21173a).

Impact / workaround

This is the documented OOM fallback for models larger than device memory, so it is exactly the path users reach for when they have no alternative. For a 72B model (145 GB BF16 on a 121 GB box) I ended up going through llm-compressor with accelerate disk offload instead, which produced a correct checkpoint. Full write-up with the dequantization check and all artifacts: https://github.com/spped2000/thaillm-nvfp4-dgx-spark/blob/main/results/quant72b/FINDING_modelopt_lowmem_broken.md

Suggested direction: load real weights before mtq.quantize/mtq.compress (e.g. dispatch first with an offload folder, then quantize sequentially), or fail loudly when quantization is attempted on meta tensors.

---

Disclosure: investigated with assistance from Claude (Anthropic); every number above comes from real runs on the hardware listed.