GGUF: serve DeepSeek-V4-Flash - #210
Conversation
Adds the deepseek4 GGUF adapter and registers the architecture. Verified end to end on the real 164GB antirez/deepseek-v4-gguf Q4KExperts checkpoint: 43 layers, 256 experts, 145 GiB of expert banks, generating correct text. Extracted from FlashML-org#131 so it can be reviewed on its own; the three adapter commits also appear there and I am happy to drop them from that branch, or rebase this onto it, whichever you prefer. It depends on nothing from FlashML-org#131 beyond the shared GGUF reader that is already on main. WHAT IT DOES parse_gguf_config rebuilds DeepseekV4Args from GGUF metadata alone. The safetensors path recovers it from inference/config.json, which ships beside the weights; a standalone .gguf has no such file. Nothing falls back to a dataclass default: a silently-defaulted hyperparameter yields a model that loads and generates confidently wrong text. The served layer count is DERIVED, not assumed. block_count does not relate to the MTP block consistently across architectures -- qwen35moe counts its NextN block inside it, deepseek4 does not -- so compress_ratios is the authority, and the schedule it implies (41 layers with a compressor, 21 with the lightning indexer) is checked against the tensor table on every load. That check caught an off-by-one immediately rather than leaving it to surface as degraded output after a 145 GiB load. Routed experts stream from the offload cache; the shared expert is an ordinary quantized Linear. THINGS THAT DO NOT BEHAVE AS THEIR NAMES SUGGEST Each of these would have been a silent mis-wire: attn_output_a is Q8_0 in the file, but attn.wo_a is a bare bf16 nn.Parameter, not a Linear -- no .weight, never packed. The compressor and indexer projections are F16, i.e. GGML_UNQUANTIZED, so GGUFLinear's uint8 buffer cannot hold them; they land dense. Indexer.wq_b is declared Linear(kind="fp8"), which allocates a .scale no GGUF tensor can fill. It is rebuilt as bf16 rather than populated. tid2eid is a routing index, not a weight: read as int32 and widened, never through a float dequant that would round token ids. deepseek_v4 is raw nn.Module and loads via load_state_dict over named_parameters(), unlike the BaseOP-based trees the other GGUF adapters target. FreeToken's GGUFLinear holds qweight as a plain attribute, invisible to that loader, and assigning a non-Module over a Module child raises. Hence GGUFLinearNN / GGUFEmbeddingNN, carrying packed bytes in a non-grad uint8 Parameter named .weight. RECONCILIATION 1199 model parameters, 1199 matched, 0 unmapped tensors, 0 shape mismatches, 0 unfilled -- checked both directions against the real file before any load was attempted. MEASURED Quadro RTX 6000 (Turing sm_75, 24GB), 204 GiB WSL RAM. Banks split 24 layers GPU-pinned / 19 OS-locked for CPU decode, ~18GB VRAM. Temperature 0: "The capital city of France is" -> " Paris. The capital city of England is London. The capital city of Spain" "The largest planet in our solar system is" -> " Jupiter. It is so big that more than 1,300 Earths" KNOWN LIMITATIONS CUDA graph capture crashes the worker on this configuration, so it runs with --cuda-graph-max-bs 0 and decode is slower than it should be. Not yet diagnosed; being worked separately. Two factual prompts are a smoke test, not an eval. They show the pipeline is wired correctly, not that quality is intact. Of the thirteen published unsloth DeepSeek-V4 variants, none load: eleven mix ggml types across layers (the offload slot pool is one allocation with one stride) and the two uniform ones are MXFP4, which has no entry in BLOCK_SHAPE and no vendored kernel. The antirez builds are uniform and do load.
- 'gguf' is a container tag, not a layout: two call sites tested it directly, so no GGUF checkpoint could reach the CPU executor and the WSL residency split never engaged. - gemm1_dot fell through to NVFP4 with null pointers on an unhandled format; now raises. - Unquantized types are raw bytes with no dequant kernel; reinterpreted. - DeepSeek-V4 now serves end to end (see PR FlashML-org#210).
|
Correction to the description: there is no CUDA graph bug. I reported one and I was wrong. Capture was crashing the worker on my test box, but the cause was a stale compiled Re-verified with capture enabled: Output is identical with graphs on: The real limitation is throughput, and it is worth stating precisely: about 2 tok/s on this configuration. That is not a defect in this PR, it is what the hardware forces. The banks are 145 GiB and WSL caps CUDA pinning near 40% of RAM (measured 81.78 GiB of 204), so the residency planner OS-locks 19 of 43 layers for CPU decode. The Q4_K CPU dot is a scalar reference kernel — deliberately, since it landed as correctness-first with SIMD deferred — so each token does 6 experts x 19 layers of scalar work. Prefill is worse: 104s TTFT for a 25-token prompt, because locked layers prefill through synchronous pageable copies. Two ways forward, and the second is better:
So the numbers in the original description stand for correctness and stand as a worst case for speed. I would not want anyone reading 2 tok/s as the architecture's ceiling — it is the cost of running a 145 GiB model on a box that can only pin 82 GiB of it. |
1 participant
Adds the
deepseek4GGUF adapter and registers the architecture. Verified end to end on the real 164GBantirez/deepseek-v4-ggufQ4KExperts checkpoint, generating correct text.Extracted from #131 so it can be reviewed on its own. The three adapter commits also appear there; happy to drop them from that branch or rebase this onto it, whichever you prefer. It depends on nothing from #131 beyond the GGUF reader already on main.
Reconciliation before any load
1199 model parameters, 1199 matched, 0 unmapped tensors, 0 shape mismatches, 0 unfilled, checked in both directions against the real file. That check is the difference between "loads" and "loads and emits fluent nonsense", which is the failure mode this architecture invites.
Four things that do not behave as their names suggest
Each would have been a silent mis-wire:
attn_output_aattn.wo_a, a bare bf16 nn.Parameter -- no.weight, never packed.weight;GGUFLinear's uint8 buffer cannot hold themIndexer.wq_bLinear(kind="fp8"), which allocates a.scalenothing can fill -- rebuilt as bf16tid2eidThe layer count is derived, not assumed
block_countdoes not relate to the MTP block consistently across architectures: qwen35moe counts its NextN block inside it, deepseek4 does not. Copying that rule silently dropped the last real layer.compress_ratiosis the authority instead, and the schedule it implies (41 compressor layers, 21 indexer layers) is cross-checked against the tensor table on every load. That check caught the off-by-one in seconds rather than after a 145 GiB load.Why this needed its own Linear
deepseek_v4is rawnn.Moduleand loads viaload_state_dictovernamed_parameters(), unlike the BaseOP-based trees the other GGUF adapters target. FreeToken'sGGUFLinearholdsqweightas a plain attribute, invisible to that loader, and assigning a non-Module over a Module child raises outright.GGUFLinearNN/GGUFEmbeddingNNcarry the packed bytes in a non-grad uint8Parameternamed.weight.Measured
Quadro RTX 6000 (Turing sm_75, 24GB), 204 GiB WSL RAM. 145 GiB of banks split 24 layers GPU-pinned / 19 OS-locked for CPU decode, ~18GB VRAM. Temperature 0:
Known limitations, stated plainly
CUDA graph capture crashes the worker on this configuration, so it runs with
--cuda-graph-max-bs 0and decode is slower than it should be. Not yet diagnosed; being worked separately.Two factual prompts are a smoke test, not an eval. They show the pipeline is wired correctly, not that quality is intact.
Most published checkpoints will not load. Of the thirteen unsloth DeepSeek-V4 variants, eleven mix ggml types across layers (the offload slot pool is one allocation with one stride) and the two uniform ones are MXFP4, which has no
BLOCK_SHAPEentry and no vendored kernel. Theantirezbuilds are uniform and do load. Worth knowing before anyone downloads 90GB.