perf(gguf): sm_120 dispatch thresholds + upstream int8-MMA MMQ port (Ornith)

[lucaspirola]

What

Two related optimizations for Ornith-1.5-35B-Q4_K_M on RTX 5080 (sm_120,
Blackwell), on top of the existing Ornith INT4 262K-context serving path:

1. Arch-aware GGUF dispatch thresholds — the dense/MoE dequant-vs-MMQ
   crossover points measured on Ada don't hold on sm_120; make them
   architecture-gated instead of hardcoded.
2. int8-tensor-core MMQ port — the vendored GGUF kernels (llama.cpp
   b2899 via vLLM/sgl-kernel) are DP4A-only with zero tensor-core use.
   Upstream llama.cpp master has since rewritten Q4_K/Q6_K MMQ around
   int8 tensor-core MMA tiles (turing_mma, sm_75+). This PR vendors
   that upstream kernel verbatim (kernel/csrc/gguf_mmq/, llama.cpp
   eab8ee41) and wires it in for both dense projections and grouped
   MoE experts, gated to sm_120 for now.

FREETOKEN_GGUF_DISABLE_MMA=1 forces the old DP4A/dequant path (debugging,
or a toolchain that can't build the extension) — used below as the control
in every comparison.

Gains

Synthetic kernel benchmarks (real Ornith-1.5-35B-Q4_K_M tensors, sm_120, median of CUDA-event timings):

shape	rows/tokens	int8-MMA MMQ	DP4A MMQ (old)	dequant+cuBLAS	speedup vs DP4A
Q4_K attn_q [8192x2048] (dense)	8192 rows	1.79 ms	22.9 ms	2.40 ms	12.8x
Q6_K lm_head [248320x2048] (dense)	2048 tokens	17.5 ms	262 ms	19.2 ms	15.0x
Q4_K gate_up (MoE, E=256 top-8)	8192 tokens	4.16 ms	23.2 ms	n/a	5.6x
Q6_K down (MoE, E=256 top-8)	8192 tokens	4.90 ms	15.3 ms	n/a	3.1x

MMA beats dequant+cuBLAS too (the prior best option at large batch), not just
the DP4A kernel it replaces — e.g. attn_q 1.79ms vs 2.40ms, lm_head 17.5ms vs
19.2ms.

Live end-to-end A/B, full 262K-context serve config, same box, same prompt, minutes apart:

Config: --attention-backend triton --kv-cache-dtype q4_0 --num-tokens 262144 --kv-reserve-tokens 262144 --max-seq-len-override 262144 --max-running-requests 1 --moe-backend offload --moe-cache-auto --max-prefill-length 8192. Prompt: 28K words of seeded non-repetitive text
(~50K tokens, 6+ chunked prefill batches) with three distinct needles at
10/50/90% depth, greedy decode.

	int8-MMA (this PR)	DP4A/dequant (FREETOKEN_GGUF_DISABLE_MMA=1)
wall time	13.88 s	21.94 s
effective prefill rate	~4,700 tok/s	~2,700 tok/s
needles recovered	3/3 exact	3/3 exact
tracebacks	0	0

~1.75x prefill, ~1.58x end-to-end wall time, identical answers on the
same hardware and prompt.

Dispatch thresholds (sm_120 vs the existing Ada-tuned defaults, unchanged
off sm_120):

• dense dequant-vs-MMQ crossover: 24 rows (was 32) — Q4_K attention shapes
  cross at 24 (0.0645ms dequant vs 0.0778ms MMQ); 16 would regress the Q6_K
  lm_head, where MMQ still wins at 16.
• MoE grouped-MMQ-vs-vec crossover: 16 tokens (was 32) — 0.314ms MMQ vs
  0.324ms vec at 16 tokens, widening to 0.382 vs 0.475 at 24.
• Live-reverified on real Ornith tensors: 24 rows now 0.069ms via dequant vs
  0.081ms with the old threshold.

How

• python/freetoken/kernel/csrc/gguf_mmq/ — llama.cpp master's CUDA MMQ
  vendored verbatim (mmq/mma/load-tiles/vec-dot/configs/quantize/mmid + the
  ggml headers it needs). mmq_ext.cu is the only hand-written file: backend
  shims (device info, a torch-allocator-backed pool, error/abort plumbing)
  and the torch bindings for the dense (ggml_mul_mat_a8_mma) and grouped-MoE
  (ggml_moe_a8_mma) entry points. Only Q4_K/Q6_K mul_mat_q cases are
  instantiated to keep JIT compile time down.
• layers/gguf.py — dequant_gemm_min_rows(cc) (arch-gated threshold),
  _use_mma_mmq dispatch gate (sm_120 + supported type + successful JIT
  build), wired into fused_mul_mat_gguf.
• moe/fused_gguf.py — mmq_min_tokens(cc) (arch-gated threshold),
  _use_mma_moe gate (adds a block-alignment check on the padded expert
  slot stride), wired into _moe_matmul for the 320–16384 token band
  (below 320, DP4A wins — per-expert tiles waste work at small batches;
  MMVQ still owns decode).
• kernel/gguf.py — _mma_module() lazy JIT loader, mma_mmq_supported().

Testing

• Numerics: MMA output verified against the dequantized reference on real
  Ornith tensors (rel error <= 0.013, on par with the existing DP4A kernel)
  and against gguf-py on random-but-safe packed bytes
  (tests/kernels/test_gguf_mma.py). MoE broadcast (gate/up) and gather
  (down) forms verified against the per-expert dense reference and the
  existing vec kernel, end-to-end through fused_experts_gguf (rel ~1e-3).
• tests/kernels/test_gguf_dispatch.py — pure threshold-function tests for
  both archs, plus CUDA dispatch-branch tests with faked device capability.
• Kernel + benchmark suites: 336 passed, 1 skipped.
• Full non-slow suite (-m "not slow", excluding e2e/server): failure set
  unchanged from clean main (same 7 pre-existing failures).
• Live 262K-context A/B above, including a hostile (non-repetitive,
  multi-needle) prompt specifically to catch any tile/row-range corruption a
  compressible-filler prompt could hide.
• ruff check clean on all changed files.

Falls back cleanly to the existing DP4A/dequant path off sm_120, on a build
failure, or with FREETOKEN_GGUF_DISABLE_MMA=1; existing sm_89/Ada behavior
is unchanged (verified via the A/B, not just by inspection).