flashinfer

Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.

SGLang Kernel Wheel Index

NVFP4 inference on Blackwell GeForce (RTX 5090/5080/5070 Ti/RTX PRO 6000) — SM120 patches for vLLM + FlashInfer + CUTLASS. 175 tok/s on Qwen3.6-35B MoE.

Agent-assisted and full-agent reproducibility package for MLSys 2026 FlashInfer AI Kernel Generation Contest submissions: kernels, agent workflows, skills, configs, writeup, benchmark artifacts, and full optimization records.

Pure-Rust LLM runtime: one binary serves (OpenAI-compatible), runs local agents, and distills models on their own rollouts — on Apple Silicon and NVIDIA. No Python on the hot path.

Reproducible SGLang recipe + public prebuilt image (ghcr.io) for DeepSeek-V4-Flash-0731 on 4x RTX PRO 6000 Blackwell (SM120): TP4/DP4/EP4, 1M ctx, benchmarks, and the DSPARK draft-depth corruption boundary

Production runbook for Qwen3.5-122B hybrid INT4+FP8 on NVIDIA DGX Spark GB10 — optimization stack, PD firmware wedge diagnosis, bench results

Production-oriented Qwen3.6-35B-A3B-NVFP4-Fast vLLM deployment for NVIDIA DGX Spark / GB10

🚀 Accelerate attention mechanisms with FlashMLA, featuring optimized kernels for DeepSeek models, enhancing performance through sparse and dense attention.

DeepSeek-V4-Flash-DSpark on 2× DGX/ASUS Spark (GB10, SM120) using the stock spark-vllm container - recipes, a faster SM120 topk fix, and one-command run scripts.

Production-grade fine-tuning & LoRA toolkit for Chatterbox-Flash zero-shot TTS models. Combines parallel block diffusion and FlashInfer acceleration with smart placeholder vocabulary extension supporting languages. Features offline feature preprocessing, Silero VAD silence trimming, zero-padding leakage prevention, and fast voice cloning for custom

Single-GPU LLM decode research prototype: paged KV cache, Triton attention, CUDA append, scheduling, shared prefixes, and multi-layer transactions.

Dockerfile for installing vllm on newer 50 series cards

⚡ Optimize attention mechanisms with FlashMLA, a library of advanced sparse and dense kernels for DeepSeek models, improving performance and efficiency.

Correctness-first microbenchmarks for LLM attention and sampling kernels.

面向 RTX 5060 Blackwell SM120 的离线 LLM 推理引擎：Paged Attention、KV Cache、连续批处理、CUDA Graph 与算子融合

Run DeepSeek V4 Flash 0731 with EXL3 and DSpark speculative decoding on a single NVIDIA DGX Spark (GB10, 128GB).

vLLM plugin: per-layer attention backend selection for Gemma 4 heterogeneous head dims (drag-and-drop backport of PR #38891), with a text-only fast path + MTP-safe head=512 Triton pin.

Evidence-driven single-GPU LLM inference systems research with vLLM, FlashInfer, Triton, CUDA, L20-first measurements, and A100 controls.

Reproducible TP=2 deployment, container build, and experiment report for DeepSeek V4 Flash on two DGX Sparks