NVIDIA and Princeton Cut Decoding Bottlenecks in Hybrid SSM Models
In short: NVIDIA, working with Princeton's Dao AI Lab, built ReplaySSM, a new way to store state in hybrid models that mix state-space (SSM) layers with attention layers. Instead of rewriting the entire state on every token, the method keeps a periodic checkpoint plus a short list of recent inputs, splitting the job of producing output from the job of updating memory. The result is roughly half the memory traffic, decode kernels running up to 1.80x faster at scale, and speculative decoding throughput more than doubling on 120B-class models, all while keeping accuracy close to full precision. The method already ships inside TensorRT-LLM, vLLM, and FlashInfer, and is running in production for GLM-5.3-Flash and Kimi-K3.
This summary was generated automatically by AI from NVIDIA Developer's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1State is stored as a periodic checkpoint plus a buffer of recent inputs, rather than being rewritten in full on every step
- 2State memory traffic drops by roughly half because only the checkpoint, not the full state, needs writing on most steps
- 3Decode kernels run 1.46x-1.80x faster at batch size 256 on Blackwell Ultra B300 GPUs; the full decode step is 1.20x-1.47x faster
- 4Speculative decoding no longer needs a state snapshot per draft token, giving 1.85x-2.10x higher throughput on 120B-class models at batch size 512 with matched acceptance lengths
- 5A high-precision checkpoint paired with a low-precision read copy keeps three of four tested models within about 2 points of FP32 accuracy on GPQA-Diamond
- 6Already integrated into TensorRT-LLM, vLLM, and FlashInfer, and running in production inside GLM-5.3-Flash and Kimi-K3
| Parameter | Before | Now |
|---|---|---|
| State memory traffic | Full state read and written every decode step | Checkpoint read each step, written only once per buffer flush — roughly half the traffic |
| Kernel speed (batch 256, B300) | Baseline vLLM kernel | 1.46x-1.80x faster; full decode step 1.20x-1.47x faster |
| Speculative decoding throughput (120B-class) | Baseline vLLM speculative decoding | 1.85x-2.10x higher throughput at batch size 512, matched acceptance lengths |
| Quantization accuracy | Single-precision state prone to rounding error accumulation | 3 of 4 models stay within ~2 points of FP32 accuracy on GPQA-Diamond using dual-precision checkpoint/read |
Why it matters
Hybrid models combining SSM and attention layers (like Nemotron 3, Qwen3.5, Kimi-K3, GLM-5.3-Flash) are increasingly used for long-context, efficient inference, but their SSM layers had become the real bottleneck at serving scale — this fix restores much of the speed advantage SSMs were originally built for, and makes speculative decoding actually useful for these architectures.
Sources
- NVIDIA DeveloperOfficialPrimary sourceOriginal article →„ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding“12 Oct 2026, 19:00
- Published by source
- 12 Oct 2026, 19:00
- Found by our system
- 12 Oct 2026, 19:10
- Summary generated
- 12 Oct 2026, 19:12
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy