Research· 🧪 Worth Testing

AWS open-sources MoE kernel optimizations for vLLM, cuts latency and boosts throughput on H200

In short: AWS detailed three kernel-level optimizations for Mixture-of-Experts (MoE) model serving on Hopper GPUs, implemented in collaboration with the vLLM community and shipped in Amazon SageMaker AI's LMI container. The changes target the inefficiencies of fused MoE computation during low-batch decode, which previously left Tensor Cores underutilized. Combined, the optimizations deliver up to 10% higher throughput and up to 9% faster time-to-first-token (TTFT).

Source: AWS Machine Learning13 Oct 2026, 18:17AWS Original article ↗

This summary was generated automatically by AI from AWS Machine Learning's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.

What changed?

1Swap-AB transposition realigns matrix operands to match Hopper's WGMMA hardware layout, giving 2-4% end-to-end throughput gain on H200 for large MoE models; upstreamed to vLLM 0.24.0+
2Routing FP8 tensor-quantized MoE models to the Triton backend instead of default FlashInfer+CUTLASS gives a 10% end-to-end throughput improvement on H200, available in AWS LMI v28 container
3Eliminating unnecessary Triton JIT recompilation (via do_not_specialize annotations) for variable token counts improved TTFT by 9% on H200 at 8-way tensor parallelism; upstreamed to vLLM 0.22.0+

Why it matters

For teams running open-weight MoE LLMs in production inference pipelines, these changes reduce cost per token and latency without requiring code changes beyond upgrading vLLM/LMI container versions, which is directly relevant to real-time or near-real-time conversational AI workloads.

What it means for AI agents and contact centers

If any part of your stack runs self-hosted open-weight LLMs via vLLM (for example behind voice agents, call summarization, or tool-calling pipelines) on SageMaker or similar Hopper-based infrastructure, upgrading to the versions mentioned (vLLM 0.22.0+/0.24.0+, LMI v28) could lower latency and cost with minimal engineering effort; worth benchmarking TTFT and throughput before/after on your actual workload.

🧪 Worth Testing

Teams self-hosting MoE-based LLMs for low-latency conversational tasks can directly benefit from faster TTFT and higher throughput by simply upgrading vLLM/LMI container versions, making this a low-effort, measurable test.

Fused MoE kernel optimizations (Swap-AB, Triton FP8 routing, JIT recompilation fix)· New

MoE vLLM Inference Optimization SageMaker GPU Latency Throughput

Sources

AWS Machine LearningOfficialPrimary source
„Mixture-of-Experts kernel engineering on Amazon SageMaker AI“
13 Oct 2026, 18:17
Original article →

Published by source: 13 Oct 2026, 18:17
Found by our system: 13 Oct 2026, 18:41
Summary generated: 13 Oct 2026, 18:42

This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy