Google launches native PyTorch inference engines for vLLM and SGLang on TPU
In short: Google announced new vLLM and SGLang inference engines built directly on PyTorch for TPU, developed with Inferact and RadixArk respectively. The engines run on a new TorchTPU layer that makes TPUs a native PyTorch device, replacing the previous JAX-based backend approach. Google reports a public benchmark result of 2.01 cents per million tokens in high-concurrency configurations via the SemiAnalysis InferenceX project, plus support for agentic workloads, long context, disaggregated serving and multimodal models.
This summary was generated automatically by AI from Google Developers's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1New native PyTorch backends for vLLM and SGLang on TPU via TorchTPU layer
- 2Support for prefix caching, tool calling, structured output, speculative decoding, disaggregated prefill/decode serving
- 3New TPU Sync library for moving KV cache between hosts without extra staging copies
- 4Lowest reported cost of 2.01 cents per million tokens in public InferenceX benchmarks
- 5700+ tokens/sec per user demonstrated for 2.8T-parameter Kimi K3 model on Ironwood TPU
- 6Support for popular open model families: Qwen, DeepSeek, GLM, Kimi, GPT-OSS, Gemma4
- 7Existing JAX-based vLLM-Jax and SGLang-JAX projects remain maintained
| Parameter | Before | Now |
|---|---|---|
| Serving framework | JAX-based vLLM/SGLang backends on TPU | Native PyTorch backends (TorchTPU) for vLLM and SGLang on TPU |
| Cost per million tokens | Not specified | 2.01 cents at high-concurrency configurations (lowest reported in SemiAnalysis InferenceX benchmark) |
| Agentic/long-context support | Not specified | Automatic prefix caching, KV cache offloading via TPU Sync, tool calling, structured output, disaggregated prefill/decod |
| Tokens per second (single user) | Not specified | 700+ tokens/sec for 2.8T-parameter Kimi K3 model on Ironwood TPU via Pallas megakernel |
Why it matters
This is an infrastructure change for anyone running large-scale LLM inference on Google TPUs using open-source serving stacks, lowering the barrier for teams already using PyTorch to deploy models on TPU hardware and potentially cutting inference costs for high-concurrency workloads, including agentic and multi-turn use cases.
What it means for AI agents and contact centers
This news is about low-level LLM inference infrastructure on Google TPU hardware and is not directly relevant unless hosting your own large language models for voice agents on TPU infrastructure is under consideration; it does not affect STT/TTS pipelines or telephony integrations directly.
Sources
- Google DevelopersOfficialPrimary sourceOriginal article →„Announcing New vLLM and SGLang Inference Engines on TPU“20 Oct 2026, 03:00
- Published by source
- 20 Oct 2026, 03:00
- Found by our system
- 20 Oct 2026, 18:15
- Summary generated
- 20 Oct 2026, 18:17
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy