AWS introduces push-based KV transfer to cut latency in disaggregated vLLM inference
In short: AWS released a push-based NIXL connector for vLLM that changes how key-value (KV) cache data moves between prefill and decode instances in disaggregated LLM serving. Instead of the decode instance pulling KV blocks after prefill finishes, the decode instance registers its target memory in advance and the prefill instance writes the KV data directly once ready, moving coordination work off the critical path. Benchmarked on a 1-prefill/1-decode Qwen3-32B deployment using EC2 P5en instances with EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) than the default pull mode, especially under high request rates.
This summary was generated automatically by AI from AWS Machine Learning's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1New NixlPushConnector added to vLLM as an alternative to the default pull-based NixlConnector
- 2Decode instance pre-registers destination memory blocks while prefill is still computing, removing a post-prefill round trip
- 3Prefill side runs KV transfer setup and NIXL WRITE on a background thread, off the latency-critical decode loop
- 4Benchmarked on Qwen3-32B, 1P1D topology, EC2 P5en with EFA: up to 3x TTFT improvement (~1,744ms to ~540ms at 8 QPS/16k input) and TPOT improvement from ~10.4ms to ~7ms at high load
- 5Feature was built and validated on AWS Trainium clusters, then contributed as an accelerator-agnostic feature that also works on GPUs
- 6Pull mode remains better for workloads needing load-aware routing or bidirectional transfer; push mode is recommended when low TTFT is the priority
Why it matters
Disaggregated prefill/decode serving is becoming common for scaling LLM inference efficiently, and the KV cache transfer step directly determines how fast a user gets the first response token and how smoothly subsequent tokens stream — both critical for any latency-sensitive, real-time application.
What it means for AI agents and contact centers
For companies running self-hosted LLM inference behind voice agents or real-time contact-center automation, this update is relevant mainly if you operate vLLM with disaggregated prefill/decode on GPU or Trainium clusters with RDMA networking like EFA. Lower TTFT means the agent starts speaking sooner after a user stops talking, and more stable TPOT means less jitter in generated speech during high call volume. It's worth comparing the two connector modes (NixlConnector vs NixlPushConnector) under your actual concurrent call load and input lengths before switching, since the benefit scales with request rate and prompt size.
🧪 Worth Testing
Switching to the push-based connector could reduce response latency and improve consistency under concurrent call load if you run your own LLM inference stack with vLLM disaggregated serving — directly impacting perceived responsiveness of voice agents.
Push-based NIXL KV connector for disaggregated vLLM inference· New
Sources
- AWS Machine LearningOfficialPrimary sourceOriginal article →„Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM“19 Oct 2026, 19:05
- AWS Machine LearningOfficialOriginal article →„Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM“19 Oct 2026, 19:03
- Published by source
- 19 Oct 2026, 19:05
- Found by our system
- 19 Oct 2026, 19:13
- Summary generated
- 19 Oct 2026, 19:15
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy