Skip to content
Sunday, 22 November 2026
Tenesys AI News
Subscribe

All News

TENESYS AI NEWS tracks the most important AI news and explains what changed, why it matters and whether a technology is worth testing.

#Latency — 22 articles ✕

Agents· Important· 🧪 Worth Testing

Google Launches Tool Retrieval for Gemini API Agents

Google has introduced Tool Retrieval, a new preview feature for the Gemini Interactions API that allows agents to load tool schemas on demand. By keeping only names and descriptions in the context and fetching parameter details only when needed, developers can handle large tool libraries without bloating input tokens.

Google AI
Automation· Important· 🧪 Worth Testing

Optimizing large JSON workloads for AI agents

AWS details three strategies to manage massive JSON payloads (20MB+) in AI agents, achieving up to 71% fewer tokens and 80% lower latency. The approach involves offloading state to Amazon S3, paginating large tool results, and compacting dictionary outputs.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Amazon SageMaker AI introduces KV cache-aware routing for LLMs

Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.

AWS Machine Learning
Automation· Important· 🧪 Worth Testing

Amazon SageMaker now supports Multi-Instance GPU for managed endpoints

Amazon SageMaker Inference has introduced support for NVIDIA Multi-Instance GPU (MIG), allowing you to partition a single physical GPU into multiple independent units. This hardware-level division provides dedicated compute and memory to each partition, ensuring predictable latency and strong isolation between models hosted on the same accelerator.

AWS Machine Learning
Research· Important· 🧪 Worth Testing

Mitigating AI Agent Distraction in Large Codebases

AI agents frequently misinterpret legacy code comments, such as TODOs or FIXMEs, as current instructions. Research shows that 6% of coding sessions result in agents chasing irrelevant, closed, or obsolete bug tickets, leading to increased costs and scope creep.

Google Developers
Models· Important· 🧪 Worth Testing

Optimizing RAG Retrieval with Learned Neural Routers

Apple researchers have introduced an approach to accelerate Maximum Inner Product Search (MIPS) in RAG systems using small, task-specific neural networks. By training models (SupportNet and KeyNet) on query distributions, systems can achieve higher recall at lower latency compared to traditional centroid-based partitioning.

Apple Machine Learning
Research· 🧪 Worth Testing

Optimizing Multi-Draft Speculative Decoding Efficiency

Researchers have developed a method to calculate the theoretical upper bound for Multi-Draft Speculative Decoding (MDSD) efficiency by solving the dual of the optimal transport problem. The study reveals that current verification algorithms often fall short of this theoretical limit, and that sampling without replacement significantly improves acceptance rates compared to sampling with replacement.

arXiv cs.CL
Voice AI· Important· 🧪 Worth Testing

Cartesia releases SSM-based voice models on AWS for sub-100ms latency

Cartesia has introduced two models for voice applications: Sonic (text-to-speech) and Ink (speech-to-text). Built on State Space Model (SSM) architecture, these models are designed for ultra-low-latency, real-time performance suitable for production voice agents.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Anthropic adds message threads to Messages API for improved latency

The new message threads feature in beta allows developers to keep conversation state on the server. By sending only incremental updates instead of the full history, you can achieve faster response times in complex agentic loops.

Claude Platform Release Notes
Models· Important· 🧪 Worth Testing

Google updates A2UI with swappable inference formats and Express mode

Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.

Google Developers
Models· Important· 🧪 Worth Testing

Boa Compressor Brings Online Long-Context Compression to LLMs

Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

Reduce AI agent costs on Bedrock with new prompt caching

Amazon Bedrock now supports explicit and implicit prompt caching for OpenAI models, allowing you to store and reuse static prompt prefixes like tools, instructions, and conversation history. By avoiding the re-processing of identical tokens across turns, you can significantly lower input costs and improve responsiveness in complex agent workflows.

AWS Machine Learning
Research· Important· 🧪 Worth Testing

IBM plugin lets vLLM monitor and steer LLMs in real time with minimal slowdown

IBM Research released vLLM-Hook MIA, a plugin for the open-source vLLM inference engine that lets developers observe and intervene in a model's internal states at runtime — for example to decline malicious prompts, flag hallucinated facts, or adjust tone — without disabling vLLM's fast graph-replay execution mode. According to IBM's technical report, MIA is up to 20x faster than the earlier vLLM-Hook and up to 14x faster than methods that insert code directly into the model. In tests, steering an 8B Llama or 1.5B Qwen model added only about 2% latency, and a 70B Llama model added about 6%.

IBM Research
Research· Important· 🧪 Worth Testing

AWS introduces push-based KV transfer to cut latency in disaggregated vLLM inference

AWS released a push-based NIXL connector for vLLM that changes how key-value (KV) cache data moves between prefill and decode instances in disaggregated LLM serving. Instead of the decode instance pulling KV blocks after prefill finishes, the decode instance registers its target memory in advance and the prefill instance writes the KV data directly once ready, moving coordination work off the critical path. Benchmarked on a 1-prefill/1-decode Qwen3-32B deployment using EC2 P5en instances with EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) than the default pull mode, especially under high request rates.

AWS Machine Learning
Voice AI· Important· 🧪 Worth Testing

Deepgram's Flux STT cuts unwanted interruptions in Anam's avatar conversations by ~50%

Anam, which builds photorealistic interactive avatars, switched to Deepgram's Flux speech-to-text model to solve turn-taking problems in unscripted, real-time conversation. In A/B testing against a self-hosted open-source STT option, Flux cut unwanted avatar barge-ins by roughly 50% with no added latency penalty, and complaints about turn-taking on English sessions dropped to near zero. Anam deployed Flux self-hosted on its own Kubernetes/H100 infrastructure, rolling it out to 100% of turnkey users after a two-week self-hosted deployment (versus days for Deepgram's hosted option).

Deepgram
Research· 🧪 Worth Testing

AWS open-sources MoE kernel optimizations for vLLM, cuts latency and boosts throughput on H200

AWS detailed three kernel-level optimizations for Mixture-of-Experts (MoE) model serving on Hopper GPUs, implemented in collaboration with the vLLM community and shipped in Amazon SageMaker AI's LMI container. The changes target the inefficiencies of fused MoE computation during low-batch decode, which previously left Tensor Cores underutilized. Combined, the optimizations deliver up to 10% higher throughput and up to 9% faster time-to-first-token (TTFT).

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Google Open-Sources ML Drift, a New GPU Engine for On-Device AI Inference

Google released ML Drift, an open-source, cross-platform GPU compute engine for on-device ML inference, replacing the legacy TFLite GPU delegate. It unifies shader handling across OpenGL ES, OpenCL, Metal and WebGPU, adds 5D tensor support, and introduces stage-aware optimizations for LLM prefill/decode. It is already running in production in Chrome, YouTube Shorts, Google Photos, Meet and partner apps from Adobe and Snap.

Google Developers
Voice AI· Important· 🧪 Worth Testing

AssemblyAI launches Sync API: full transcript in a single HTTP call, ~134ms latency

AssemblyAI introduced the Sync API, a new transport for transcribing short audio clips: one HTTP POST request returns a finished Universal-3.5 Pro transcript in the same response, at roughly 134ms median latency. It fills the gap between the Async API (submit-and-poll, 5-6 seconds of added latency) and the Realtime API (WebSocket, built for ongoing sessions), targeting workloads like dictation, voice-agent turn transcription, IVR, and push-to-talk. Clips from 80ms to 2 minutes and up to 40MB are supported, with word error rate of 1.59% on short-form audio. Pricing is $0.45/hr, the same rate as Universal-3.5 Pro Realtime.

AssemblyAI
Voice AI· Important· 🧪 Worth Testing

AssemblyAI adds Qwen3.5 4B to LLM Gateway for fast voice-transcript rewriting

AssemblyAI launched a hosted deployment of the open-source Qwen3.5 4B model on its LLM Gateway, optimized specifically for rewriting raw speech-to-text output into clean, formatted text. On voice rewrite tasks like dictation cleanup and transcript formatting, the model averaged 612ms response time—1.9x faster than GPT-4.1—at 94% lower cost per hour of audio processed. The model is priced at $0.10/$0.50 per million tokens (prompt/completion) and supports only max_tokens, temperature, and stream parameters, with no tool calling.

AssemblyAI