Google has introduced Tool Retrieval, a new preview feature for the Gemini Interactions API that allows agents to load tool schemas on demand. By keeping only names and descriptions in the context and fetching parameter details only when needed, developers can handle large tool libraries without bloating input tokens.
AWS details three strategies to manage massive JSON payloads (20MB+) in AI agents, achieving up to 71% fewer tokens and 80% lower latency. The approach involves offloading state to Amazon S3, paginating large tool results, and compacting dictionary outputs.
Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.
Google has introduced the Gemini Decisions API, powered by the new gemini-decision-1 model. Unlike standard generative models, this tool performs single-pass classification and scoring across text, audio, image, video, and PDF inputs, returning confidence scores instead of text.
Amazon has expanded the capabilities of the AgenticRetrieveStream API, allowing developers to specify the iteration level for knowledge base requests. You can now choose between zero, one, or multiple retrieval steps to balance speed, cost, and complexity per request.
Amazon SageMaker Inference has introduced support for NVIDIA Multi-Instance GPU (MIG), allowing you to partition a single physical GPU into multiple independent units. This hardware-level division provides dedicated compute and memory to each partition, ensuring predictable latency and strong isolation between models hosted on the same accelerator.
AI agents frequently misinterpret legacy code comments, such as TODOs or FIXMEs, as current instructions. Research shows that 6% of coding sessions result in agents chasing irrelevant, closed, or obsolete bug tickets, leading to increased costs and scope creep.
Apple researchers have introduced an approach to accelerate Maximum Inner Product Search (MIPS) in RAG systems using small, task-specific neural networks. By training models (SupportNet and KeyNet) on query distributions, systems can achieve higher recall at lower latency compared to traditional centroid-based partitioning.
Researchers have developed a method to calculate the theoretical upper bound for Multi-Draft Speculative Decoding (MDSD) efficiency by solving the dual of the optimal transport problem. The study reveals that current verification algorithms often fall short of this theoretical limit, and that sampling without replacement significantly improves acceptance rates compared to sampling with replacement.
Cartesia has introduced two models for voice applications: Sonic (text-to-speech) and Ink (speech-to-text). Built on State Space Model (SSM) architecture, these models are designed for ultra-low-latency, real-time performance suitable for production voice agents.
The new message threads feature in beta allows developers to keep conversation state on the server. By sending only incremental updates instead of the full history, you can achieve faster response times in complex agentic loops.
AssemblyAI has released AssemblyAI-TTS-1, a text-to-speech model built for voice agents. It features sub-100ms latency, 4,000+ available voices, and instant voice cloning, priced at $15 per 1M characters without concurrency limits.
Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.
Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.
Amazon Bedrock now supports explicit and implicit prompt caching for OpenAI models, allowing you to store and reuse static prompt prefixes like tools, instructions, and conversation history. By avoiding the re-processing of identical tokens across turns, you can significantly lower input costs and improve responsiveness in complex agent workflows.
IBM Research released vLLM-Hook MIA, a plugin for the open-source vLLM inference engine that lets developers observe and intervene in a model's internal states at runtime — for example to decline malicious prompts, flag hallucinated facts, or adjust tone — without disabling vLLM's fast graph-replay execution mode. According to IBM's technical report, MIA is up to 20x faster than the earlier vLLM-Hook and up to 14x faster than methods that insert code directly into the model. In tests, steering an 8B Llama or 1.5B Qwen model added only about 2% latency, and a 70B Llama model added about 6%.
AWS released a push-based NIXL connector for vLLM that changes how key-value (KV) cache data moves between prefill and decode instances in disaggregated LLM serving. Instead of the decode instance pulling KV blocks after prefill finishes, the decode instance registers its target memory in advance and the prefill instance writes the KV data directly once ready, moving coordination work off the critical path. Benchmarked on a 1-prefill/1-decode Qwen3-32B deployment using EC2 P5en instances with EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) than the default pull mode, especially under high request rates.
Anam, which builds photorealistic interactive avatars, switched to Deepgram's Flux speech-to-text model to solve turn-taking problems in unscripted, real-time conversation. In A/B testing against a self-hosted open-source STT option, Flux cut unwanted avatar barge-ins by roughly 50% with no added latency penalty, and complaints about turn-taking on English sessions dropped to near zero. Anam deployed Flux self-hosted on its own Kubernetes/H100 infrastructure, rolling it out to 100% of turnkey users after a two-week self-hosted deployment (versus days for Deepgram's hosted option).
AWS detailed three kernel-level optimizations for Mixture-of-Experts (MoE) model serving on Hopper GPUs, implemented in collaboration with the vLLM community and shipped in Amazon SageMaker AI's LMI container. The changes target the inefficiencies of fused MoE computation during low-batch decode, which previously left Tensor Cores underutilized. Combined, the optimizations deliver up to 10% higher throughput and up to 9% faster time-to-first-token (TTFT).
Google released ML Drift, an open-source, cross-platform GPU compute engine for on-device ML inference, replacing the legacy TFLite GPU delegate. It unifies shader handling across OpenGL ES, OpenCL, Metal and WebGPU, adds 5D tensor support, and introduces stage-aware optimizations for LLM prefill/decode. It is already running in production in Chrome, YouTube Shorts, Google Photos, Meet and partner apps from Adobe and Snap.
AssemblyAI introduced the Sync API, a new transport for transcribing short audio clips: one HTTP POST request returns a finished Universal-3.5 Pro transcript in the same response, at roughly 134ms median latency. It fills the gap between the Async API (submit-and-poll, 5-6 seconds of added latency) and the Realtime API (WebSocket, built for ongoing sessions), targeting workloads like dictation, voice-agent turn transcription, IVR, and push-to-talk. Clips from 80ms to 2 minutes and up to 40MB are supported, with word error rate of 1.59% on short-form audio. Pricing is $0.45/hr, the same rate as Universal-3.5 Pro Realtime.
AssemblyAI launched a hosted deployment of the open-source Qwen3.5 4B model on its LLM Gateway, optimized specifically for rewriting raw speech-to-text output into clean, formatted text. On voice rewrite tasks like dictation cleanup and transcript formatting, the model averaged 612ms response time—1.9x faster than GPT-4.1—at 94% lower cost per hour of audio processed. The model is priced at $0.10/$0.50 per million tokens (prompt/completion) and supports only max_tokens, temperature, and stream parameters, with no tool calling.