Google has introduced Tool Retrieval, a new preview feature for the Gemini Interactions API that allows agents to load tool schemas on demand. By keeping only names and descriptions in the context and fetching parameter details only when needed, developers can handle large tool libraries without bloating input tokens.
AWS details three strategies to manage massive JSON payloads (20MB+) in AI agents, achieving up to 71% fewer tokens and 80% lower latency. The approach involves offloading state to Amazon S3, paginating large tool results, and compacting dictionary outputs.
Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.
Google has introduced the Gemini Decisions API, powered by the new gemini-decision-1 model. Unlike standard generative models, this tool performs single-pass classification and scoring across text, audio, image, video, and PDF inputs, returning confidence scores instead of text.
Amazon has expanded the capabilities of the AgenticRetrieveStream API, allowing developers to specify the iteration level for knowledge base requests. You can now choose between zero, one, or multiple retrieval steps to balance speed, cost, and complexity per request.
Amazon SageMaker Inference has introduced support for NVIDIA Multi-Instance GPU (MIG), allowing you to partition a single physical GPU into multiple independent units. This hardware-level division provides dedicated compute and memory to each partition, ensuring predictable latency and strong isolation between models hosted on the same accelerator.
AI agents frequently misinterpret legacy code comments, such as TODOs or FIXMEs, as current instructions. Research shows that 6% of coding sessions result in agents chasing irrelevant, closed, or obsolete bug tickets, leading to increased costs and scope creep.
Apple researchers have introduced an approach to accelerate Maximum Inner Product Search (MIPS) in RAG systems using small, task-specific neural networks. By training models (SupportNet and KeyNet) on query distributions, systems can achieve higher recall at lower latency compared to traditional centroid-based partitioning.
Researchers have developed a method to calculate the theoretical upper bound for Multi-Draft Speculative Decoding (MDSD) efficiency by solving the dual of the optimal transport problem. The study reveals that current verification algorithms often fall short of this theoretical limit, and that sampling without replacement significantly improves acceptance rates compared to sampling with replacement.
Cartesia has introduced two models for voice applications: Sonic (text-to-speech) and Ink (speech-to-text). Built on State Space Model (SSM) architecture, these models are designed for ultra-low-latency, real-time performance suitable for production voice agents.
The new message threads feature in beta allows developers to keep conversation state on the server. By sending only incremental updates instead of the full history, you can achieve faster response times in complex agentic loops.
AssemblyAI has released AssemblyAI-TTS-1, a text-to-speech model built for voice agents. It features sub-100ms latency, 4,000+ available voices, and instant voice cloning, priced at $15 per 1M characters without concurrency limits.
Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.
Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.
Amazon Bedrock now supports explicit and implicit prompt caching for OpenAI models, allowing you to store and reuse static prompt prefixes like tools, instructions, and conversation history. By avoiding the re-processing of identical tokens across turns, you can significantly lower input costs and improve responsiveness in complex agent workflows.