Skip to content
Sunday, 22 November 2026
Tenesys AI News
Subscribe

All News

TENESYS AI NEWS tracks the most important AI news and explains what changed, why it matters and whether a technology is worth testing.

#Latency — 15 articles ✕

Agents· Important· 🧪 Worth Testing

Google Launches Tool Retrieval for Gemini API Agents

Google has introduced Tool Retrieval, a new preview feature for the Gemini Interactions API that allows agents to load tool schemas on demand. By keeping only names and descriptions in the context and fetching parameter details only when needed, developers can handle large tool libraries without bloating input tokens.

Google AI
Automation· Important· 🧪 Worth Testing

Optimizing large JSON workloads for AI agents

AWS details three strategies to manage massive JSON payloads (20MB+) in AI agents, achieving up to 71% fewer tokens and 80% lower latency. The approach involves offloading state to Amazon S3, paginating large tool results, and compacting dictionary outputs.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Amazon SageMaker AI introduces KV cache-aware routing for LLMs

Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.

AWS Machine Learning
Automation· Important· 🧪 Worth Testing

Amazon SageMaker now supports Multi-Instance GPU for managed endpoints

Amazon SageMaker Inference has introduced support for NVIDIA Multi-Instance GPU (MIG), allowing you to partition a single physical GPU into multiple independent units. This hardware-level division provides dedicated compute and memory to each partition, ensuring predictable latency and strong isolation between models hosted on the same accelerator.

AWS Machine Learning
Research· Important· 🧪 Worth Testing

Mitigating AI Agent Distraction in Large Codebases

AI agents frequently misinterpret legacy code comments, such as TODOs or FIXMEs, as current instructions. Research shows that 6% of coding sessions result in agents chasing irrelevant, closed, or obsolete bug tickets, leading to increased costs and scope creep.

Google Developers
Models· Important· 🧪 Worth Testing

Optimizing RAG Retrieval with Learned Neural Routers

Apple researchers have introduced an approach to accelerate Maximum Inner Product Search (MIPS) in RAG systems using small, task-specific neural networks. By training models (SupportNet and KeyNet) on query distributions, systems can achieve higher recall at lower latency compared to traditional centroid-based partitioning.

Apple Machine Learning
Research· 🧪 Worth Testing

Optimizing Multi-Draft Speculative Decoding Efficiency

Researchers have developed a method to calculate the theoretical upper bound for Multi-Draft Speculative Decoding (MDSD) efficiency by solving the dual of the optimal transport problem. The study reveals that current verification algorithms often fall short of this theoretical limit, and that sampling without replacement significantly improves acceptance rates compared to sampling with replacement.

arXiv cs.CL
Voice AI· Important· 🧪 Worth Testing

Cartesia releases SSM-based voice models on AWS for sub-100ms latency

Cartesia has introduced two models for voice applications: Sonic (text-to-speech) and Ink (speech-to-text). Built on State Space Model (SSM) architecture, these models are designed for ultra-low-latency, real-time performance suitable for production voice agents.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Anthropic adds message threads to Messages API for improved latency

The new message threads feature in beta allows developers to keep conversation state on the server. By sending only incremental updates instead of the full history, you can achieve faster response times in complex agentic loops.

Claude Platform Release Notes
Models· Important· 🧪 Worth Testing

Google updates A2UI with swappable inference formats and Express mode

Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.

Google Developers
Models· Important· 🧪 Worth Testing

Boa Compressor Brings Online Long-Context Compression to LLMs

Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

Reduce AI agent costs on Bedrock with new prompt caching

Amazon Bedrock now supports explicit and implicit prompt caching for OpenAI models, allowing you to store and reuse static prompt prefixes like tools, instructions, and conversation history. By avoiding the re-processing of identical tokens across turns, you can significantly lower input costs and improve responsiveness in complex agent workflows.

AWS Machine Learning