Skip to content
Sunday, 22 November 2026
Tenesys AI News
Subscribe

AI Models

New models and versions from OpenAI, Anthropic, Google, Meta, Mistral, xAI, DeepSeek, Qwen and others.

#Latency — 9 articles ✕

Models· Important· 🧪 Worth Testing

Amazon SageMaker AI introduces KV cache-aware routing for LLMs

Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Optimizing RAG Retrieval with Learned Neural Routers

Apple researchers have introduced an approach to accelerate Maximum Inner Product Search (MIPS) in RAG systems using small, task-specific neural networks. By training models (SupportNet and KeyNet) on query distributions, systems can achieve higher recall at lower latency compared to traditional centroid-based partitioning.

Apple Machine Learning
Models· Important· 🧪 Worth Testing

Anthropic adds message threads to Messages API for improved latency

The new message threads feature in beta allows developers to keep conversation state on the server. By sending only incremental updates instead of the full history, you can achieve faster response times in complex agentic loops.

Claude Platform Release Notes
Models· Important· 🧪 Worth Testing

Google updates A2UI with swappable inference formats and Express mode

Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.

Google Developers
Models· Important· 🧪 Worth Testing

Boa Compressor Brings Online Long-Context Compression to LLMs

Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

Reduce AI agent costs on Bedrock with new prompt caching

Amazon Bedrock now supports explicit and implicit prompt caching for OpenAI models, allowing you to store and reuse static prompt prefixes like tools, instructions, and conversation history. By avoiding the re-processing of identical tokens across turns, you can significantly lower input costs and improve responsiveness in complex agent workflows.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Google Open-Sources ML Drift, a New GPU Engine for On-Device AI Inference

Google released ML Drift, an open-source, cross-platform GPU compute engine for on-device ML inference, replacing the legacy TFLite GPU delegate. It unifies shader handling across OpenGL ES, OpenCL, Metal and WebGPU, adds 5D tensor support, and introduces stage-aware optimizations for LLM prefill/decode. It is already running in production in Chrome, YouTube Shorts, Google Photos, Meet and partner apps from Adobe and Snap.

Google Developers