Amazon SageMaker AI has launched a new 'KV_AWARE' routing strategy that directs LLM inference requests to instances based on their actual KV cache state rather than just prompt headers. This optimizes multi-turn conversations and long-context workloads by ensuring requests land where relevant data is already cached.
OpenAI has launched a public beta for the Responses API that allows you to select specific background processing windows. By opting for a two-hour or eight-hour completion window, your company can achieve substantial cost savings on token usage.
Researchers introduced OmniTag-Bench, a new five-language framework for evaluating multimodal LLMs on simultaneous speech transcription and metadata tagging. Instead of using separate specialist models, the study demonstrates that a single, fine-tuned multimodal LLM can extract transcripts, emotion, gender, language, and named entities in a single inference pass.
Anthropic has implemented a text-based watermark for its newest Claude models, including Opus 4.6, Opus 4.5, Sonnet 4.5, and Haiku 4.5. The watermark applies when using default sampling configurations but may be absent if users modify temperature, top_p, or top_k settings.
OpenAI has introduced omni-moderation-csam-latest, a specialized model for identifying child sexual abuse material (CSAM) in images. This tool is available to approved organizations at no cost to assist in prioritizing content for human moderation.
Hugging Face has partnered with OpenRouter to power its language model inference. All language model requests now use OpenRouter to automatically select the optimal provider based on real-time price, speed, and reliability metrics.
Amazon has released a reference architecture that integrates the NVIDIA NeMo Switchyard library with LiteLLM to dynamically route coding agent requests between different models on Bedrock. By using stage or random routing, you can shift traffic between high-capability and high-efficiency models to balance performance and cost.
NVIDIA has released a technical workflow for serving the 753B parameter GLM-5.3 MoE model across two GB300-equipped DGX Stations. By utilizing tensor parallelism (TP=2) over RDMA connections and vLLM, you can overcome single-GPU memory constraints (252 GB HBM3e) for massive frontier-scale models.
The llama.cpp project has officially reached its 1.0.0 milestone, three and a half years after its inception. The release introduces strict semantic versioning to ensure better predictability for developers relying on its C/C++ libraries, libllama and libggml.
DeepSeek has moved beyond basic model inference by exposing its full application-level toolkit to API users. Developers can now enable parameters for web search, URL reading, file parsing, and safety filtering directly in chat completions, or utilize standalone ASR, TTS, and OCR endpoints.
Amazon has released SGLang-Neuron, a plugin allowing developers to run large-scale AI models on AWS Trainium hardware using the standard SGLang interface. This integration enables native PyTorch execution and supports advanced features like PD disaggregation over EFA for optimized serving.
Google has introduced Gemini 4 Hydrogen, a new model optimized for rapid, cost-effective execution in agentic and coding environments. It features a 1-million-token context window and variable 'thinking' levels, allowing developers to balance reasoning depth, latency, and cost.
Amazon now offers a direct Web Search API for Bedrock AgentCore, enabling developers to perform web searches via standard AWS SDKs and CLI. This bypasses the need for the AgentCore Gateway, simplifying the process for custom agent loops and non-agent applications that require real-time information.
Google has introduced the Gemini Decisions API, powered by the new gemini-decision-1 model. Unlike standard generative models, this tool performs single-pass classification and scoring across text, audio, image, video, and PDF inputs, returning confidence scores instead of text.