IBM Research released vLLM-Hook MIA, a plugin for the open-source vLLM inference engine that lets developers observe and intervene in a model's internal states at runtime — for example to decline malicious prompts, flag hallucinated facts, or adjust tone — without disabling vLLM's fast graph-replay execution mode. According to IBM's technical report, MIA is up to 20x faster than the earlier vLLM-Hook and up to 14x faster than methods that insert code directly into the model. In tests, steering an 8B Llama or 1.5B Qwen model added only about 2% latency, and a 70B Llama model added about 6%.
Google Cloud and NVIDIA announced general availability of A5X instances built on NVIDIA Vera Rubin NVL72 GPUs, ConnectX-9 SuperNICs and Google's Virgo network, targeting large-scale agentic inference and reinforcement learning. Google Cloud plans to scale Vera Rubin infrastructure to up to 960,000 Rubin GPUs across multiple sites, with general availability next month and planned expansion to Europe. London-based Ineffable Intelligence, founded by AlphaGo creator David Silver, is the first customer, using the platform for continuous reinforcement-learning 'superlearner' systems. WPP also demonstrated a creative agent architecture on A5X using NVIDIA Nemotron models and OpenShell, running workloads on a single instance that previously needed a multi-node cluster.
NVIDIA has added three new open, commercially licensed components to its Nemotron Retriever lineup: Embed models (8B and 1B, in BF16 and NVFP4) for searching text and images, a Rerank model (8B) that reorders retrieved evidence by relevance, and a compact 631M Parse model that converts document images into structured text. The goal is to feed AI agents with better, more relevant evidence so they need fewer search rounds and fewer inference tokens to answer a question. NVIDIA reports the new Embed and Rerank models top the ViDoRe v3.1 leaderboards for dense retrieval and reranking respectively.
AI research company i14 partnered with AWS to port its Laplace 3B model, a 24-layer architecture built on Gated Linear Attention (GLA) instead of standard transformer attention, onto AWS Trainium2 using native PyTorch (TorchNeuron Native). Rather than a KV cache that expands with every token, the model keeps a constant 29.5 MB recurrent state per sequence. On a single trn2.48xlarge instance, the team reached about 120,000 training tokens per second and served 11,544 output tokens per second across 1,280 simultaneous sequences through an unmodified vLLM engine.
NVIDIA released Cosmos3-H-Dreams, a world foundation model that generates interactive, real-time surgical scenes based on robot kinematics from actual clinical cholecystectomy procedures recorded on the CMR Surgical Versius platform. Built on the 16-billion-parameter Cosmos3-H-Embodied-Nano model and accelerated with NVIDIA's FlashDreams streaming-inference library, it runs on a single RTX PRO 6000 GPU. The system learns visual surgical dynamics directly from video and kinematics data rather than explicit physics modeling, and offers four-step and two-step distilled checkpoints trading quality for speed.
Google announced new vLLM and SGLang inference engines built directly on PyTorch for TPU, developed with Inferact and RadixArk respectively. The engines run on a new TorchTPU layer that makes TPUs a native PyTorch device, replacing the previous JAX-based backend approach. Google reports a public benchmark result of 2.01 cents per million tokens in high-concurrency configurations via the SemiAnalysis InferenceX project, plus support for agentic workloads, long context, disaggregated serving and multimodal models.
Google Research announced the final milestone of its Coral NPU open hardware standard — a 32-bit RISC-V architecture with a new Matrix Execution Unit for up to 1 TOPS at roughly 10mW — plus a v1.0 open-source IREE/MLIR compiler that takes JAX, PyTorch and TensorFlow models directly to the hardware. Synaptics is launching the first commercial silicon built on this standard, including the Astra SR200 microcontroller series (400 GOPS) and new Coralboard SR and SL developer kits, with the SL now generally available via DigiKey. Google is also partnering with VeriSilicon and GlobalFoundries to bring the open RTL into production chips.
Red Hat, IBM, and Google announced llm-d 1.0, a Kubernetes-native distributed inference serving system that moves beyond proof-of-concept performance tricks into full production readiness. The release adds multi-tenant quality of service, zero-downtime operations, and agent-aware serving so the same endpoint can handle both human users and autonomous agents efficiently. In one test, llm-d's prefix-aware router captured 97% of available cache reuse on an agentic coding workload, and IBM/Red Hat ran 3,000 concurrent software-engineering agents for 5 to 10 times less cost than using the same model as a hosted service.
AWS has rolled out structured outputs across every Anthropic Claude model hosted on Amazon Bedrock, reaching both the standard runtime endpoint and the newer Mantle endpoint built for agentic workloads. Instead of hoping the model returns valid JSON, developers can now lock responses to an exact schema through constrained decoding, which blocks the model from generating tokens that would break the format. A companion feature, strict tool use, applies the same guarantee to function-calling arguments so agent workflows don't fail on malformed inputs.
AWS and Deepgram jointly rebuilt the Nova-3 speech-to-text model to run natively on Trainium2 chips instead of standard GPU-based PyTorch setups, targeting large offline transcription jobs. The work involved rewriting bottleneck layers as custom kernels and replacing Deepgram's Python serving layer with a Rust runtime calling Trainium directly. The result: a single trn2.3xlarge instance running eight parallel model copies delivers 11-70% more transcribed audio per dollar than Deepgram's prior infrastructure, with similar savings potentially available to customers self-hosting on SageMaker.
AWS released a push-based NIXL connector for vLLM that changes how key-value (KV) cache data moves between prefill and decode instances in disaggregated LLM serving. Instead of the decode instance pulling KV blocks after prefill finishes, the decode instance registers its target memory in advance and the prefill instance writes the KV data directly once ready, moving coordination work off the critical path. Benchmarked on a 1-prefill/1-decode Qwen3-32B deployment using EC2 P5en instances with EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) than the default pull mode, especially under high request rates.
Microsoft and NVIDIA released a joint whitepaper, 'Risk Management for Open-Weight Models in Enterprise,' the first in a planned series offering enterprises a shared risk taxonomy and methodology for evaluating open-weight AI models. The paper argues that whether a model is open or closed-weight is not the key risk factor — what matters is the use case, deployment architecture, and surrounding safeguards. It organizes risks into five categories (Cybersecurity, Data Protection, Content, Human-AI Interaction, Bias) and maps evaluations to those risks, aligning with ISO standards and the NIST AI Risk Management Framework.
Reflection, a new US AI lab, has launched Beam, its first model, as an open-weight mixture-of-experts system with 501B total parameters (23B active per token) and a 1M-token context window, released under Apache 2.0. The model is built for coding and agentic use cases, ships with a custom Rust-based inference harness, and has day-0 support in transformers, vLLM and SGLang. Reflection reports strong results on coding, reasoning and tool-calling benchmarks across BF16, FP8 and NVFP4 checkpoints.
IBM has released Granite Switch 4.1 in 8B and 30B sizes, a modular system that runs chat, decision-making and speech transcription from one hosted endpoint instead of separate models. The update adds a general-purpose Decision adapter, a specialized Guardian adapter for safety-related classifications, and an embedded ASR model, Granite-Speech-5.0-Turbo-CTC, which tops the OpenASR Leaderboard. Despite bundling three capabilities, the combined model uses roughly the same VRAM as the base model alone.
Apple's MLX team published benchmarks showing large open-weight models — including a near-trillion-parameter MoE model called Muse Spark Open — running on Mac Studio hardware, both on a single machine and distributed across up to four Mac Studios using a new RDMA-over-Thunderbolt driver. The new M5 Ultra chip, with dedicated GPU Neural Accelerators and 1.2 TB/s memory bandwidth, delivers up to 4.6x faster time-to-first-token and 1.6x faster generation speed versus the previous M3 Ultra. For models too large to fit on one machine, MLX now supports pipeline and tensor parallelization across multiple Macs connected via Thunderbolt RDMA, letting a four-Mac-Studio cluster generate over 36 tokens per second on an 816B-parameter model.
Mistral AI has launched Timestral, its first family of time-series foundation models, available in four sizes from 5M to 209M parameters. The largest model ranks first on all three major public forecasting benchmarks (GIFT-Eval, TIME, fev-bench), beating previous state-of-the-art models that have more parameters. The models produce probabilistic, multivariate forecasts, support past and known-future covariates, and run efficiently on CPUs without needing a GPU.
Cohere built a controlled testbed to study how teams of coding agents should be organized, testing where work is delegated, how leads wait on workers, which models play which role, team size, and communication patterns. On ProgramBench, well-organized teams outperformed single agents by up to ~10 percentage points but at 3–7x higher cost. The key finding: delegating only exploration and planning to workers matched the performance of delegating all stages (implementation and verification too) at half the cost, and a strong lead paired with 16 cheap planning workers reached 61.8% reward versus 63.2% for an all-strong-model team, cutting cost from $510 to $155.
Tamnoon built Tami, an agentic orchestrator on Amazon Bedrock that investigates cloud security findings from CNAPPs and cloud providers, scores remediation safety, and generates fixes with guardrails. The company reports a greater than 90% reduction in critical cloud exposure within 90 days with zero production issues. The system uses four specialized agents (investigation, safety-scoring, remediation, code-generation) each calling different foundation models through Bedrock, with Bedrock Guardrails and Cross-Region inference providing safety and scale.
GitHub replaced its LLM-based pipeline for detecting generic secrets (like passwords and API keys) with a fine-tuned ModernBERT classifier combined with regex candidate generation. The new two-stage system improved detection accuracy while cutting scan time and serving costs dramatically, and it's now fast enough to run during push protection, not just after code is committed.
Liquid AI has published LFM2.5-24B-A2B, a mixture-of-experts model with 24B total parameters but only 2B active per token, designed to run reasoning agents on local hardware. The release expands the context window to 131K tokens, doubles the vocabulary size, and was trained on roughly double the pretraining data of its predecessor. On agentic and tool-use benchmarks the model performs on par with or better than models that have about 50% more parameters, while delivering fast inference speeds on both consumer and data-center hardware.
AWS detailed three kernel-level optimizations for Mixture-of-Experts (MoE) model serving on Hopper GPUs, implemented in collaboration with the vLLM community and shipped in Amazon SageMaker AI's LMI container. The changes target the inefficiencies of fused MoE computation during low-batch decode, which previously left Tensor Cores underutilized. Combined, the optimizations deliver up to 10% higher throughput and up to 9% faster time-to-first-token (TTFT).
NVIDIA has released the full Run:ai platform as open source, adding it as a foundation of the NVIDIA DSX AI Factory Platform for AI workload and GPU orchestration on Kubernetes. The release includes four modular components — KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, and Workload Map — that can be adopted together or independently. Eight ecosystem partners, including Crusoe, Mirantis, Nebius, OpenNebula Systems, Rafay, Saturn Cloud, Spectro Cloud, and vCluster, are integrating the platform into their own offerings.
IBM Research used a coordinated multi-agent 'agent factory' to explore hardware design optimizations for a quantum error-correction decoder, comparing proprietary and open frontier models under the same 100-hypothesis search budget. The open-model configuration, run on-premises via IBM's llm-d distributed inference framework, achieved a 52x speedup over baseline for about $26 in inference cost, versus $299 for a proprietary-model configuration that reached a higher 204x speedup. The study highlights that cache-aware routing and efficient serving (98% prefix-cache hit rate, ~$0.11 per million tokens) made the open-model approach dramatically cheaper while still finding valid, compact hardware designs.
A new Apple research paper introduces Minions, a system that pools unused GPU capacity across Apple Silicon Macs into a single distributed cloud for running machine learning inference. Instead of buying additional GPU hardware, teams could tap spare compute sitting idle on existing macOS devices. The authors say the design handles task scheduling, failure recovery, and efficient use of available resources, offering an alternative when dedicated GPU capacity is scarce or expensive.