Skip to content
Thursday, 19 November 2026
Tenesys AI News
Subscribe

All News

TENESYS AI NEWS tracks the most important AI news and explains what changed, why it matters and whether a technology is worth testing.

“Inference” — 82 articles ✕

Models· Important· 🧪 Worth Testing

NVIDIA Simplifies Edge AI Deployment with TensorRT Edge-LLM

NVIDIA has released a new command-line interface for the TensorRT Edge-LLM runtime, allowing developers to quickly prototype and test AI models on the DGX Spark 64GB workstation. The tool chain facilitates local model serving and supports DSpark speculative decoding to boost generation performance on edge-constrained hardware.

NVIDIA Developer
Agents· Important· 🧪 Worth Testing

NVIDIA Releases NemoClaw for Optimized and Secure Agent Deployment

NVIDIA introduced NemoClaw, an open blueprint framework designed to combine agent harnesses with optimized Nemotron models and secure runtimes. It enables developers to deploy agents that are governed by strict security policies while maintaining high performance on owned infrastructure.

NVIDIA Developer
Research· Important· 🧪 Worth Testing

Artifact Distillation: Boosting Small Models Without Fine-Tuning

Researchers have proposed 'Artifact Distillation,' a method where frontier models externalize their reasoning into an 'artifact bank.' Smaller models query this bank at inference time to improve performance without requiring expensive model retraining or gradient updates.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

NVIDIA releases multimodal Nemotron 3.5 Super VL for AWS EKS

NVIDIA has introduced Nemotron 3.5 Super VL, a 120B-parameter multimodal model featuring a 1M token context window. The model utilizes a hybrid Mamba-2 and MoE architecture to balance reasoning capabilities with efficient inference, and is now available for deployment on Amazon EKS and SageMaker HyperPod.

AWS Machine Learning
BreakingModels· 🧪 Worth Testing

Mistral AI Launches Mistral Large 4: A 1T Parameter Open-Weight Model

Mistral AI has released Mistral Large 4 (Le Chonk), a 1-trillion-parameter open-weight model optimized for coding, cybersecurity, and reasoning. The model introduces three distinct reasoning modes to balance performance with inference costs and features enhanced guardrails for secure agent deployments.

Mistral AI
Automation· Important· 🧪 Worth Testing

AWS launches optimized Ray Train Deep Learning Containers for EKS

AWS introduced pre-built Ray Train Deep Learning Containers (DLCs) that consolidate PyTorch, CUDA, NCCL, and EFA networking libraries into a single tested image. This approach allows your team to deploy distributed training clusters on Amazon EKS without maintaining custom build pipelines or resolving complex driver compatibility issues.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

Optimizing GPU Infrastructure with CUDA 13.4 Memory Enforcement

NVIDIA released CUDA 13.4, adding hardware-transparent memory boundary enforcement for GPU sharing. Paired with the new open-source GPU Fractioning Operator and KAI Scheduler v0.18.0, teams can now define specific memory requests, burst limits, and compute modes directly in Kubernetes Pod annotations.

NVIDIA Developer
Models· Important· 🧪 Worth Testing

Boosting Large-Scale MoE Model Throughput on NVIDIA Hardware

NVIDIA has documented a threefold optimization approach to accelerate MoE inference, tested on the 124-billion-parameter Ling-3.0-flash model. By utilizing specialized Humming kernels, FP8 precision for the LM head, and Multi-Token Prediction, developers can achieve a 42.5% throughput increase and a 24.5% reduction in latency.

NVIDIA Developer
Models· Important· 🧪 Worth Testing

Google updates A2UI with swappable inference formats and Express mode

Google has updated its A2UI generative UI protocol to include swappable inference formats, addressing latency and reliability issues found in the previous Direct JSON approach. A new format, A2UI Express, reduces token usage by up to 3x and latency by up to 3x by using a concise Python-like syntax.

Google Developers
Research· Important· 🧪 Worth Testing

TraceQuery optimizes inference for diffusion language models

Researchers have developed TraceQuery, a sparse decoding framework that reduces the computational cost of diffusion large language models (dLLMs). By analyzing denoising trajectories, the method selectively recomputes only necessary token representations rather than performing redundant operations, maintaining high accuracy with fewer computations.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

Boa Compressor Brings Online Long-Context Compression to LLMs

Researchers have introduced the Boa Compressor, an online tool utilizing a Mamba state-space model to compress long-form context into shorter representations in linear time. By distilling knowledge from full-context LLM interactions, this method allows for efficient per-query inference without the massive latency overhead of transformer-based compression.

arXiv cs.CL
Models· Important· 🧪 Worth Testing

IBM Spyre accelerator gains native PyTorch and vLLM integration

IBM has updated its software ecosystem to treat the Spyre dataflow accelerator as a first-class PyTorch backend. By leveraging TorchDynamo and TorchInductor, developers can now run models on IBM Z, LinuxONE, and Power systems using standard PyTorch workflows and vLLM plugins.

IBM Research
Cyber Security· Important· 🧪 Worth Testing

Cohere launches confidential computing for Model Vault AI inference

Cohere has released a public beta of Confidential Computing on Model Vault, running on NVIDIA H100 GPUs via Microsoft Azure. The system encrypts data in use across both CPU and GPU, encrypts the request payload itself using Oblivious HTTP (OHTTP), and uses remote attestation so clients can verify the environment before sending sensitive data. If verification fails, the client refuses to send the protected request. Cohere plans to open-source the confidential inference stack and extend support to NVIDIA Blackwell (B200) GPUs toward general availability.

Cohere
Agents· 🧪 Worth Testing

AWS shows hybrid local-plus-cloud agent framework to cut LLM costs

AWS published a walkthrough for the Adaptive Agentic Intelligence Framework (AAIF), built on Strands Agents, which routes simple requests to small local models (via Ollama) and only sends complex requests to cloud models through Amazon Bedrock. A lightweight 1B-parameter model (llama3.2:1b) acts as an orchestrator deciding where each request should go, with an example IoT monitoring agent that uses local telemetry retrieval, local vision analysis, and cloud-based expert insights. AWS states that in the hybrid setup, roughly 80 percent of traffic can be handled locally without hitting a paid API, and the architecture can run fully on edge hardware (e.g. Raspberry Pi with 8GB memory), fully in the cloud, or in a hybrid mode.

AWS Machine Learning
Voice AI· Important· 🧪 Worth Testing

IBM refreshes Granite Speech lineup: new ASR model, quicker edge transcription, sharper speaker attribution

IBM rolled out three updates to its Granite Speech family: a brand-new 2B speech-language model (granite-speech-5.0-2b), a revised 470M Turbo CTC model built for speed, and an improved 4.1-2B-Plus model tuned for multi-speaker recordings. Together they target high-accuracy transcription/translation, fast on-device recognition, and accurate speaker/timestamp labeling. All three remain under Apache 2.0.

Hugging Face
Research· Important· 🧪 Worth Testing

IBM plugin lets vLLM monitor and steer LLMs in real time with minimal slowdown

IBM Research released vLLM-Hook MIA, a plugin for the open-source vLLM inference engine that lets developers observe and intervene in a model's internal states at runtime — for example to decline malicious prompts, flag hallucinated facts, or adjust tone — without disabling vLLM's fast graph-replay execution mode. According to IBM's technical report, MIA is up to 20x faster than the earlier vLLM-Hook and up to 14x faster than methods that insert code directly into the model. In tests, steering an 8B Llama or 1.5B Qwen model added only about 2% latency, and a 70B Llama model added about 6%.

IBM Research
Models· Important· 🧪 Worth Testing

NVIDIA expands Nemotron with new retrieval, reranking and document-parsing models for AI agents

NVIDIA has added three new open, commercially licensed components to its Nemotron Retriever lineup: Embed models (8B and 1B, in BF16 and NVFP4) for searching text and images, a Rerank model (8B) that reorders retrieved evidence by relevance, and a compact 631M Parse model that converts document images into structured text. The goal is to feed AI agents with better, more relevant evidence so they need fewer search rounds and fewer inference tokens to answer a question. NVIDIA reports the new Embed and Rerank models top the ViDoRe v3.1 leaderboards for dense retrieval and reranking respectively.

Hugging Face
Models· Important· 🧪 Worth Testing

AWS and i14 ditch the KV cache with a 3B linear-attention model on Trainium2

AI research company i14 partnered with AWS to port its Laplace 3B model, a 24-layer architecture built on Gated Linear Attention (GLA) instead of standard transformer attention, onto AWS Trainium2 using native PyTorch (TorchNeuron Native). Rather than a KV cache that expands with every token, the model keeps a constant 29.5 MB recurrent state per sequence. On a single trn2.48xlarge instance, the team reached about 120,000 training tokens per second and served 11,544 output tokens per second across 1,280 simultaneous sequences through an unmodified vLLM engine.

AWS Machine Learning
Agents· Important· 🧪 Worth Testing

llm-d 1.0 released: open-source Kubernetes-native inference stack matures for production

Red Hat, IBM, and Google announced llm-d 1.0, a Kubernetes-native distributed inference serving system that moves beyond proof-of-concept performance tricks into full production readiness. The release adds multi-tenant quality of service, zero-downtime operations, and agent-aware serving so the same endpoint can handle both human users and autonomous agents efficiently. In one test, llm-d's prefix-aware router captured 97% of available cache reuse on an agentic coding workload, and IBM/Red Hat ran 3,000 concurrent software-engineering agents for 5 to 10 times less cost than using the same model as a hosted service.

IBM Research
Agents· Important· 🧪 Worth Testing

Amazon Bedrock Enforces Schema-Accurate JSON for All Claude Models

AWS has rolled out structured outputs across every Anthropic Claude model hosted on Amazon Bedrock, reaching both the standard runtime endpoint and the newer Mantle endpoint built for agentic workloads. Instead of hoping the model returns valid JSON, developers can now lock responses to an exact schema through constrained decoding, which blocks the model from generating tokens that would break the format. A companion feature, strict tool use, applies the same guarantee to function-calling arguments so agent workflows don't fail on malformed inputs.

AWS Machine Learning
Voice AI· Important· 🧪 Worth Testing

Deepgram cuts batch transcription costs by porting Nova-3 to AWS Trainium2

AWS and Deepgram jointly rebuilt the Nova-3 speech-to-text model to run natively on Trainium2 chips instead of standard GPU-based PyTorch setups, targeting large offline transcription jobs. The work involved rewriting bottleneck layers as custom kernels and replacing Deepgram's Python serving layer with a Rust runtime calling Trainium directly. The result: a single trn2.3xlarge instance running eight parallel model copies delivers 11-70% more transcribed audio per dollar than Deepgram's prior infrastructure, with similar savings potentially available to customers self-hosting on SageMaker.

AWS Machine Learning
Research· Important· 🧪 Worth Testing

AWS introduces push-based KV transfer to cut latency in disaggregated vLLM inference

AWS released a push-based NIXL connector for vLLM that changes how key-value (KV) cache data moves between prefill and decode instances in disaggregated LLM serving. Instead of the decode instance pulling KV blocks after prefill finishes, the decode instance registers its target memory in advance and the prefill instance writes the KV data directly once ready, moving coordination work off the critical path. Benchmarked on a 1-prefill/1-decode Qwen3-32B deployment using EC2 P5en instances with EFA, push mode delivered up to 3x better time-to-first-token (TTFT) and up to 30% better time-per-output-token (TPOT) than the default pull mode, especially under high request rates.

AWS Machine Learning
Models· Important· 🧪 Worth Testing

IBM combines transcription, decision-making and chat in a single Granite Switch model

IBM has released Granite Switch 4.1 in 8B and 30B sizes, a modular system that runs chat, decision-making and speech transcription from one hosted endpoint instead of separate models. The update adds a general-purpose Decision adapter, a specialized Guardian adapter for safety-related classifications, and an embedded ASR model, Granite-Speech-5.0-Turbo-CTC, which tops the OpenASR Leaderboard. Despite bundling three capabilities, the combined model uses roughly the same VRAM as the base model alone.

Hugging Face