Amazon has released the Universal Agent Evaluation Framework (UAEF), an open-source tool designed to standardize the evaluation of AI agents. It uses adapters to normalize execution traces from various frameworks like LangChain and LangGraph into a single format, enabling consistent measurement across 36 metrics and 7 quality dimensions.
Amazon Bedrock AgentCore now includes a comprehensive catalog of built-in, third-party, and custom evaluators to monitor agent behavior in production. The platform allows you to run metrics from DeepEval and AutoEval in a managed environment, supporting on-demand, online, and batch evaluation workflows.
A new research paper introduces the Simple Retrieval-Reconstruction Memory (SRRM) benchmark to better evaluate long-context performance in Small Language Models (SLMs). Findings show that as context length increases, a model's ability to reconstruct information degrades much faster than its ability to retrieve specific facts, proving that standard retrieval tests are insufficient for measuring true memory.
NVIDIA introduced a workflow for turning production agent failures into structured test cases using NeMo Compass and NeMo Eval Author. This process analyzes trace logs to identify recurring issues, generates targeted evaluations, and validates harness improvements.
Google has introduced Physical-Engineering's Last Exam (PLE), a new benchmark designed to test AI agents' capabilities in mechanical, electrical, civil, and structural engineering. Unlike standard textbook tests, PLE requires agents to process large project folders, including CAD drawings and building codes, to produce compliant, safe deliverables.
Google released a new Agents CLI evaluation namespace designed to help developers perform 'hill-climbing'—an iterative process of improving agent performance through testing, grading, and analysis. The tool enables regression tracking by comparing agent traces across model swaps or prompt changes, surfacing issues like tool-use regressions that standard unit tests miss.
Researchers introduced Recursive Semantic Divergence (RSD), an unsupervised metric that measures and penalizes inconsistency in LLM agent trajectories. By applying RSD as a reward signal during fine-tuning, models exhibit significantly more stable behavior and higher success rates across tool-use benchmarks.