Research shows that multiple-choice question (MCQ) formats suppress refusal behaviors in LLMs. Experiments on Llama-3.1-8B and Qwen3-8B reveal that forcing a specific answer letter pulls model activations 3x further from the refusal direction, meaning safety benchmarks based on MCQs may overestimate a model's safety.
AI agents frequently misinterpret legacy code comments, such as TODOs or FIXMEs, as current instructions. Research shows that 6% of coding sessions result in agents chasing irrelevant, closed, or obsolete bug tickets, leading to increased costs and scope creep.
Researchers have identified that LLMs suffer from a bias toward providing immediate, often incorrect answers in ambiguous situations instead of asking clarifying questions. They introduced BACE (Belief-Aware Critic with Epistemic tracking), a lightweight method that runs on top of a frozen language model to strategically decide when to request clarification.
Researchers introduced SLATE, a benchmark for evaluating long-horizon, large-scale API selection in enterprise software environments. Unlike current benchmarks, it forces models to navigate roughly 1,000 tools to complete complex tasks.
Google has introduced Physical-Engineering's Last Exam (PLE), a new benchmark designed to test AI agents' capabilities in mechanical, electrical, civil, and structural engineering. Unlike standard textbook tests, PLE requires agents to process large project folders, including CAD drawings and building codes, to produce compliant, safe deliverables.
Apple researchers have unveiled an online training infrastructure for mobile UI agents that utilizes a two-part architecture to split navigational logic from visual interpretation. By employing step-level feedback provided by an LLM-as-a-Judge and a novel value estimation technique known as GiGPPO, these agents demonstrate significantly better adaptability than static models trained on pre-recorded sequences.
Recent research reveals that language models fail to adapt to evolving user preferences, instead defaulting to 'exploit-first' behavior. While models can identify the correct strategy in multiple-choice scenarios, they show a 35-46% drop in calibration during open-ended generation, persistently over-relying on stale historical data.
Researchers introduced Recursive Semantic Divergence (RSD), an unsupervised metric that measures and penalizes inconsistency in LLM agent trajectories. By applying RSD as a reward signal during fine-tuning, models exhibit significantly more stable behavior and higher success rates across tool-use benchmarks.
A new study questions the common belief that whether an AI agent falls for a prompt injection is settled the moment it reads the malicious text. The researchers built a branching-rollout technique on AgentDojo benchmarks: they replayed an agent's run up to the point it encounters an injected instruction, then let the rest of the episode play out many different ways to measure how much of the final outcome was already locked in versus still open. For a Qwen3 agent running without extended reasoning, 91% of the outcome was already determined at that moment; switching on reasoning ('thinking') cut that figure to 62%, showing the agent's compliance decision often keeps forming well after it first sees the attack. Enabling reasoning also cut Qwen3's susceptibility to successful attacks, but made two other tested models more vulnerable, and benchmark scores for reasoning agents flipped on a rerun as often as 35% of the time.
Google Research describes new work on grounding AI agents in physical lab reality, using a digital-twin framework called VERID to let agents safely plan and execute liquid-handling robot protocols. Combined with closed-loop molecular design cycles for aptamers and proteins, the agents iteratively learned from wet-lab results, improving hit rates over multiple rounds and discovering single-digit nanomolar protein binders for a cancer-related target.
IBM Research used a coordinated multi-agent 'agent factory' to explore hardware design optimizations for a quantum error-correction decoder, comparing proprietary and open frontier models under the same 100-hypothesis search budget. The open-model configuration, run on-premises via IBM's llm-d distributed inference framework, achieved a 52x speedup over baseline for about $26 in inference cost, versus $299 for a proprietary-model configuration that reached a higher 204x speedup. The study highlights that cache-aware routing and efficient serving (98% prefix-cache hit rate, ~$0.11 per million tokens) made the open-model approach dramatically cheaper while still finding valid, compact hardware designs.
GitHub has published ReviewBench, a research-preview benchmark designed to compare AI code review agents on a common, auditable standard. The dataset draws on 219 real public pull requests across 19 languages and 187 repositories, chosen to mirror the language and size distribution found across 103.9 million GitHub pull requests. Ground truth comes from a blended pool of human reviewers, static analysis tools, and multiple frontier LLMs, cross-checked by senior engineers who agreed with the benchmark's labels 96.6% of the time. GitHub says it has already used ReviewBench internally to track its own Copilot code review (CCR) product, and that offline benchmark movement reliably predicted the direction of later production A/B tests.