Skip to content
Friday, 20 November 2026
Tenesys AI News
Subscribe

All News

TENESYS AI NEWS tracks the most important AI news and explains what changed, why it matters and whether a technology is worth testing.

#Benchmark — 11 articles ✕

Research· Important· 🧪 Worth Testing

MCQ benchmarks artificially mask LLM safety refusals

Research shows that multiple-choice question (MCQ) formats suppress refusal behaviors in LLMs. Experiments on Llama-3.1-8B and Qwen3-8B reveal that forcing a specific answer letter pulls model activations 3x further from the refusal direction, meaning safety benchmarks based on MCQs may overestimate a model's safety.

arXiv cs.CL
Research· 🧪 Worth Testing

New Benchmark Reveals Retrieval Accuracy Doesn't Equal Long-Context Memory

A new research paper introduces the Simple Retrieval-Reconstruction Memory (SRRM) benchmark to better evaluate long-context performance in Small Language Models (SLMs). Findings show that as context length increases, a model's ability to reconstruct information degrades much faster than its ability to retrieve specific facts, proving that standard retrieval tests are insufficient for measuring true memory.

arXiv cs.CL

Ai2 ships open satellite embedding model OlmoEarth v1.3 with free global map explorer

Ai2 has published OlmoEarth v1.3, an embedding model that condenses a full year of radar and multispectral satellite data into a single compact vector for every 10-meter patch of the planet. Alongside it comes a free browser tool, the OlmoEarth Embeddings Explorer, that lets anyone compare landscapes across years or train tiny classifiers without touching a GPU. Benchmarked against AlphaEarth Foundations and Tessera v2 across 27 tests, the new model comes out on top overall while costing far less to train than its predecessors.

Hugging Face
Research· Important

Allen Institute Launches FairStress Benchmark to Expose Hidden Bias in LLM Decisions

Allen Institute for AI (Ai2) released FairStress, a benchmark that tests demographic fairness in LLMs not just by their final decisions but by how easily those decisions change under persuasive pressure. Testing open and frontier models including Olmo, Llama, Qwen, Gemma, Claude, and GPT found that models often abandon correct, well-supported decisions more readily when doing so favors a minority-coded candidate — a pattern the authors call 'anti-stereotypical over-correction.' The benchmark covers 300 scenarios across 29 domains (hiring, lending, healthcare, education, housing, criminal justice), expanded via demographic and persuasion variations into over 13 million test cases, plus a smaller 50K-item 'FairStress-Core' subset for faster evaluation.

Hugging Face
Agents· Important· 🧪 Worth Testing

IBM's ALTK-Evolve boosts customer-service agents with learned prompt guidelines, no fine-tuning needed

IBM Research's ALTK-Evolve extracts short behavioral rules from an agent's own training-task attempts and appends them to the system prompt of a basic ReAct agent running on GPT-4.1. On the τ²-bench customer-service benchmark (retail and airline domains), this learned-memory approach raised accuracy and reliability without updating model weights or using human annotation. The full 'lossless' guideline set pushed a GPT-4.1 agent to a retail Pass^4 score of 67.5%, topping the October 12, 2026 leaderboard snapshot, including beating a model fine-tuned specifically on synthetic retail data. The technique also improved a frontier GPT-5.5 agent on airline tasks, taking it to the top of that leaderboard as well.

Hugging Face
Cyber Security· Important

AWS's autonomous pentest agent exploits 33 of 37 fresh CVEs end-to-end

AWS tested its AWS Security Agent (now part of AWS Continuum) against a freshly built benchmark of 37 real, recently disclosed CVEs, each run as a live containerized target with a hidden flag. Starting from a low-privilege endpoint with only a one-line hint and no proof-of-concept, the agent, running on Claude Opus 5, captured 33 of 37 flags through genuine end-to-end exploitation after an integrity audit, with 2 captures reclassified as unintended-path solves and 2 genuine misses where the agent broke the target before extracting the flag. AWS built this benchmark to avoid the contamination problem of public CVE suites, which models may have memorized from training data.

AWS Security
Agents· Important· 🧪 Worth Testing

Cohere study: well-organized agent teams beat solo coding agents at a fraction of the cost

Cohere built a controlled testbed to study how teams of coding agents should be organized, testing where work is delegated, how leads wait on workers, which models play which role, team size, and communication patterns. On ProgramBench, well-organized teams outperformed single agents by up to ~10 percentage points but at 3–7x higher cost. The key finding: delegating only exploration and planning to workers matched the performance of delegating all stages (implementation and verification too) at half the cost, and a strong lead paired with 16 cheap planning workers reached 61.8% reward versus 63.2% for an all-strong-model team, cutting cost from $510 to $155.

Cohere
Voice AI· 🧪 Worth Testing

TII releases Falcon-ASR, a 1.6B speech recognition model focused on Arabic and Emirati dialect

The Technology Innovation Institute (TII) has released Falcon-ASR, a 1.6 billion parameter automatic speech recognition model built primarily for Arabic, with particular focus on the Emirati dialect. It also transcribes English, French, Spanish and Portuguese using the same model weights. TII reports it beats the best published leaderboard result on a standard Arabic benchmark and leads on an internal Emirati evaluation, while also including word-level timestamps.

Hugging Face
Voice AI· Important· 🧪 Worth Testing

AssemblyAI launches Universal-3.5 Pro with native code-switching and improved speaker diarization

AssemblyAI released Universal-3.5 Pro, a new flagship async speech-to-text model priced at $0.21/hr. It natively transcribes code-switched conversations across 18 languages without separate configuration, introduces jointly modeled speaker diarization built directly into the transcript (rather than stitched from a separate system), and supports contextual prompting to prime the model with domain knowledge. Benchmarks show lower word error rate on code-switched audio and higher cpWER accuracy on diarization compared to several competitor models and AssemblyAI's own prior Universal-3 Pro.

AssemblyAI
Research· Important· 🧪 Worth Testing

GitHub opens up ReviewBench to standardize how AI code reviewers get graded

GitHub has published ReviewBench, a research-preview benchmark designed to compare AI code review agents on a common, auditable standard. The dataset draws on 219 real public pull requests across 19 languages and 187 repositories, chosen to mirror the language and size distribution found across 103.9 million GitHub pull requests. Ground truth comes from a blended pool of human reviewers, static analysis tools, and multiple frontier LLMs, cross-checked by senior engineers who agreed with the benchmark's labels 96.6% of the time. GitHub says it has already used ReviewBench internally to track its own Copilot code review (CCR) product, and that offline benchmark movement reliably predicted the direction of later production A/B tests.

GitHub