Skip to content
Friday, 20 November 2026
Tenesys AI News
Subscribe

All News

TENESYS AI NEWS tracks the most important AI news and explains what changed, why it matters and whether a technology is worth testing.

#Benchmark — 8 articles ✕

Research· Important· 🧪 Worth Testing

MCQ benchmarks artificially mask LLM safety refusals

Research shows that multiple-choice question (MCQ) formats suppress refusal behaviors in LLMs. Experiments on Llama-3.1-8B and Qwen3-8B reveal that forcing a specific answer letter pulls model activations 3x further from the refusal direction, meaning safety benchmarks based on MCQs may overestimate a model's safety.

arXiv cs.CL
Research· 🧪 Worth Testing

New Benchmark Reveals Retrieval Accuracy Doesn't Equal Long-Context Memory

A new research paper introduces the Simple Retrieval-Reconstruction Memory (SRRM) benchmark to better evaluate long-context performance in Small Language Models (SLMs). Findings show that as context length increases, a model's ability to reconstruct information degrades much faster than its ability to retrieve specific facts, proving that standard retrieval tests are insufficient for measuring true memory.

arXiv cs.CL
Agents· Important· 🧪 Worth Testing

IBM's ALTK-Evolve boosts customer-service agents with learned prompt guidelines, no fine-tuning needed

IBM Research's ALTK-Evolve extracts short behavioral rules from an agent's own training-task attempts and appends them to the system prompt of a basic ReAct agent running on GPT-4.1. On the τ²-bench customer-service benchmark (retail and airline domains), this learned-memory approach raised accuracy and reliability without updating model weights or using human annotation. The full 'lossless' guideline set pushed a GPT-4.1 agent to a retail Pass^4 score of 67.5%, topping the October 12, 2026 leaderboard snapshot, including beating a model fine-tuned specifically on synthetic retail data. The technique also improved a frontier GPT-5.5 agent on airline tasks, taking it to the top of that leaderboard as well.

Hugging Face
Agents· Important· 🧪 Worth Testing

Cohere study: well-organized agent teams beat solo coding agents at a fraction of the cost

Cohere built a controlled testbed to study how teams of coding agents should be organized, testing where work is delegated, how leads wait on workers, which models play which role, team size, and communication patterns. On ProgramBench, well-organized teams outperformed single agents by up to ~10 percentage points but at 3–7x higher cost. The key finding: delegating only exploration and planning to workers matched the performance of delegating all stages (implementation and verification too) at half the cost, and a strong lead paired with 16 cheap planning workers reached 61.8% reward versus 63.2% for an all-strong-model team, cutting cost from $510 to $155.

Cohere
Voice AI· 🧪 Worth Testing

TII releases Falcon-ASR, a 1.6B speech recognition model focused on Arabic and Emirati dialect

The Technology Innovation Institute (TII) has released Falcon-ASR, a 1.6 billion parameter automatic speech recognition model built primarily for Arabic, with particular focus on the Emirati dialect. It also transcribes English, French, Spanish and Portuguese using the same model weights. TII reports it beats the best published leaderboard result on a standard Arabic benchmark and leads on an internal Emirati evaluation, while also including word-level timestamps.

Hugging Face
Voice AI· Important· 🧪 Worth Testing

AssemblyAI launches Universal-3.5 Pro with native code-switching and improved speaker diarization

AssemblyAI released Universal-3.5 Pro, a new flagship async speech-to-text model priced at $0.21/hr. It natively transcribes code-switched conversations across 18 languages without separate configuration, introduces jointly modeled speaker diarization built directly into the transcript (rather than stitched from a separate system), and supports contextual prompting to prime the model with domain knowledge. Benchmarks show lower word error rate on code-switched audio and higher cpWER accuracy on diarization compared to several competitor models and AssemblyAI's own prior Universal-3 Pro.

AssemblyAI
Research· Important· 🧪 Worth Testing

GitHub opens up ReviewBench to standardize how AI code reviewers get graded

GitHub has published ReviewBench, a research-preview benchmark designed to compare AI code review agents on a common, auditable standard. The dataset draws on 219 real public pull requests across 19 languages and 187 repositories, chosen to mirror the language and size distribution found across 103.9 million GitHub pull requests. Ground truth comes from a blended pool of human reviewers, static analysis tools, and multiple frontier LLMs, cross-checked by senior engineers who agreed with the benchmark's labels 96.6% of the time. GitHub says it has already used ReviewBench internally to track its own Copilot code review (CCR) product, and that offline benchmark movement reliably predicted the direction of later production A/B tests.

GitHub