MCQ benchmarks artificially mask LLM safety refusals
In short: Research shows that multiple-choice question (MCQ) formats suppress refusal behaviors in LLMs. Experiments on Llama-3.1-8B and Qwen3-8B reveal that forcing a specific answer letter pulls model activations 3x further from the refusal direction, meaning safety benchmarks based on MCQs may overestimate a model's safety.
This summary was generated automatically by AI from arXiv cs.CL's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1Mechanistic evidence shows MCQ formatting triggers rapid answer selection, competing with safety refusal mechanisms.
- 2Activations in MCQ format are 3x further from the refusal vector compared to open-ended queries.
- 3Ablating attention heads associated with MCQ answers increases refusal rates.
Why it matters
Companies relying on MCQ-based safety scores to validate their models are likely getting a false sense of security. True safety testing requires open-ended, adversarial evaluation rather than standardized multiple-choice formats.
What it means for AI agents and contact centers
When evaluating or fine-tuning models for safety-critical voice agents, do not rely on MCQ benchmarks alone. Use adversarial, open-ended tests to ensure the agents maintain refusal behavior when they lack sufficient information or are faced with harmful intent.
🧪 Worth Testing
Testing model safety with open-ended inputs instead of MCQs provides a realistic view of how your agents perform in unpredictable live call environments.
Sources
- arXiv cs.CLOfficialPrimary sourceOriginal article →„Forcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDP“9 Nov 2026, 07:00Licence: arXiv metadata CC0; papers under their own licences · our summary (content changed)
- Published by source
- 9 Nov 2026, 07:00
- Found by our system
- 9 Nov 2026, 07:20
- Summary generated
- 9 Nov 2026, 07:20
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy