Measuring the cost of LLM refusals via existing logs
In short: A new causal method named RAW (Refusal Aversion in the Wild) uses LLM stochasticity to measure how often user refusals lead to platform abandonment. By analyzing existing interaction logs, you can identify the exact causal effect of refusals on subsequent user re-engagement.
This summary was generated automatically by AI from arXiv cs.CL's publication. It is our own text, not a copy of the original — facts, figures and quotes belong to the source, linked above and below.
What changed?
- 1Introduces RAW methodology for causal impact measurement of refusals.
- 2Requires only interaction logs for execution.
- 3Costs approximately $0.30 per 1,000 conversations to run.
- 4Demonstrates that refusals cause a 3–17 pp drop in re-engagement for many models.
Why it matters
Understanding the 'refusal tax' on user experience is critical for production systems. This method lets you quantify whether over-sensitive safety guardrails are actively driving users away from your platform.
What it means for AI agents and contact centers
This is a valuable analytical tool for evaluating how your AI agent's refusal behavior affects conversation retention. By implementing this across your logs, you can benchmark if specific safety triggers correlate with churn and refine your guardrail thresholds accordingly.
🧪 Worth Testing
It allows you to turn raw interaction logs into actionable data about user experience quality and retention, which is crucial for optimizing AI agents.
Sources
- arXiv cs.CLOfficialPrimary sourceOriginal article →„RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs“18 Nov 2026, 07:00Licence: arXiv metadata CC0; papers under their own licences · our summary (content changed)
- Published by source
- 18 Nov 2026, 07:00
- Found by our system
- 18 Nov 2026, 07:23
- Summary generated
- 18 Nov 2026, 22:59
This article was written by AI from the original source. Facts, numbers and prices come from the source; missing values are marked “Not specified”. Legal notice, copyright and privacy