Editor's Synopsis
Ethan Mollick critiques a new hallucination benchmark for AI models, arguing that it ultimately measures refusal thresholds rather than actual hallucination rates. He notes that the benchmark's trivia questions are "incredibly specific and nearly impossible without web lookup," suggesting the test essentially evaluates how willing a model is to attempt answering obscure questions given a set system prompt, rather than how accurately it generates information.
This is a meaningful distinction in the AI evaluation space — a model that refuses to answer difficult questions will appear to hallucinate less, but that doesn't necessarily mean it's more truthful when it does respond. Mollick's observation highlights the ongoing challenge of designing meaningful AI benchmarks that capture genuine performance rather than surface-level behavioral patterns.