- Researchers from the Oxford Internet Institute examined 445 AI benchmarks and found that roughly half lack clear definitions of what they actually measure.
- A separate study found that popular evaluation methods can misjudge an AI agent’s true capabilities by up to 100 percent.
- Lead researcher Adam Mahdi warned that “when we ask AI models to perform certain tasks, we often actually measure completely different concepts.”
- The findings cast doubt on industry claims about models approaching human-level performance on specific tasks.
What Happened
A team led by Adam Mahdi and Andrew Bean at the Oxford Internet Institute, in partnership with over three dozen researchers from institutions including Stanford, Berkeley, and the UK’s AI Security Institute, published a study examining 445 leading AI benchmarks. Their conclusion: the tests the AI industry relies on to measure model progress routinely oversell performance and lack scientific rigor.
The study, reported in November 2025, identified several categories of methodological problems. These include data contamination where test questions appear in training data, inconsistent scoring criteria that allow partial credit for wrong answers, narrow task definitions that conflate pattern matching with genuine understanding, and benchmark saturation where tests become too easy to differentiate between models.
A separate academic paper co-authored by researchers from top universities and Amazon found that popular AI agent evaluation methods can misjudge an agent’s true capabilities by up to 100 percent. Meanwhile, a Stanford analysis of thousands of benchmarks found that approximately 5 percent contain significant flaws such as labeling errors, ambiguities, or biases, identified through a combined statistical and AI-based review framework with 84 percent precision.
Why It Matters
Benchmark scores drive billions of dollars in AI investment. When a company announces that its latest model achieves “PhD-level performance” on a reasoning test, that claim rests on the assumption that the benchmark reliably measures what it says it measures. This study suggests that assumption is frequently wrong.
“You need to really take it with a grain of salt when you hear things like ‘a model achieves PhD-level intelligence,'” Andrew Bean said. The issue is not that AI models are failing to improve. The issue is that the magnitude and nature of improvement may differ substantially from what benchmark scores indicate.
For regulators assessing AI risk based on benchmark performance, unreliable measurements could lead to either over- or under-regulation of specific capabilities. For enterprise buyers comparing models, inflated scores may obscure the actual performance gap between expensive frontier models and cheaper alternatives.
Technical Details
The researchers identified a concrete example of construct validity failure using Grade School Math 8K (GSM8K), one of the most widely cited math benchmarks. Models can produce correct numerical answers without demonstrating actual mathematical reasoning, meaning the benchmark measures pattern recognition rather than the mathematical understanding it claims to test.
“When we ask AI models to perform certain tasks, we often actually measure completely different concepts,” Adam Mahdi explained. Roughly half of the 445 benchmarks examined lacked clear definitions of the concepts they intended to measure, making it impossible to verify whether scores reflect genuine capability.
The research team proposed a framework for evaluating the evaluations themselves. This meta-benchmark approach includes adversarial testing where intentionally wrong answers are scored to verify rubrics work correctly, contamination audits that check for test data appearing in training corpora, and construct validity assessments that verify benchmarks measure intended cognitive capabilities rather than proxies.
Who’s Affected
The findings affect every stakeholder in the AI ecosystem. Model developers who optimize for benchmark performance may be chasing metrics that do not reflect real-world utility. Investors using benchmark improvements as a proxy for technical progress may be mispricing capability gaps. Enterprise customers selecting models based on published benchmark comparisons may be making decisions on unreliable data.
AI safety researchers face a particular challenge. If the benchmarks used to detect dangerous capabilities are themselves flawed, then assessments of which models pose risks and which do not become less trustworthy. Government agencies that reference benchmark scores in policy documents or regulatory frameworks inherit whatever measurement errors those benchmarks contain.
What’s Next
The research team published eight recommendations for improving benchmark methodology, including specifying evaluation scope, constructing representative task batteries, and applying statistical analysis for performance comparison. Adoption remains uncertain. Benchmark creation is decentralized across academia and industry, and no single body enforces methodological standards. Until the incentive structure changes, model developers will continue to benefit from benchmarks that produce impressive-sounding scores regardless of their scientific validity.
Related Reading
- Study Finds AI Users Adopt Incorrect Answers 73% of the Time When Models Err
- Humanity’s Last Exam Exposes the Gap Between AI Hype and Expert-Level Knowledge
- GPT-5.2 and Claude Opus 4.6 Both Go Silent on ‘Ontologically Null’ Prompts, Study Finds
- Open-Source ATLAS System on a $500 GPU Outperforms Claude Sonnet on Coding Benchmarks
- Function Calling Harness Pushes Qwen From 6.75 Percent to 100 Percent Success on Complex Schemas
