- The Holistic Agent Leaderboard (HAL), detailed in Kapoor et al.’s paper at ICLR 2026, spent $40,000 running 21,730 agent rollouts across 9 models and 9 benchmarks.
- A single GAIA run on a frontier model costs $2,829 before caching; Exgentic’s configuration sweep found a 33× cost spread on identical tasks driven by scaffold choice alone.
- Compression techniques that achieved 100–200× savings on static benchmarks deliver only 2–3.5× savings on agent evals, because multi-turn rollouts resist subsampling.
- In scientific ML, evaluating one new architecture on The Well benchmark requires 960 H100-hours (~$2,400); a full four-baseline sweep costs 3,840 H100-hours (~$9,600).
What Happened
A Hugging Face analysis published April 30, 2026 documents how AI evaluation has crossed a cost threshold that reshapes who can build competitive AI systems: the Holistic Agent Leaderboard, detailed in Kapoor et al.’s paper at ICLR 2026, spent approximately $40,000 to run 21,730 agent rollouts across 9 models and 9 benchmarks, with a single GAIA run on a frontier model reaching $2,829 before caching.
By April 2026, the leaderboard had grown to 26,597 rollouts. An independent reproduction by Ndzomga arrived at nearly the same figure: $46,000 across 242 agent runs.
Why It Matters
Evaluation costs have been rising since Stanford’s CRFM released HELM in 2022, when running one large model cost between $85 for OpenAI’s code-cushman-001 and $10,926 for AI21’s J1-Jumbo, with open models consuming up to 4,200 GPU-hours and the full 30-model, 42-scenario aggregate reaching roughly $100,000.
The inflection point identified by Perlitz et al. (2024) came with checkpoint-dense training pipelines. Analyzing EleutherAI’s Pythia release — which published 2,464 checkpoints across 16 model sizes — the researchers concluded that evaluation costs “may even surpass those of pretraining when evaluating checkpoints.” The growth of inference-time scaling has since amplified that dynamic: more compute at inference means more compute to measure inference.
Technical Details
The cost problem in agent evaluations is not simply a function of model pricing, and higher spend does not reliably buy better accuracy. Claude Opus 4.1 charges $15 per million input tokens and $75 per million output; Gemini 2.0 Flash charges $0.10 and $0.40 respectively — a two-order-of-magnitude spread on input tokens alone. Agent benchmarks rarely evaluate a model in isolation; they evaluate a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×.
On Online Mind2Web, Browser-Use with Claude Sonnet 4 cost $1,577 for 40% accuracy, while SeeAct with GPT-5 Medium achieved 42% accuracy for $171. The HAL paper documents “a 9× difference in cost despite just a two-percentage-point difference in accuracy.” Across six state-of-the-art agents on 300 enterprise tasks, CLEAR found that “accuracy-optimal configurations cost 4.4 to 10.8× more than Pareto-efficient alternatives” with comparable real-world performance.
The static-era compression playbook has not transferred cleanly to agents. Perlitz et al.’s analysis showed a 100–200× reduction in HELM’s compute still preserved nearly identical model rankings, and the tinyBenchmarks project compressed MMLU from 14,000 items to 100 anchor items at roughly 2% error using Item Response Theory. But Ndzomga’s mid-difficulty filter — selecting tasks with 30–70% historical pass rates — achieves only a 2–3.5× reduction on agent benchmarks, because each item is a multi-turn rollout with its own variance rather than a static prediction.
Who’s Affected
The $40,000 entry cost for a complete HAL submission places full leaderboard participation out of reach for most academic labs and early-stage AI startups, widening the gap between well-resourced organizations and independent researchers as benchmarks grow more complex. Large labs can amortize evaluation costs across training runs; smaller groups cannot.
Scientific ML faces a structurally distinct version of the same problem. The Well — a benchmark spanning 16 datasets across fluid dynamics, magnetohydrodynamics, biological systems, and supernova simulation — requires approximately 80 separate 12-hour H100 training runs to evaluate a single new neural operator, inverting the traditional assumption that training dominates compute costs in any given development cycle.
What’s Next
Proposed mitigations such as coarse-to-fine evaluation pipelines and mid-difficulty task filtering have not yet been validated at scale on agent tasks, and UK-AISI has already scaled agentic evaluation to millions of inference steps to study inference-time compute — a signal that benchmark budgets will continue to grow before standard compression methods are established.
OpenAI’s MLE-Bench, which runs 75 Kaggle-style competitions each requiring 24 hours on a single A10 GPU, estimates a single-seed run at roughly $5,500 in combined GPU and API costs; a full three-seed, six-model sweep approaches $100,000. As training-in-the-loop benchmarks proliferate, the cost structure of evaluation is converging with the cost structure of training itself.
