What is Benchmark?
Also called evaluation, eval.
A benchmark is a standardized set of tasks used to measure and compare model capability, such as graduate-level science questions, competition mathematics, or resolving real software issues. Scores from these sets are widely quoted in model announcements. They are useful for coarse comparison and are weak predictors of performance on any particular real workload.
Benchmarks differ in what they actually measure. Some test factual recall through multiple choice questions, some test multi-step reasoning, some require producing working code that passes hidden tests, and some rely on human or model judges scoring open-ended responses. Judged benchmarks introduce their own bias, since judges tend to favor length, tidy formatting, and familiar style.
Contamination is the central methodological problem. When benchmark questions appear somewhere in the pre-training corpus, a score reflects memorization rather than capability. Because training data is scraped from the public web where benchmarks are published and discussed, contamination is very hard to rule out, which is why held-out and continuously refreshed benchmarks have gained importance.
Comparison across announcements is harder than it looks. The same benchmark can be run with different prompts, different numbers of examples, different reasoning budgets, and different scoring scripts, and every one of those choices moves the result. Two quoted numbers are only comparable if the evaluation setup matched, which published figures rarely state clearly enough to confirm.
For practical decisions, an internal evaluation set built from real inputs and scored against real requirements predicts deployed quality far better than any public leaderboard. Public benchmarks are best used to build a shortlist of candidates worth trying, after which the shortlist gets tested on the workload that the system will actually face in production.
Key points
- Standardized task sets used to compare models coarsely.
- Contaminated benchmarks measure memorization rather than capability.
- Scores shift with prompt, examples, reasoning budget, and scoring script.
- Judged benchmarks reward length, formatting, and familiar style.
- Internal evaluations on real inputs predict deployed quality better.
In practice
Two models are announced within a month of each other. One reports eighty-eight percent on a reasoning benchmark, the other eighty-six, so a team adopts the first. On their own set of four hundred real support tickets, scored for correct routing and policy citation, the second model wins clearly, because their workload rewards instruction following rather than puzzle solving.