Definition
A benchmark is a defined package of tasks, data, metrics, and execution rules used to compare systems. For an agent, the execution rules matter as much as the question set: the harness, prompts, tool access, time and token budgets, retry policy, grader, and number of trials can all change the result.
A score is comparable only inside a sufficiently matched regime. If one system receives a browser, five retries, and a larger reasoning budget while another receives none of them, the benchmark is measuring two system configurations rather than isolating model capability. Good reports publish enough configuration and run-level evidence to reproduce that distinction.
Benchmarks also decay as evidence. Public tasks can leak into training or retrieval corpora, teams can tune to the leaderboard, and a fixed set can saturate. Even without leakage, stochastic agents and model-based graders add noise. OpenAI's GPT-4 report, for example, notes that contamination checks can produce false positives and false negatives. A high benchmark score should therefore be treated as evidence about a defined test, not as a certificate of production quality.
Distinguish it from nearby terms
- An evaluation is any structured measurement against criteria. A benchmark is designed to support repeated or cross-system comparison.
- A golden set is curated reference data. It may form part of a benchmark, but a benchmark also includes procedures and metrics.
- A production acceptance test measures behavior in a particular deployment context. A general benchmark may omit the tools, users, risks, and distributions that matter there.
Check your understanding
Two reports cite the same benchmark score, but one used pass@1 and the other selected the best of eight candidates. Are the results comparable? What configuration details would you need?