Definition
An evaluation, often shortened to eval, is a systematic measurement of model or system behavior against defined tasks, criteria, datasets, or operational outcomes. It should identify the object being measured, the conditions of the test, the scoring rule, and the decision the result will inform.
The object matters. A model-only evaluation measures learned capability under a particular prompt. An agent evaluation also measures tools, harness, environment, budget, and control flow. A production evaluation includes real users, changing data, latency, cost, and effects that a benchmark may omit.
Strong evaluation covers representative work, important failures, and slices where performance may differ. It preserves the test data, model and harness versions, judge settings, repetitions, and uncertainty needed to reproduce the result.
Distinguish it from nearby terms
Evaluation is the measurement process. A benchmark is a standardized dataset or protocol used for comparison. Verification checks whether a particular artifact satisfies its requirements, which is a different decision from estimating general performance.
Check your understanding
A team reports "the model scored 80 percent," but the run used retrieval, five tools, a custom verifier, and three retries. The score belongs to that evaluated system configuration, not the model alone.