Definition
A grader is the part of an evaluation that judges a model's output, action sequence, or final outcome against stated criteria. It may be a deterministic check, such as compiling code or comparing a numeric result; a model-based judge applying a rubric; or a structured human review. One evaluation can combine several graders.
The grader is part of the measurement system, so its errors propagate into every reported score. A brittle string check can reject valid alternatives. A model judge can favor style, recognize the system under test, or change behavior when told that its label has consequences. Human graders can disagree or drift. Validate graders against reviewed examples, report disagreement, allow abstention where appropriate, and keep the grading prompt and version with the result.
Whenever possible, prefer checks tied to the real outcome: tests pass, the requested file changed, the transaction reconciled, or the user goal was met. Rubric scores are useful when the outcome cannot be measured directly, but they should not quietly replace it.
Distinguish it from nearby terms
- An oracle supplies an authoritative answer for a property. A grader may consult an oracle, apply a rubric, or estimate quality without one.
- A verifier checks whether a candidate satisfies a claim or constraint. A grader often returns a score for evaluation; the roles can overlap.
- A critic provides feedback intended to improve an answer. It is not automatically a valid grader.
Check your understanding
A model-based grader gives one system higher scores than expert reviewers do. Which grader artifacts and agreement statistics would you inspect before accepting the leaderboard?