---
title: Grader
description: 'A deterministic check, model, or human protocol that judges an output or trajectory against stated criteria.'
canonical_url: 'https://darkfactory.dev/glossary/grader'
markdown_url: 'https://darkfactory.dev/glossary/grader.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Grader


## Definition

A grader is the part of an evaluation that judges a model's output, action sequence, or final outcome against stated criteria. It may be a deterministic check, such as compiling code or comparing a numeric result; a model-based judge applying a rubric; or a structured human review. One evaluation can combine several graders.

The grader is part of the measurement system, so its errors propagate into every reported score. A brittle string check can reject valid alternatives. A model judge can favor style, recognize the system under test, or change behavior when told that its label has consequences. Human graders can disagree or drift. Validate graders against reviewed examples, report disagreement, allow abstention where appropriate, and keep the grading prompt and version with the result.

Whenever possible, prefer checks tied to the real outcome: tests pass, the requested file changed, the transaction reconciled, or the user goal was met. Rubric scores are useful when the outcome cannot be measured directly, but they should not quietly replace it.

## Distinguish it from nearby terms

- An oracle supplies an authoritative answer for a property. A grader may consult an oracle, apply a rubric, or estimate quality without one.
- A verifier checks whether a candidate satisfies a claim or constraint. A grader often returns a score for evaluation; the roles can overlap.
- A critic provides feedback intended to improve an answer. It is not automatically a valid grader.

## Check your understanding

A model-based grader gives one system higher scores than expert reviewers do. Which grader artifacts and agreement statistics would you inspect before accepting the leaderboard?

## Also called

evaluator

## Related terms

- [Verification loop](https://darkfactory.dev/glossary/verification-loop)
- [Oracle](https://darkfactory.dev/glossary/oracle)

## Related factory areas

- [Verification, evaluation & quality truth](https://darkfactory.dev/factory/verification)

## Evidence and further reading

- [AgentAtlas: Control-Decision Taxonomy](https://arxiv.org/abs/2605.20530)
- [Agentic Misalignment in Summer 2026](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/)
- [Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [Where Does Agent Reliability Come From?](https://arxiv.org/abs/2607.17044)
- [The Art of Loop Engineering: How to Build Agents That Improve Over Time](https://www.youtube.com/watch?v=jPPiZ22DY3g)
- [smevals: a small eval suite for evaluating models, prompts, and harnesses](https://primeradiant.com/blog/2026/smevals.html)