Evaluation and reliability

Grader

working definition

Definition

A component that scores, classifies, or judges an output or trajectory against a rubric or expected behavior.

Distinguish it from nearby terms

A grader produces a judgment; an oracle establishes ground truth for a property.

Check your understanding

Graders need calibration, abstention, adversarial testing, and protection from consequence-sensitive framing.