---
title: 'LLM as judge'
description: 'Using a language model to evaluate, compare, classify, or score outputs produced by models or agents.'
canonical_url: 'https://darkfactory.dev/glossary/llm-as-judge'
markdown_url: 'https://darkfactory.dev/glossary/llm-as-judge.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# LLM as judge


## Definition

LLM as judge is the use of a language model to evaluate, compare, classify, or score outputs produced by models or agents. Lianmin Zheng and coauthors formalized and popularized the current label in their 2023 work on MT-Bench and Chatbot Arena.

Model judges make open-ended evaluation cheaper and easier to scale, but their verdicts are approximate. They can favor an answer because of its position, length, style, apparent confidence, or similarity to the judge's own output. Prompt wording, reference answers, candidate order, and judge version can all change the result.

Validate a judge against qualified human decisions or stronger task evidence in the domain where it will be used. Randomize order, allow ties and abstention, preserve the rationale and judge configuration, and use deterministic checks for properties a program can decide exactly.

## Distinguish it from nearby terms

A grader is any mechanism that emits a score or judgment. An oracle determines a defined property authoritatively. An LLM judge is a flexible grader, not ground truth and not automatically independent when it shares the producer's model family or context.

## Check your understanding

A judge prefers the first of two equivalent answers in 70 percent of trials. Reversing candidate order reveals a position bias that one fixed comparison would hide.

## Also called

model judge, judge model

## Related factory areas

- [Verification, evaluation & quality truth](https://darkfactory.dev/factory/verification)

## Evidence and further reading

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685)
- [AgentAtlas: Control-Decision Taxonomy](https://arxiv.org/abs/2605.20530)
- [Agentic Misalignment in Summer 2026](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/)