Definition
Calibration asks whether stated probabilities match observed frequencies across comparable predictions. If a well-calibrated classifier assigns 0.8 probability to many cases, roughly 80 percent of those cases should be correct. Calibration is a property of a collection of forecasts, not something one confident answer can establish.
Accuracy and calibration measure different things. A system can choose the right label often while assigning probabilities that are too extreme. It can also be calibrated but uninformative: when 10 percent of cases are positive, predicting 0.1 for every case may be calibrated while doing nothing to separate one case from another.
Reliability diagrams compare predicted confidence with observed frequency. Expected calibration error summarizes binned gaps but is sensitive to how bins are chosen. Proper scoring rules such as log loss and the Brier score use the full probability forecast. Glenn Brier introduced the scoring rule later named for him in 1950. In the binary case, it is the mean squared difference between the predicted probability and the zero-or-one outcome, with lower values better.
Calibration can break under distribution shift and should be rechecked on the population where probabilities will be used. Chuan Guo and coauthors showed that post-hoc temperature scaling can improve classifier calibration on the datasets they studied. That method rescales logits for probability estimates; it is distinct from decoding temperature during text generation.
Distinguish it from nearby terms
- Confidence calibration concerns explicit numeric probabilities. A forceful natural-language rationale is not a probability estimate.
- Discrimination or ranking asks whether positive cases receive higher scores than negative cases. A model can rank well and still be miscalibrated.
- Uncertainty is broader than calibration. It includes missing knowledge, ambiguous inputs, and variation that a single probability may not capture.
Check your understanding
Among 100 predictions assigned 70 percent confidence, 50 are correct. Is that group calibrated, and what does the gap suggest?