Definition
Model distillation trains a student model to imitate information produced by a teacher model or ensemble. The student may learn from the teacher's output probabilities, logits, generated examples, intermediate features, or selected reasoning traces. Training often combines teacher-derived targets with the original labels.
Soft target distributions carry more information than a hard class label. If a teacher assigns 0.60 probability to cat, 0.35 to fox, and 0.05 to truck, the student can learn which mistakes the teacher considers similar. A temperature parameter can soften the distribution so those relative probabilities contribute more to the loss.
The student is often smaller or cheaper, but size reduction is a goal rather than part of the definition. Distillation can transfer an ensemble into one model, move behavior between architectures, or specialize a model for a narrower task. It can also copy the teacher's errors, biases, calibration problems, and missing edge cases.
Name and attribution
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean's 2015 paper put the name "knowledge distillation" on a temperature-based teacher-student method and popularized the term. The paper credited earlier work on compressing an ensemble into one model, so it should not be read as the origin of model compression itself.
Distinguish it from nearby terms
Distillation trains a new model. Quantization changes how an existing model's numbers are represented. Pruning removes parameters or connections. Caching reuses prior computation, and routing chooses among models that already exist.
Check your understanding
A student matches the teacher's average benchmark score but fails rare safety cases that the teacher handles. What does that tell you about the distillation data and the limits of aggregate evaluation?