Definition
Training a smaller or otherwise cheaper student model to reproduce useful behavior from a larger teacher model or ensemble.
Distinguish it from nearby terms
Distillation transfers behavior into a model; caching reuses prior computations and routing chooses among existing models.
Check your understanding
A distilled model may not preserve rare capabilities, calibration, or safety behavior.