Definition
Pretraining is training performed before a later adaptation stage. Its purpose is usually to learn representations or capabilities that can be reused across tasks, domains, or deployments. The word describes the model's place in a training sequence, not one objective or dataset.
Language-model pretraining often uses self-supervised prediction on large text corpora. A causal model predicts later tokens from earlier context; a masked model reconstructs selected hidden tokens. Vision systems may use labels, contrastive objectives, reconstruction, or multimodal pairs. Pretraining can therefore be supervised or self-supervised, broad or domain-specific.
The pretrained checkpoint carries more than general knowledge. It also carries the effects of data selection, tokenization, objective design, architecture, and optimization. Later fine-tuning can redirect behavior but does not recreate a clean model independent of those choices. Evaluation data found in the pretraining corpus can also make benchmark results hard to interpret even when the downstream team never trained on that benchmark directly.
Language-model adoption
Alec Radford and colleagues' 2018 paper "Improving Language Understanding by Generative Pre-Training" paired left-to-right Transformer pretraining with task-specific fine-tuning. BERT used masked-language-model pretraining and fine-tuned the resulting encoder for downstream tasks later that year. Both were influential adoption points in modern NLP, but neither paper invented the general practice of training reusable representations before a target task.
Distinguish it from nearby terms
Fine-tuning continues weight training for a narrower purpose. Instruction tuning is a type of fine-tuning built around task instructions and desired responses. Prompting and retrieval change runtime context without another pretraining phase.
Check your understanding
A model is pretrained on legal and general web text, then fine-tuned for contract classification. Which stage supplies the reusable language representation, and which stage targets the classification task? What evidence would reveal benchmark contamination from the earlier stage?