Definition
Training data comprises the examples and signals used to fit a model's learned parameters. It can include observations, labels, demonstrations, preference comparisons, rewards, synthetic examples, and transformed derivatives. Pretraining, fine-tuning, distillation, and reinforcement learning use different data forms, but each can change model weights.
The effective training distribution depends on more than the files collected. Sampling, filtering, deduplication, weighting, ordering, augmentation, and tokenization determine which examples the optimizer sees and how often. A large corpus can therefore have narrow effective coverage or repeat a small set of patterns.
Training-data lineage should record sources, collection method, consent and licensing, transformations, labeling, known gaps, retention, and version. Timnit Gebru and coauthors' Datasheets for Datasets proposal supplies a structured set of questions for documenting motivation, composition, collection, preprocessing, distribution, maintenance, and recommended use. A datasheet improves traceability but does not prove that the data is lawful, representative, or fit for a particular model.
Train-test overlap can inflate evaluation. Poisoned examples can introduce targeted behavior. Stale or unrepresentative data can produce failures even when average training loss is low. These risks require data-level checks and model evaluation because the final weights do not expose a readable inventory of their causes.
Distinguish it from nearby terms
A dataset is a collection that may serve several roles. Training data updates parameters. Validation data guides choices such as hyperparameters or stopping. Test data estimates performance after development decisions. Runtime prompts and retrieval corpora supply context without becoming training data unless a later learning process uses them for weight updates.
Check your understanding
A production-feedback table is copied into next month's fine-tuning run after users requested deletion of some records. When did the table become training data, and which lineage and deletion controls must follow it into the derived dataset?