Definition
Cross-validation is a resampling method for estimating how a model will generalize. In common k-fold cross-validation, the data is divided into k subsets. The model is trained k times, each time holding out a different fold for evaluation, and the results are summarized across folds.
The split must follow the structure of the real prediction problem. Records from the same person, organization, document, or time period can leak information across folds. Time-dependent problems usually need forward-looking splits rather than random shuffling.
Cross-validation helps compare approaches when data is limited. If it is used repeatedly for model selection, the process can overfit to the folds, so an untouched final test set may still be needed.
Distinguish it from nearby terms
A single train-test split produces one estimate. Cross-validation rotates the held-out subset. A held-out test set remains untouched until the final estimate, while validation folds may influence model selection.
Check your understanding
Customer tickets from the same conversation appear in both training and validation folds. The model recognizes repeated language, so the cross-validation score overstates performance on new conversations.