Definition
Regularization changes the learning problem to improve performance on unseen data, often at the cost of a worse fit to the training sample. It encodes a preference for one solution over another when both explain the observed examples.
Explicit regularization adds a term to the objective. L2 penalties discourage large weights; L1 penalties favor sparse weights. Other methods change the training procedure instead. Data augmentation perturbs inputs, dropout randomly removes units during training, early stopping limits how long the optimizer fits the sample, and label smoothing changes the target distribution.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov's 2014 dropout paper randomly omitted units and their connections during training, then used the full network at inference with adjusted activations. The paper established a widely used neural-network regularizer, but regularization as a general idea is much older.
Regularization strength is a hyperparameter. Too little may leave a large generalization gap. Too much can cause underfitting by preventing the model from learning useful structure. The right setting depends on data volume, noise, architecture, objective, and the target distribution, so it must be selected with representative validation data.
Distinguish it from nearby terms
Regularization changes training. Validation measures held-out performance and helps choose settings; it does not itself constrain the model. Data augmentation is regularization when its transformations preserve the task's intended label, but harmful transformations can introduce bias instead.
Check your understanding
Training loss rises after stronger regularization, while error on a clean validation set falls. Why can that be the desired result, and what would show that the setting has crossed into underfitting?