Regularization

Regularization is the set of techniques that reduce a model's overfitting and improve its ability to generalize to new data, usually by penalizing its complexity. Its underlying idea is to prefer simpler models that capture the general patterns in the data, rather than memorizing the training noise; it is a way of applying Occam's razor.

The classic penalties: L1 and L2

The two best-known techniques add to the loss function a term that penalizes the size of the weights. L2 regularization (ridge or weight decay) penalizes the sum of their squares and shrinks them smoothly, without driving them to zero. L1 regularization (lasso) penalizes the sum of their absolute values and tends to drive some weights exactly to zero, producing sparse models, which amounts to feature selection. The combination of the two is called Elastic-Net. Primary source.

Other techniques

In neural networks there are specific techniques with the same aim. Dropout deactivates neurons at random during training so they do not co-adapt too much; early stopping halts training when the validation error stops falling; data augmentation generates new examples by transforming existing ones; and batch normalization usually has a regularizing effect too. Primary source.

The balance to be tuned

The strength of the regularization is controlled by a hyperparameter that grades the balance between fitting the data well and keeping the model simple. It is a direct expression of the bias-variance trade-off: more regularization reduces variance—and thus overfitting—at the cost of introducing some bias, and less regularization does the opposite. Primary source.