Overfitting

Overfitting occurs when a machine learning model fits the training data too well—to the point of memorizing its noise and peculiarities—and, as a consequence, generalizes poorly to new, unseen data. The model learns the “exam” by heart instead of the subject, and so it fails when different questions arrive. It is the natural enemy of every model's goal: to generalize well.

How it is detected

The characteristic symptom is a gap: the error on the training data is very low, but the error on a validation or test set is high. If, during training, the loss keeps decreasing on the training set but starts to rise on the validation set, it is a sign that the model has begun to overfit.

Its opposite and the underlying trade-off

At the opposite extreme is underfitting: a model too simple to capture the pattern even in the training data. Both are the two faces of the bias-variance trade-off: underfitting corresponds to high bias (the model is rigid), and overfitting to high variance (the model is so flexible that it molds itself to the noise). The optimal point lies in balance.

How to avoid it

There is a good arsenal of remedies. Getting more data or generating it through data augmentation; using cross-validation to monitor generalization; applying regularization (L1 or L2 penalties) to curb complexity; simplifying the model; stopping training in time (early stopping); resorting to dropout in neural networks; or pruning decision trees. They all pursue the same thing: that the model learn the pattern, not the noise.

A modern nuance

The classic picture of overfitting as a simple U-shaped curve has an interesting nuance today: the phenomenon of “double descent,” observed in very large models. In certain regimes, continuing to increase the model's size beyond the point where it seemed to overfit improves performance again, a counterintuitive behavior that remains a subject of active research.