Gradient Descent

Gradient descent is an iterative optimization algorithm that minimizes a function by moving, step by step, in the direction opposite to its gradient.

The mechanics

The gradient of a function points in the direction of steepest increase; that is why, to minimize, one must move in the opposite direction. At each iteration, the algorithm updates the parameters with a simple rule: new parameters = parameters − (learning rate) × gradient. By repeating that step, it descends the loss surface toward a minimum.

The learning rate

The most delicate hyperparameter is the learning rate, the size of the step. If it is too large, the algorithm overshoots the minimum and may oscillate or diverge; if it is too small, it converges with exasperating slowness. Setting it well is a central practical problem, and schedules that gradually reduce it over training are often used.

Its variants

Depending on how much data they use per step, there are three versions. Batch gradient descent computes the gradient over the whole dataset: stable, but slow. Stochastic (SGD) uses a single example per step: fast, but noisy. And mini-batch, which uses small batches, balances the two and is the standard. On this basis, extensions are built such as momentum, which accelerates the descent, and adaptive optimizers such as Adam, which tunes each parameter's step. Documentation: technical gradient-descent guide; original Adam paper.

Its limits

The method has weak points. In non-convex functions—like those of a neural network—it can get trapped in local minima or slow down near saddle points, which in high dimensions are more frequent than bad minima. And it is sensitive to the scale of the features: hence the advice to normalize the data before training. Documentation: saddle-point analysis.