Stochastic Gradient Descent

The Stochastic Gradient Descent (SGD) is an iterative optimization method that minimizes a loss function —the measure of a model's error— by adjusting its parameters step by step in the direction opposite to the function's gradient. The gradient points to where the error grows; moving against it drives the error down.

What sets SGD apart from classic gradient descent is how it estimates that gradient. Rather than sweeping the entire dataset at every step, SGD approximates it from a single example or a small subset. It is the workhorse of neural network training, and its roots reach back to the stochastic approximation that Herbert Robbins and Sutton Monro formalized in 1951.

From full batch to mini-batch

Batch gradient descent computes the exact gradient over every training example before taking a single step. It is precise, but with millions of data points each update becomes slow and memory-hungry. SGD flips the trade-off: it estimates the gradient from one example, so it updates the parameters far more often per pass through the data. The price is that every estimate is noisy, an imperfect approximation of the true gradient.

In practice, almost no one uses a single example or the full batch, but the middle ground: the mini-batch, which averages the gradient over tens or hundreds of examples. Léon Bottou popularized this approach for large-scale learning: it blends the speed of the stochastic method with the stability of averaging and maps neatly onto the parallel arithmetic of GPUs. When people say «SGD» today, they almost always mean mini-batch SGD.

Noise and the learning rate

SGD's noise is not merely a flaw. Its oscillations can help the process escape local minima or the flat regions where batch descent would stall —valuable on the tangled error surfaces of deep networks. Yet that same noise keeps the algorithm from settling into a minimum unless it is reined in.

The control lever is the learning rate: the size of each step. Too large, and the updates diverge or bounce without converging; too small, and training crawls. Robbins and Monro already proved that convergence requires shrinking the step gradually. Hence the schedules that lower the rate over the course of training, now a routine part of the craft.

Variants: momentum, RMSProp and Adam

On the SGD skeleton, practitioners have built variants that smooth the noise and speed up convergence. Momentum, an idea traceable to Boris Polyak's 1964 work, accumulates a running average of recent gradients to damp oscillations and gain speed along consistent directions.

Adaptive methods go further, tuning a separate step for each parameter. RMSProp, presented by Geoffrey Hinton and Tijmen Tieleman in a 2012 course, divides the step by a moving average of the gradient's recent magnitude. Adam, proposed by Diederik Kingma and Jimmy Ba in 2014, fuses that idea with momentum and adds a bias correction; it has become the default optimizer for much of deep learning. No variant dethrones SGD: all are refinements of the same idea —follow the gradient, one noisy example at a time.