Learning Rate

The learning rate is the hyperparameter that sets the step size in gradient descent: it determines how much a model's parameters are adjusted at each update. On every iteration the algorithm computes the gradient of the loss function and multiplies it by the learning rate to decide the size of the change, which is why the optimization literature also calls it the «step size».

In their textbook Deep Learning, Goodfellow, Bengio and Courville call it «perhaps the most important hyperparameter» when training a neural network, because it directly governs whether training converges, does so stably, or fails outright.

What happens when it is too high or too low

The right value lives in a narrow balance. If the rate is too high, each step overshoots the minimum of the loss function: the optimization oscillates back and forth or diverges, and the loss may blow up instead of falling. If it is too low, each step moves so little that training becomes painfully slow and burns time and compute; the model can also stall on plateaus or settle into poor local minima with no momentum to escape. One nuance that popular explanations often get wrong is worth stating: in high-dimensional networks the more common obstacles are not so much local minima as saddle points and flat regions of the loss landscape.

Learning rate schedules

A fixed rate is rarely kept for the whole run. Schedules vary it over time. The classic ones apply decay: cutting it in steps (step decay) or exponentially as training progresses, so the model takes large strides early and fine-tunes later. Warmup does the opposite over the first iterations, starting from a very small value and ramping up to the target, which stabilizes the start of large models. Loshchilov and Hutter popularized cosine annealing in their SGDR work: the rate follows a half-cosine curve from a maximum down to a minimum, with optional periodic restarts that reset it to the peak to re-explore. Large language models commonly pair warmup with a cosine decay.

Adaptive methods and how to choose it

Adaptive methods tune a separate effective step for each parameter from the history of gradients. AdaGrad, by Duchi and colleagues, was the pioneer; RMSProp, proposed by Geoffrey Hinton, uses a moving average of squared gradients; and Adam, by Kingma and Ba, combines that idea with momentum. It is worth remembering something many texts blur: these methods do not remove the learning rate, they still depend on a base learning rate that must be set as a hyperparameter. To choose that value, Leslie N. Smith proposed the learning rate range test: train for a few epochs while increasing the rate linearly from a low value to a high one, then watch the range where the loss falls fastest before it starts to blow up. This simple sweep replaces much of the trial and error and remains the most widely used practical way to bracket a good starting value.