Learning Rate
The learning rate sets the step size of gradient descent: too high and training diverges, too low and it crawls. We cover its role, schedules like warmup and cosine annealing, and adaptive methods such as Adam.
The learning rate is the hyperparameter that sets the step size in gradient descent: it determines how much a model's parameters are adjusted at each update. On every iteration the algorithm computes the gradient of the loss function and multiplies it by the learning rate to decide the size of the change, which is why the optimization literature also calls it the «step size».
In their textbook Deep Learning, Goodfellow, Bengio and Courville call it «perhaps the most important hyperparameter» when training a neural network, because it directly governs whether training converges, does so stably, or fails outright.
What happens when it is too high or too low
The right value lives in a narrow balance. If the rate is too high, each step overshoots the minimum of the loss function: the optimization oscillates back and forth or diverges, and the loss may blow up instead of falling. If it is too low, each step moves so little that training becomes painfully slow and burns time and compute; the model can also stall on plateaus or settle into poor local minima with no momentum to escape. One nuance that popular explanations often get wrong is worth stating: in high-dimensional networks the more common obstacles are not so much local minima as saddle points and flat regions of the loss landscape.
Learning rate schedules
A fixed rate is rarely kept for the whole training run. Schedules vary it over time. Classic schedules apply decay: reducing it in steps (step decay) or exponentially so that training takes larger steps early and finer ones later. Warmup does the opposite at the beginning, starting from a small value and gradually raising it to the target. There is no single recipe. The original Transformer used 4,000 warmup steps followed by inverse-square-root decay. SGDR, by Loshchilov and Hutter, proposed cosine annealing: the rate follows a half-wave from a maximum to a minimum and warm restarts return it to the peak. GPT-3 documented linear warmup followed by cosine decay. These are dated examples of the mechanisms, not a rule for every model.
Adaptive methods and how to choose it
Adaptive methods tune a separate effective step for each parameter from the history of gradients. AdaGrad, by Duchi and colleagues, was the pioneer; RMSProp, proposed by Geoffrey Hinton, uses a moving average of squared gradients; and Adam, by Kingma and Ba, combines that idea with momentum. It is worth remembering something many texts blur: these methods do not remove the learning rate, they still depend on a base learning rate that must be set as a hyperparameter. To choose that value, Leslie N. Smith described a learning rate range test: run training while the rate increases between two bounds and record accuracy and loss against each value. The sweep helps propose boundaries for a later experiment; by itself, it does not guarantee the best rate for every model.
Pieces using this term
- How to read an AI paper without being a researcher (2026-07-26)
- What it actually takes to train your own model: the math nobody shows you (2026-07-25)
- Data Centers in Space: The Physics That Will Decide If AI Moves to Orbit (2026-07-13)
- Andrej Karpathy leaves OpenAI to focus on personal projects (2024-02-13)
- How a Deep Network Learns: From Forward Pass to Gradient (2023-05-09)
- Machine Learning from Scratch: Seven Decisions Behind a Model (2023-05-09)
- Transfer Learning and Semi-Supervised Learning in Language Models (2023-05-09)
- Tools and Techniques for Training Custom Language Models (2023-05-09)
This article was produced with artificial intelligence under human editorial oversight.
