Cosine Restart (Warm Restarts)

Cosine restart—known in the literature as cosine annealing with warm restarts or SGDR—is a strategy for scheduling the learning rate during the training of a neural network. Ilya Loshchilov and Frank Hutter proposed it in 2017. It combines two ideas: a cosine-shaped decay and periodic restarts.

How it works

The cosine decay reduces the learning rate along a cosine curve, from a maximum value at the start of each cycle to a minimum at the end. The warm restarts mean that, at the end of a cycle, the rate jumps abruptly back to the high value and begins to decay again. They are “warm” because the model's weights are not reset—the solution reached is kept—only the learning rate goes back up.

Why it is used

The abrupt jump to a high rate gives the optimizer the push it needs to escape local minima or plateaus and explore other regions of the loss landscape, while the subsequent decay fine-tunes convergence again. In practice it speeds up training and improves generalization compared with a fixed decay, and it is very common in computer vision and in model fine-tuning.

Its parameters

It is controlled by four values: the maximum and minimum rates it oscillates between; the length of the first cycle; and a multiplication factor that lengthens successive cycles—for example, with a factor of 2 the cycles last 10, 20 and 40 iterations. One elegant application is saving the model at the end of each cycle: since each converges to a different minimum, a set of models (a “snapshot ensemble”) is obtained at the cost of training just one.