Backpropagation

Backpropagation is the algorithm that efficiently computes the gradient of the loss function with respect to all of a neural network's parameters, propagating the error from the output backward via the calculus chain rule. It is efficient because it reuses intermediate computations instead of repeating them layer by layer.

Computing the gradient is not updating the weights

A common confusion is worth undoing. Backpropagation only computes the gradient; it does not update the weights. What updates the weights is the optimizer, which uses that gradient: gradient descent and its variants, such as SGD, momentum or Adam. Put another way, backpropagation is how the gradient is obtained and the optimizer is what is done with it. Confusing the two is a common mistake.

How it works, step by step

Training alternates two passes. In the forward pass, the example is fed in and propagated through the network to the output, and the loss function measures the error; along the way the intermediate values are stored. In the backward pass, the error is propagated from the output by applying the chain rule layer by layer, obtaining the gradient of each weight. Backpropagation ends there; next, the optimizer adjusts the weights.

History

Its mathematical foundations lie in the reverse-mode differentiation described by Seppo Linnainmaa in 1970, which Paul Werbos connected to neural networks in 1974. But it was the paper by Rumelhart, Hinton and Williams in Nature (1986) that popularized the algorithm and showed that it lets hidden layers learn useful representations. Today it is the basis of deep-learning training, and frameworks such as PyTorch or TensorFlow implement it through automatic differentiation.