ResNet

ResNet (from Residual Network) is a convolutional neural network architecture that introduced residual connections—also called “skip” connections—the innovation that made it possible to train networks far deeper than had been feasible until then. It was presented by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun, of Microsoft Research, in 2015.

The problem: degradation

Before ResNet there was a puzzling obstacle: as more and more layers were stacked, very deep networks got worse, and not because of overfitting, but because they became hard to optimize. This is the so-called degradation problem: a network that was too deep ended up with more error even on the training data itself. It seemed an insurmountable ceiling.

The idea: learn the residual

The solution was elegant. Instead of asking a block of layers to learn the desired transformation directly, it is asked to learn only the difference with respect to its input—the “residual”—and that input is added back at the end via a skip connection that lets it pass intact. Thus, the block's output is “the input plus whatever the block has learned.” This has two virtues: it helps the gradient flow without vanishing during training, and it makes it trivial for a layer to “stay out of the way” if it adds nothing, simply learning the identity.

A winning origin

ResNet was not a timid experiment: it reached 152 layers deep, eight times more than earlier networks, and won the great ImageNet visual recognition challenge of 2015 by a wide margin. From it came its variants by number of layers, such as ResNet-50, ResNet-101 or ResNet-152.

An idea that endures

The impact of residual connections has been enormous and lasting. They stopped being a peculiarity of one specific architecture and became a standard ingredient of deep learning. In fact, they are present in Transformers: the architectures that underpin today's large language models could not be trained at such depth without them.