Activation Function

The activation function is the function applied to a neuron's output—to the weighted sum of its inputs—to produce its response, and its essential role is to introduce nonlinearity into the network. It is a small piece with enormous importance.

Why it is essential

The reason is algebraic. Composing two affine transformations still yields an affine transformation: without a nonlinear activation between layers, matrix products and bias sums can be rewritten as one layer. The Deep Learning chapter on feedforward networks shows why the hidden layer needs that nonlinearity. The theoretical result requires care: Cybenko's approximation theorem establishes, under specific conditions, that a network with enough sigmoidal units can approximate continuous functions on a compact domain. It does not say that every architecture, depth, or training run will automatically find the desired function.

The classic functions

There are several, each with its own character. The sigmoid squashes any value into the range (0, 1), which makes it useful for expressing probabilities, but it saturates and suffers from the vanishing gradient. The tanh squashes to (−1, 1) and has the advantage of being zero-centered. The ReLU passes positive values and sets negative ones to zero. Nair and Hinton showed in 2010 that rectified units could train deep models effectively without the positive saturation of sigmoid or tanh. Simplicity does not guarantee a useful neuron: if inputs remain in the negative region, the local gradient is zero and the unit may stop updating. Leaky ReLU keeps a small slope in that region; GELU smooths the gate through a different formulation. These are alternatives with different assumptions, not universal repairs. And the softmax is placed in the output layer to turn a vector into a probability distribution over the classes.

ReLU, a turning point

It deserves a place apart. AlexNet applied ReLU to every convolutional and fully connected layer; in an auxiliary CIFAR-10 comparison, a four-layer network with ReLUs reached 25% training error six times sooner than its tanh counterpart. The authors warned that the size of the effect varied with the architecture, although they consistently observed learning several times faster than with saturating units. It was a small change in appearance, but with enormous consequences for the rise of deep learning.