Cross-Entropy

Cross-entropy is a measure from information theory that compares two probability distributions defined over the same events. It quantifies how many extra bits are needed, on average, to encode samples drawn from a true distribution p when using a code optimized for a different distribution q. The closer q is to p, the smaller that excess; when the two coincide, cross-entropy reaches its lowest possible value.

Its definition, in plain text, is H(p, q) = − sum over x of p(x)·log q(x), where the sum runs over every possible event x. The base of the logarithm sets the unit: base 2 gives bits, the natural logarithm gives nats.

What it measures and its link to entropy and Kullback-Leibler divergence

Cross-entropy decomposes exactly: H(p, q) = H(p) + D(p‖q), the Shannon entropy of the true distribution plus the Kullback-Leibler divergence between p and q. The entropy H(p) is the minimum, unavoidable cost of describing p; the divergence D(p‖q), introduced by Solomon Kullback and Richard Leibler, is the penalty for using the wrong code — always non-negative and zero only when q = p. A central consequence follows: if p is fixed, minimizing cross-entropy is exactly the same as minimizing the Kullback-Leibler divergence, because H(p) is a constant that does not depend on q. This identity explains why it measures discrepancy between a distribution and its approximation.

Cross-entropy as a loss function in classification

For classification models that output a categorical or Bernoulli distribution, cross-entropy is a common loss. Here p is the true label, often encoded as a one-hot vector—all probability mass on the correct class—and q is the distribution predicted through softmax or sigmoid. With a one-hot label, the formula reduces to −log q(correct class): the negative logarithm of the probability assigned to the right answer, so a highly confident mistake produces a large loss. The binary case is known as log loss; the multiclass case as categorical cross-entropy. The choice depends on the probabilistic model: this is not a universal loss for every classification task.

The comparison with mean squared error (MSE) needs the model beside it. In a classifier with a sigmoid output, the MSE gradient includes the sigmoid’s slope; when the output saturates near 0 or 1, that slope can become tiny and slow correction even for a confident error. Negative log-likelihood —cross-entropy in this case— avoids that saturation factor for many output models and can provide a more useful gradient. This does not mean it converges better than MSE on every task: the advantage depends on matching the loss to an appropriate probabilistic output.

Why it is used: maximum likelihood and better gradients

In probabilistic classification with a categorical likelihood, minimizing cross-entropy is equivalent to minimizing negative log-likelihood. That equivalence depends on the chosen probabilistic model; it does not turn every loss or task into maximum-likelihood estimation.

In a neural network, that loss is normally optimized by gradient descent or stochastic variants: the system computes how the loss changes with each parameter and updates the weights in the direction that reduces it. The Deep Learning optimization chapter distinguishes batch, stochastic and minibatch descent, as well as extensions such as momentum. The loss function defines the gradient; the optimizer determines how to turn it into a sequence of updates.