Cross-Entropy

Cross-entropy is a measure from information theory that compares two probability distributions defined over the same events. It quantifies how many extra bits are needed, on average, to encode samples drawn from a true distribution p when using a code optimized for a different distribution q. The closer q is to p, the smaller that excess; when the two coincide, cross-entropy reaches its lowest possible value.

Its definition, in plain text, is H(p, q) = − sum over x of p(x)·log q(x), where the sum runs over every possible event x. The base of the logarithm sets the unit: base 2 gives bits, the natural logarithm gives nats.

What it measures and its link to entropy and Kullback-Leibler divergence

Cross-entropy decomposes exactly: H(p, q) = H(p) + D(p‖q), the Shannon entropy of the true distribution plus the Kullback-Leibler divergence between p and q. The entropy H(p) is the minimum, unavoidable cost of describing p; the divergence D(p‖q), introduced by Solomon Kullback and Richard Leibler, is the penalty for using the wrong code — always non-negative and zero only when q = p. A central consequence follows: if p is fixed, minimizing cross-entropy is exactly the same as minimizing the Kullback-Leibler divergence, because H(p) is a constant that does not depend on q. This reading, set out in detail by Thomas Cover and Joy Thomas in Elements of Information Theory, is why cross-entropy works as a measure of discrepancy between a reality and its approximation.

Cross-entropy as a loss function in classification

In machine learning it is the standard loss function for classification. Here p is the true label, almost always encoded as a one-hot vector (all the probability mass on the correct class), and q is the model's prediction, which yields a distribution through a softmax function in the multiclass case or a sigmoid in the binary one. With a one-hot label, the formula collapses to −log q(correct class): the loss is simply the negative logarithm of the probability the model assigned to the right answer, and it punishes confident mistakes severely. The two-class case is known as log loss; the many-class case as categorical cross-entropy.

Why it is used: maximum likelihood and better gradients

Minimizing cross-entropy over a dataset is identical to maximizing the model's likelihood: the loss coincides with the negative log-likelihood, so training with it is maximum-likelihood estimation in disguise. That statistical footing, developed in texts such as Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville and Pattern Recognition and Machine Learning by Christopher Bishop, is one reason for its dominance. The other is practical: paired with softmax or sigmoid, cross-entropy yields better gradients than mean squared error. With probabilistic outputs, squared error produces gradients that vanish precisely when the model is confidently wrong; cross-entropy instead cancels the troublesome term and leaves a gradient proportional to the error, which speeds up and stabilizes learning. That is why, barring exceptions, modern classification is trained with cross-entropy rather than with Euclidean distances.