Entropy

Shannon entropy measures the uncertainty or average information content of a random variable: how much information is obtained, on average, on learning its value. For a variable with outcomes of probabilities p₁, …, pₙ, it is computed as the sum, with a negative sign, of each probability times the logarithm of that probability; if the logarithm is base 2, the result is expressed in bits. Claude Shannon formulated it in 1948, in the work that founded information theory.

What it means

Entropy indicates the minimum number of bits needed, on average, to encode the outcomes of a source. It is maximal when all outcomes are equally likely—uncertainty is total—and zero when the outcome is certain: the toss of a fair coin has an entropy of one bit, whereas a coin with two identical faces has entropy zero, because it provides no new information.

Entropy and cross-entropy are not the same

A key distinction for machine learning is in order: entropy is defined over one distribution—its internal uncertainty—while cross-entropy is defined between two distributions, the true one and the one the model predicts. Cross-entropy is the most-used loss function in classification: it measures the discrepancy between the two and, when minimized, pushes the model to assign more probability to the correct class. It is related to the Kullback-Leibler divergence, which quantifies the information lost when approximating one distribution with another.

Other uses in machine learning

Entropy appears in more places. Information gain—the reduction in entropy when splitting a set by an attribute—is the criterion that decision trees such as ID3 or C4.5 use to choose their splits. And in reinforcement learning, adding an entropy term to the objective encourages exploration and avoids converging too soon to a suboptimal policy.