Entropy: measuring uncertainty without calling it disorder

In July and October 1948, Claude Shannon published the two parts of A Mathematical Theory of Communication. His problem was not deciding what a message meant, but quantifying the choice among possible messages and the resources needed to transmit them. In that framework, entropy measures the average uncertainty of a probability distribution. It does not measure “disorder” in the colloquial sense, nor quality, intelligence or truth.

The distribution comes before the number

Calculating entropy begins by defining a random variable and its possible outcomes. It might be the symbol emitted by a source, the class of an example or the result of a coin toss. Every outcome receives a probability. Without that list and those probabilities, “the system’s entropy” is incomplete: we do not know what is observed, which alternatives count or how often they are expected.

The same physical situation permits different variables. For a die, one may record the face value, whether the result is odd or even, or whether it exceeds four. Each choice groups events differently and creates a different distribution. Entropy belongs to that declared representation, not to the isolated object. Changing the categories changes the question even when the original observations stay the same.

This is the first defense against misleading use: require the complete support. If an incident analysis retains only “severe” and “minor” but drops “unclassified,” the distribution has been narrowed before calculation. The formula can run without an error and return many decimal places, but it describes the retained categories, not all incidents.

From the surprise of one outcome to average uncertainty

A highly probable result carries little surprise; a rare one carries much more. The information associated with an outcome of probability p is expressed as minus the logarithm of p. Entropy takes the probability-weighted average of those surprises. This is why counting the number of options is not enough: the way probability is distributed among them also matters.

A fair coin has two outcomes with probability 0.5. With a base-two logarithm, its entropy is one bit. If the probabilities are 0.9 and 0.1, it falls to about 0.469 bits. The scipy.stats.entropy documentation reproduces both calculations. This does not mean that the second coin produces “less data” on every toss; it means that its result is more predictable on average under that distribution.

The logarithm base fixes the unit. Base two produces bits; base e produces nats. Two reports can start with the same distribution and publish different numbers because they use different bases. Neither is necessarily wrong, but the values cannot be compared without converting units. An entropy figure without a declared base is as incomplete as a length without meters or inches.

When entropy is largest or smallest

If one outcome has probability one and all others have probability zero, there is no uncertainty about what will happen and entropy is zero. For a fixed number of outcomes, entropy is largest when all are equally probable. With n uniform options, that maximum is the logarithm of n in the selected base. A fair coin therefore has more entropy than a strongly biased one.

The maximum always depends on the number of categories. Two bits is the maximum for four equiprobable outcomes, not for an alphabet of one thousand symbols. Splitting one category into subcategories may raise entropy even though the underlying phenomenon did not change. When comparing models, periods or populations, the outcome vocabulary must remain fixed or the difference will mix uncertainty with a redefinition of the space.

High entropy does not mean error, and low entropy does not mean correctness. A classifier assigning 0.99 to one class has a low-entropy distribution; if the correct class was another, it is confident and wrong. Predictive entropy describes concentration. Accuracy requires observing actual outcomes, while calibration requires comparing many announced probabilities with later frequencies.

The estimator inherits the sample’s defects

True probabilities are rarely known in practice. They are estimated from sample frequencies or a model’s outputs. In a small sample, a rare category may never appear; a collection process may exclude some cases; an uncalibrated model may produce probabilities that do not match real frequencies. Calculated entropy inherits those problems because it operates on the probabilities it is given.

Three objects should remain separate. Empirical entropy summarizes the distribution observed in a sample. Model entropy summarizes probabilities assigned by the model. Population entropy, usually unknown, would belong to the process generating the cases. Agreement between the first two does not by itself show that either represents the third: both can share the same selection bias.

A useful check recalculates the value after grouping rare categories, applying justified smoothing or adding defensible omitted cases. The goal is not to choose whichever version favors a conclusion, but to learn how much the figure depends on reasonable choices about support and estimation. If a minor categorization choice reverses the ordering of two systems, the difference is not robust.

How a decision tree uses entropy

In classification, a tree node contains examples from several classes. Their frequencies form a distribution, and entropy measures how mixed the classes are. A pure node, where all examples belong to one class, has zero entropy. A candidate split produces child nodes; the tree compares the parent’s entropy with the weighted average entropy of the children.

The scikit-learn decision-tree formulation defines Shannon entropy from the class proportions reaching a node and chooses splits that reduce weighted impurity. The reduction is known as information gain. It is not a universal property of a feature: it depends on the training set, classes, sample weights and candidate thresholds available at that node.

A split can achieve a large gain on a handful of examples and fail to generalize. The tree therefore needs constraints such as maximum depth or minimum samples per leaf and must be evaluated on data that was not used to choose its splits. Entropy helps build the tree; it does not certify that the learned structure represents a stable pattern beyond the sample.

Entropy, cross-entropy and KL divergence

The three quantities share logarithms but answer different questions. Entropy H(p) describes average uncertainty when outcomes follow distribution p. Kullback-Leibler divergence D(p‖q) measures the average extra cost of using distribution q when p is the true one. It is not a symmetric distance: exchanging p and q changes both the question and, generally, the value.

Cross-entropy between p and q is the entropy of p plus D(p‖q). The same SciPy reference gives the formulas and also notes that its function normalizes inputs when they do not sum to one. That software convenience does not settle the conceptual question: normalizing counts may be correct, but normalizing negative scores or incompatible categories does not turn them into comparable probabilities.

In machine learning, minimizing cross-entropy encourages a model to assign high probability to observed classes. The value combines uncertainty inherent in the data with the model’s disagreement. Loss values from datasets with different class mixtures therefore cannot be compared directly as if every difference belonged to the model.

Conditioning narrows the question, not necessarily the decision

Joint entropy measures uncertainty about two variables considered together. Conditional entropy asks how much uncertainty about one remains after the other is known. Shannon showed that joint uncertainty can be decomposed into the uncertainty of the first variable plus what remains of the second after the first is known, and that information does not increase conditional entropy under the true distribution.

This relation lets mutual information be read as a reduction in uncertainty: how much, on average, observing one variable reduces uncertainty about another. But statistical reduction does not establish causality. One variable can inform about another because both share a cause or because sampling introduced an association. Entropy quantifies probabilistic dependence; it does not identify the mechanism producing it.

The trap in carrying discrete intuition into continuous variables

Continuous variables use differential entropy, defined over a density. Its formula resembles the discrete one, but its value changes with coordinates or measurement units and may be negative. The original paper states this explicitly. Differential entropy should not be read as an absolute amount of “disorder” or compared between a variable measured in meters and the same variable expressed in millimeters without accounting for the transformation.

This distinction also prevents silently mixing a categorical distribution with a continuous density. In the discrete case, refining categories changes the alphabet; in the continuous case, changing scale changes the density. A comparison must hold the variable type, representation and reference fixed. The formula does not remove those decisions; it incorporates them.

A test that fits on one sheet

To audit an entropy figure, write five lines: observed variable; complete list of outcomes; probabilities and how they were estimated; logarithm base; formula applied. Then add a sensitivity check: what happens if a missing category appears or probabilities move within their plausible range? That sheet makes the number reproducible and reveals whether two reports measured different objects.

The transferable skill is specific: whenever “entropy fell,” ask of which distribution, over what support, in which unit and estimated how. Only then interpret the direction of change. Entropy turns declared probabilities into average uncertainty; it does not replace the work of declaring those probabilities well.