Maximum Entropy

The maximum entropy principle is a rule of inference: among all probability distributions consistent with what we know—say, a known mean—choose the one with the greatest entropy, that is, the most uncertain or least “committed” to what we do not know. The physicist Edwin T. Jaynes formulated it in 1957 to derive statistical mechanics from information theory. The idea is to avoid assumptions the data do not justify.

When the constraints are expected values, the solution to that optimization problem takes an exponential form: fixing only the support yields the uniform distribution; fixing the mean, the exponential; fixing mean and variance, the Gaussian.

Three ideas worth keeping apart

The word “entropy” gathers three distinct notions. The maximum entropy principle is the above: maximizing the entropy of a distribution subject to constraints. Cross-entropy is something else: a loss function measuring the discrepancy between two distributions—the true one and the model's—which, when minimized, pushes the model toward confident, low-entropy predictions; the opposite direction to “maximizing entropy.” And entropy regularization is a term added to the objective to deliberately raise the output's entropy and prevent overconfidence, as in label smoothing or maximum-entropy reinforcement learning.

The maximum entropy classifier

In machine learning, the “maximum entropy model” denotes something very specific: the maximum entropy classifier is exactly multinomial logistic regression (the softmax function). This is not an analogy. Constraints are imposed so that the expected values of certain features under the model match those observed in the data; among all distributions satisfying that, the maximum-entropy one has a log-linear form. Training that model by maximum likelihood—minimizing cross-entropy—yields, at the optimum, that maximum-entropy distribution.

What is not a “maximum entropy model”

It is misleading to call generative adversarial networks (GANs) or Transformers “maximum entropy models.” GANs are trained as an adversarial game that minimizes a divergence between distributions, not under any maximum entropy principle. Transformers are an architecture trained with cross-entropy; using a softmax layer does not turn the whole network into a “maximum entropy model.” The legitimate connection is narrower: the softmax function is the maximum-entropy distribution given a constraint on its values, and there is a setting where the principle appears explicitly—maximum-entropy reinforcement learning (for example, the Soft Actor-Critic algorithm), which adds the policy's entropy to the reward to encourage exploration.