Maximum Entropy Method

The maximum entropy principle states that, among all probability distributions consistent with the information we have (the “constraints”), one should choose the one with the greatest entropy. Put another way: the most “spread out” or least committed distribution that still respects the known data. It is a principle of modesty: not assuming more information than one actually has.

Jaynes's idea

It was formulated by the physicist Edwin T. Jaynes in 1957, in a work that built a bridge between statistical mechanics and Shannon's information theory. Jaynes showed that the entropy of thermodynamics and that of information theory obey the same logic of inference.

Why it is the most honest distribution

The intuition is powerful. If you only know some properties of a phenomenon—a mean, some frequencies—there are infinitely many distributions that satisfy them. Choosing any but the maximum-entropy one amounts to slipping in unjustified assumptions, inventing structure the data do not support. The maximum-entropy one is the only one that sticks strictly to what is known and remains maximally agnostic about the rest.

In machine learning

The principle has a very concrete embodiment in AI: the maximum entropy classifier (MaxEnt), which turns out to be equivalent to multinomial logistic regression—softmax regression. It was a very popular tool in natural language processing before the rise of deep learning: it was used to classify text, tag grammatical categories and extract information, prized because, unlike other methods, it does not require assuming that the features are independent.

Where else it appears

Maximum entropy is related to Shannon entropy, to log-linear models and to Markov random fields. And it goes beyond AI: it is a reference method in statistical physics and, curiously, in ecology, where a program called Maxent is a standard for modeling the geographic distribution of species.