Maximum Likelihood Estimation

Maximum Likelihood Estimation (MLE) is the most common way to fix the parameters of a statistical model. Its rule is simple: among all possible parameter values, choose the ones that make the data you actually observed most probable.

To do this we define the likelihood: the probability of the data, but read as a function of the parameters rather than of the data. Maximizing it asks, «which parameter values best explain what I saw?». In his 1922 paper, Ronald Fisher defined likelihood and presented the «Method of Maximum Likelihood» as a way to choose the estimate that maximizes that function.

The idea and the log-likelihood

In practice, almost nobody maximizes the likelihood directly. When observations are independent, the likelihood is a product of many probabilities, and multiplying hundreds of numbers smaller than one yields quantities so tiny that a computer rounds them to zero. The standard fix is to take logarithms: the log-likelihood turns that product into a sum, which is more numerically stable and easier to differentiate. Because the logarithm is increasing, the maximum sits in the same place. That is why the objective actually optimized is usually written as a sum of log-probabilities, or its negative when minimizing is preferred.

Why it is everywhere

Although it was born in classical statistics, training much of modern machine learning is doing maximum likelihood, even when nobody calls it that. Fitting a Gaussian to data by MLE returns exactly the sample mean and variance. Logistic regression estimates its coefficients by maximizing the likelihood of the labels. And minimizing the cross-entropy while training a classification neural network is, term for term, the same as maximizing the log-likelihood: cross-entropy and the negative log-likelihood are the same expression. That is the bridge linking a method from the 1920s to today's gradient descent (see Cross-entropy).

Against the Bayesian view and overfitting

Maximum likelihood looks only at the data. The Bayesian approach adds a prior distribution over the parameters —what was believed before seeing them— and combines it with the likelihood through Bayes' theorem (see Bayes' theorem). Its close relative, maximum a posteriori (MAP) estimation, picks the parameters that maximize that posterior probability instead of the bare likelihood. It helps not to confuse likelihood from posterior probability: the first is the probability of the data given the parameters; the second, of the parameters given the data. The distinction matters because with little data MLE can overfit, taking the noise in the sample too seriously. Some regularizers, such as an L2 penalty associated with a Gaussian prior, form a bridge to MAP; not every penalty corresponds to a prior. MLE does not by itself provide a distribution of uncertainty: it returns one «optimal» parameter value, not a credible range, and its quality depends on the chosen model being reasonable.