Bayes' Theorem

Bayes' theorem is a rule for updating what we believe as new data arrive. Under the Bayesian interpretation, probability expresses a revisable belief: we start with an initial estimate about a hypothesis and adjust it when we observe evidence that supports or weakens it. That idea underpins Bayesian inference: combining a prior probability with the likelihood of the evidence to obtain a posterior probability.

Bayes' theorem relates conditional probabilities and derives the probability of a hypothesis given evidence from the reverse conditional, the prior probability and the evidence's total probability. Under a Bayesian interpretation, the rule updates beliefs as data arrive.

Its form is P(A|B) = P(B|A) · P(A) / P(B): the probability of hypothesis A given that we have observed evidence B equals the probability of observing B if A were true, multiplied by the initial probability of A and divided by the overall probability of B. The result is not an absolute truth but a belief better calibrated than the one we began with.

The terms: prior, likelihood and posterior

Each part of the formula has a name and a job. P(A) is the prior probability: what we believed before seeing the evidence. P(B|A) is the likelihood: how compatible the evidence is with the hypothesis. P(A|B) is the posterior probability: the revised belief once the new datum is folded in. And P(B), the evidence, acts as a normaliser so that the result is a valid probability. The logic chains naturally: today's posterior is tomorrow's prior, so every fresh observation sharpens the estimate a little more.

An example that illuminates: the medical test and the base rate

Picture a disease that affects 1 % of the population and a test with 99 % sensitivity —it catches nearly every sick person— but a 5 % false-positive rate. If the test comes back positive, intuition says illness is all but certain. Bayes says otherwise. Out of every 10,000 people, 100 are ill and the test flags 99 of them; but of the 9,900 healthy ones, 5 % —495 people— test positive by mistake. Among the 594 positives, only 99 are truly sick: the real probability of disease is barely 17 %.

The lesson is not that the test is bad, but that the base rate—how rare the disease is—matters as much as the result itself. Kahneman and Tversky studied this neglect experimentally; Maya Bar-Hillel defined it in 1980 as the ‘base-rate fallacy’.

Its role in AI: Bayesian inference and Naive Bayes

In artificial intelligence, the theorem is the engine of Bayesian inference: the craft of reasoning under uncertainty by updating probabilities as a system observes the world. From it comes the Naive Bayes classifier (called «naive» because it assumes the features are independent of one another), remarkably effective at tasks such as spam filtering and text classification. The same idea underpins Bayesian networks, which model dependencies among variables. Set out by Thomas Bayes and published posthumously in 1763 by Richard Price in the Royal Society's Philosophical Transactions, and developed independently by Pierre-Simon Laplace, the theorem remains, two and a half centuries on, one of the fundamental grammars of machine learning. Documentation: scholarly Bayes theorem overview; Naive Bayes technical documentation.