LDA (Latent Dirichlet Allocation)

LDA (Latent Dirichlet Allocation) is a generative probabilistic model for uncovering the topics that run through a collection of documents. David Blei, Andrew Ng, and Michael Jordan presented LDA in a 2003 paper published by the Journal of Machine Learning Research. Its intuition fits in one sentence: every document is a mixture of topics, and every topic is a probability distribution over words. A news article might be 70 percent politics and 30 percent economics, with the economics topic assigning high probability to words such as inflation, jobs, or markets.

One ambiguity is worth clearing up: in statistics and machine learning, LDA also stands for linear discriminant analysis, a classic supervised classifier that separates classes with linear boundaries and has nothing to do with topic modeling.

How it works

LDA is a three-level hierarchical Bayesian model built around an imaginary generative process: to "write" a document, you first draw its mixture of topics and then, word by word, pick a topic from that mixture and a word from that topic, with Dirichlet distributions as priors. The algorithm runs the process in reverse: given the observed words, it infers which topics best explain the corpus. Blei and his co-authors presented approximate inference based on variational methods and an EM algorithm. In 2004, Thomas Griffiths and Mark Steyvers published an MCMC algorithm using Gibbs sampling in PNAS: they analyzed all 28,154 PNAS abstracts published from 1991 to 2001, compared models from 50 to 1,000 topics, and found that, under their corpus and hyperparameter choices, a 300-topic model accounted for the data best. The output is twofold: a list of topics described by their most probable words, and the proportion of each topic in each document. That made LDA ideal for exploring large corpora without reading them, from historical archives to newspaper collections; a Programming Historian lesson teaches how to apply it in the digital humanities with the MALLET toolkit.

Documented limits

The number of topics must be set in advance, and there is no obviously "right" value. The model treats each text as a bag of words, ignoring order and meaning, so it cannot tell that "car" and "automobile" are near-synonyms. Nor are topics always readable: the "Reading Tea Leaves" study (Chang et al., NIPS 2009) showed with human evaluators that models scoring better on statistical likelihood can produce less interpretable topics. And because inference is stochastic, two runs on the same corpus may yield different topics.

LDA versus embeddings and language models

Today's topic modeling leans on vector embeddings, which do capture semantics. One example is BERTopic, presented by Maarten Grootendorst in 2022: it chains transformer embeddings, UMAP reduction, HDBSCAN clustering, and class-based TF-IDF to describe each cluster. Across three datasets and under the author's chosen coherence and diversity metrics, the paper reports coherent topics and competitive results; it also cautions that those metrics are proxies for a subjective evaluation. Large language models are also used to label and summarize the resulting topics, a step the tool itself supports. LDA has not disappeared: it still pays off on very large corpora where computational cost matters, when no GPU is available, or when you want an explicit statistical model with stated assumptions and results reproducible from a fixed seed.