LDA (Latent Dirichlet Allocation)

LDA (Latent Dirichlet Allocation) is a generative probabilistic model for uncovering the topics that run through a collection of documents. Proposed by David Blei, Andrew Ng, and Michael Jordan in 2003 in the Journal of Machine Learning Research, it was the reference technique for topic modeling for over a decade. Its intuition fits in one sentence: every document is a mixture of topics, and every topic is a probability distribution over words. A news article might be 70 percent politics and 30 percent economics, with the economics topic assigning high probability to words such as inflation, jobs, or markets.

One ambiguity is worth clearing up: in statistics and machine learning, LDA also stands for linear discriminant analysis, a classic supervised classifier that separates classes with linear boundaries and has nothing to do with topic modeling.

How it works

LDA is a three-level hierarchical Bayesian model built around an imaginary generative process: to "write" a document, you first draw its mixture of topics and then, word by word, pick a topic from that mixture and a word from that topic, with Dirichlet distributions as priors. The algorithm runs the process in reverse: given the observed words, it infers which topics best explain the corpus. Blei and his co-authors proposed variational inference with an EM algorithm; in 2004, Thomas Griffiths and Mark Steyvers popularized Gibbs sampling in PNAS, using it to extract 300 topics from 28,154 scientific abstracts. The output is twofold: a list of topics described by their most probable words, and the proportion of each topic in each document. That made LDA ideal for exploring large corpora without reading them, from historical archives to newspaper collections; a Programming Historian lesson teaches how to apply it in the digital humanities with the MALLET toolkit.

Documented limits

The number of topics must be set in advance, and there is no obviously "right" value. The model treats each text as a bag of words, ignoring order and meaning, so it cannot tell that "car" and "automobile" are near-synonyms. Nor are topics always readable: the "Reading Tea Leaves" study (Chang et al., NIPS 2009) showed with human evaluators that models scoring better on statistical likelihood can produce less interpretable topics. And because inference is stochastic, two runs on the same corpus may yield different topics.

LDA versus embeddings and language models

Today's topic modeling leans on vector embeddings, which do capture semantics. The most cited example is BERTopic (Maarten Grootendorst, 2022), which chains transformer embeddings, UMAP dimensionality reduction, HDBSCAN clustering, and a class-based variant of TF-IDF to describe each cluster: according to its paper, it generates coherent topics and remains competitive with classical models across a variety of benchmarks. Large language models are also used to label and summarize the resulting topics, a step the tool itself supports. LDA has not disappeared: it still pays off on very large corpora where computational cost matters, when no GPU is available, or when you want an explicit statistical model with stated assumptions and results reproducible from a fixed seed.