Dirichlet Processes

A Dirichlet process (DP) is one of the central objects of Bayesian nonparametric statistics. Instead of placing probability on numbers or vectors, a Dirichlet process places probability on whole distributions: it is a distribution over distributions. Every draw from it is not a data point but an entire probability distribution and, almost surely, a discrete one. Thomas Ferguson introduced it in 1973 precisely to handle problems where we do not want to commit in advance to a fixed parametric form.

A Dirichlet process is specified by two ingredients: a base measure G0, which acts as the expected value or «sketch» of the distributions it will generate, and a concentration parameter alpha, which controls how closely the draws resemble G0. When alpha is large, the sampled distributions stay close to G0; when it is small, they pile onto a few atoms.

What it is for: clustering without fixing the number of groups

Its flagship use in machine learning is the Dirichlet process mixture model. A classical mixture model, such as k-means or a mixture of Gaussians, forces you to decide how many groups exist beforehand. We rarely know. The Dirichlet process removes that constraint: the number of clusters is not fixed but inferred from the data, and it can grow as new observations arrive. This is why it is called an «infinite mixture model». The concentration parameter alpha governs that tendency: the larger it is, the more clusters tend to emerge.

The constructions: the Chinese restaurant and stick-breaking

Two equivalent constructions make the process tangible. In the Chinese restaurant process, customers arrive one at a time; each sits at an already-occupied table with probability proportional to how many people are seated there, or starts a brand-new table with probability proportional to alpha. Tables are the clusters, and their count grows only when the data demand it. The second construction, stick-breaking, was formalized by Sethuraman in 1994: you take a stick of length one and break it again and again, assigning each piece the weight of a cluster; the infinitely many fragments sum to one and define the mixture proportions.

Versus the Dirichlet distribution, and inference

It should not be confused with the Dirichlet distribution, which is a finite case: it splits mass among a fixed number of categories and appears, for instance, in LDA (Latent Dirichlet Allocation). The Dirichlet process is its infinite-dimensional counterpart, meant for when the number of components is unknown. The hardest part remains open: inference. Fitting these models has no closed-form solution and requires approximate methods, such as Markov chain Monte Carlo (MCMC) or variational inference, whose computational cost is still their main practical limitation.