Gaussian Process Classification

Gaussian process classification is a Bayesian machine learning method for assigning each input to a class, additionally returning a measure of uncertainty about that prediction. It rests on an elegant mathematical object, the Gaussian process, and its canonical reference is the book by Carl Rasmussen and Christopher Williams, Gaussian Processes for Machine Learning (2006).

What a Gaussian process is

A Gaussian process (GP) is a probability distribution over functions: a collection of random variables such that any finite subset of them follows a joint Gaussian distribution. It is completely specified by a mean function and a covariance function, called the kernel, which encodes the assumptions about the function (its smoothness, its scale). It is a Bayesian non-parametric model: rather than fixing a number of parameters in advance, its capacity grows with the data.

From regression to classification

In regression, the Gaussian process works cleanly because the likelihood is Gaussian and everything stays tractable. In classification, by contrast, the labels are discrete and that Gaussian likelihood is no longer appropriate. The solution is to pass the GP's latent function through a link function—a logistic sigmoid or the probit—which squashes its value to the range [0, 1] and turns it into a well-defined class probability.

Why approximations are needed

That change has a price. On combining the Gaussian prior with a non-Gaussian likelihood, the posterior distribution no longer has a closed form and cannot be computed exactly. Hence GP classification resorts to approximate inference methods: the Laplace approximation, Expectation Propagation or variational inference, which replace the intractable posterior with a Gaussian approximation.

Advantages and limit

Its virtues are clear: it delivers probabilistic predictions with calibrated uncertainty—larger where data are scarce—performs well with small samples and is very flexible through the kernel. Its Achilles' heel is the computational cost: it scales as the cube of the number of data points (O(n³)), which makes it hard to use with large datasets. To mitigate this there are sparse GPs, which summarize the data into a reduced number of inducing points.