Gaussian Process Classification

Gaussian process classification is a Bayesian method that assigns inputs to classes and produces posterior probabilities under a kernel and an inference approximation. Those probabilities express the model and its assumptions; they do not by themselves guarantee calibrated uncertainty.

What a Gaussian process is

A Gaussian process (GP) is a probability distribution over functions: a collection of random variables such that any finite subset of them follows a joint Gaussian distribution. It is completely specified by a mean function and a covariance function, called the kernel, which encodes the assumptions about the function (its smoothness, its scale). It is a Bayesian non-parametric model: rather than fixing a number of parameters in advance, its capacity grows with the data.

From regression to classification

In regression, the Gaussian process works cleanly because the likelihood is Gaussian and everything stays tractable. In classification, by contrast, the labels are discrete and that Gaussian likelihood is no longer appropriate. The solution is to pass the GP's latent function through a link function—a logistic sigmoid or the probit—which squashes its value to the range [0, 1] and turns it into a well-defined class probability.

Why approximations are needed

That change has a price. When a Gaussian prior is combined with a non-Gaussian likelihood, exact inference is no longer feasible. Rasmussen and Williams's classification chapter develops two Gaussian approximations to the posterior: Laplace and Expectation Propagation. The list should not be extended without anchoring each method to its concrete formulation.

Advantages and limit

The kernel provides flexibility, and probabilities make the model’s uncertainty explicit. In the scikit-learn implementation, GP classification scales cubically with dataset size; its documentation therefore notes that one-versus-one can be cheaper by solving more problems on smaller subsets.