Gaussian Processes

A Gaussian process (GP) is a distribution over functions: rather than fixing a parametric form and estimating only its coefficients, it places probability over possible functions. Its defining property is that, for any finite set of inputs, the corresponding function values have a joint multivariate normal distribution. It is specified by a mean function and a covariance function, or kernel, as developed in the treatment by Rasmussen and Williams.

The mean expresses the expected value before and after observing data; the kernel expresses how function values covary across inputs. Instead of returning only a curve, the model yields a predictive distribution conditional on its assumptions. Gaussian Processes for Machine Learning, published in 2006, systematizes regression, classification, kernels, model selection, and approximations for large datasets.

The role of the kernel

The kernel encodes which functions are plausible: it may represent smoothness, periodicity, trends, different scales, or combinations of these properties. The scikit-learn documentation distinguishes stationary kernels, which depend on distance rather than absolute position, from non-stationary kernels. Choosing one is not cosmetic: it declares what it means for two inputs to be similar and how that similarity should carry over to their outputs.

Hyperparameters, such as length scale and amplitude, make those assumptions concrete. They may be fitted through the marginal likelihood, although optimizing it can reach local optima and cannot turn an unsuitable kernel into a suitable one. The documented implementation permits optimizer restarts precisely because the surface may have several optima.

The kernel also determines which extrapolations can be defended. Wilson and Adams built spectral-mixture kernels that discovered patterns and extrapolated them in synthetic examples, atmospheric CO₂ concentrations, and airline passenger data. The result does not say that every GP extrapolates well: it shows that the capability comes from structure the kernel can express.

Two uses: regression and classification

Regression and classification are two common supervised uses. In Gaussian process regression, a Gaussian likelihood and fixed hyperparameters permit exact predictive inference through matrix operations. Williams and Rasmussen studied this prior over functions and the resulting Bayesian predictive analysis. The prediction provides a posterior mean and variance.

That variance is uncertainty within the model: it is conditional on the kernel, likelihood, and hyperparameters. It does not by itself guarantee calibration against the world. The implementation documentation returns a mean and standard deviation, but also exposes choices such as noise level and marginal-likelihood optimization that affect both.

In Gaussian process classification, a GP supplies a prior over a latent function and a link function turns it into a class probability. Because discrete labels require a non-Gaussian likelihood, the posterior is no longer Gaussian and is commonly approximated. GaussianProcessClassifier uses a Laplace approximation; the Rasmussen and Williams book also presents the general classification framework and other approximation methods.

Limits: O(n³) cost and sparse approximations

Exact dense inference must factor or solve systems involving the covariance matrix. For n observations, the usual order of cost is O(n³) in time and O(n²) in memory. The documentation explicitly notes cubic scaling for its classifier. The practical boundary is not universally “a few thousand” observations: it depends on hardware, exploitable structure, and required precision.

Sparse approximations replace the role of all observations in inference with a smaller set of inducing variables or points. Titsias proposed a variational formulation that jointly learns inducing inputs and kernel hyperparameters by maximizing a lower bound on the marginal likelihood and minimizing divergence from the exact posterior. The computational gain comes with a visible condition: this is no longer the dense model's exact posterior, but an approximation whose quality must be checked.