Gaussian Process Regression

Gaussian process regression is a non-parametric Bayesian method that, instead of fitting the parameters of a fixed formula, defines a probability distribution over functions and updates it with the observed data. A Gaussian process is, precisely, a distribution over functions: a collection of random variables such that any finite subset of them follows a multivariate normal distribution.

The role of the kernel

A Gaussian process is defined by two elements: a mean function, usually fixed at zero, and a covariance function or kernel, which is the heart of the method. The kernel encodes the assumptions about the functions—how smooth they are and at what scale they vary—the most common being the squared exponential (RBF). Its hyperparameters are fit to the data, usually by maximizing the marginal likelihood.

Its advantage: uncertainty

The great asset of this method is that it returns not just a point prediction, but a prediction with its uncertainty: a mean and a deviation that allow confidence intervals to be built. Moreover, that uncertainty grows as we move away from the observed data, naturally reflecting that the model knows less where it has seen no examples. That is why Gaussian processes are the reference model in Bayesian optimization.

Its limit: cost

Its main drawback is computational: fitting scales as the cube of the number of data points (due to inverting the covariance matrix), which makes it impractical for large datasets. To get around this, “sparse” methods approximate the model with a reduced number of representative points.