Principal Component Analysis (PCA)

Principal component analysis (PCA) is a statistical dimensionality-reduction technique: it transforms a set of possibly correlated variables into a smaller set of uncorrelated variables, called principal components, ordered from most to least according to the variance of the data they explain. It is unsupervised—it uses no labels—and deterministic: given the same data it always yields the same result.

A bit of history

Its roots are old. Karl Pearson posed it in 1901 in geometric terms, as the search for the lines and planes that best fit a cloud of points. Harold Hotelling developed it independently in 1933, and it is to him that the name “principal components” is attributed.

How it works

The idea is to find the directions of maximum variance in the data. The first principal component is the direction along which the data vary most; the second is the direction of greatest remaining variance, subject to being perpendicular (orthogonal) to the first; and so on. Mathematically, those components are the eigenvectors of the data's covariance matrix, and the variance each one explains is given by its eigenvalue. In practice, today it is usually computed via the singular value decomposition (SVD), which is more numerically stable.

Preprocessing matters

Before applying PCA you must center the data, subtracting each variable's mean; without that step, the result is dominated by the position of the set rather than its structure. And because PCA is sensitive to the scale of the variables, when they have very different units or ranges it is advisable to standardize them (divide by their standard deviation). Otherwise a variable measured in thousands would dominate one measured in units purely because of its magnitude, not its importance.

Uses and limit

PCA is used to reduce dimensionality before training a model, to visualize data with many variables in two or three dimensions, to compress and to reduce noise. Its limit is that it only captures linear relationships: when the structure of the data is curved or tangled, it falls short. For those cases, nonlinear techniques such as t-SNE or UMAP are used, aimed mainly at visualization, which better preserve local neighborhoods but are more costly and less interpretable.