Principal Component Analysis (PCA)

Principal component analysis (PCA) is a statistical dimensionality-reduction technique: it transforms a set of possibly correlated variables into a smaller set of uncorrelated variables, called principal components, ordered from most to least according to the variance of the data they explain. It is unsupervised—it uses no labels.

A bit of history

Its roots are old. Karl Pearson posed it in 1901 in geometric terms, as the search for the lines and planes that best fit a cloud of points. Harold Hotelling developed it independently in 1933, and it is to him that the name “principal components” is attributed. Documentation: academic history of principal axes; technical PCA documentation.

How it works

The idea is to find the directions of maximum variance in the data. The first principal component is the direction along which the data vary most; the second is the direction of greatest remaining variance, subject to being perpendicular (orthogonal) to the first; and so on. Mathematically, those components are the eigenvectors of the data's covariance matrix, and the variance each one explains is given by its eigenvalue. The scikit-learn implementation chooses among full SVD, randomized SVD, ARPACK and covariance eigendecomposition according to the data shape and component count; the last option doubles the condition number and is less stable for inputs with a large range of singular values.

Preprocessing matters

Before applying PCA you must center the data, subtracting each variable's mean; without that step, the result is dominated by the position of the set rather than its structure. And because PCA is sensitive to the scale of the variables, when they have very different units or ranges it is advisable to standardize them (divide by their standard deviation). Otherwise a variable measured in thousands would dominate one measured in units purely because of its magnitude, not its importance.

Uses and limit

PCA is used to reduce dimensionality before training a model, to visualize data with many variables in two or three dimensions, to compress and to reduce noise. PCA is a linear transformation. For nonlinear visualization, t-SNE was designed to place high-dimensional data on two- or three-dimensional maps, while UMAP is presented as a manifold-learning dimension-reduction technique. Their objectives differ from preserving linear variance; no general claim about cost or interpretability follows without fixing the data, parameters and task.