Dimensionality Reduction

Dimensionality reduction covers the techniques that turn data with many variables into representations with far fewer while keeping what matters for the task. It answers the curse of dimensionality, a term Richard Bellman coined in “Dynamic Programming” (1957): as dimensions grow, space expands so fast that data becomes sparse and models need far more examples to generalize.

Three families with different goals live under that label, and telling them apart avoids common mistakes.

Three families that should not be confused

Feature selection keeps a subset of the original columns and discards the rest. Guyon and Elisseeff (JMLR, 2003) split it into filters, which score variables independently of the model; wrappers, which use the model itself as a black box to evaluate subsets; and embedded methods, which select during training. The surviving variables remain the original, readable ones. Feature extraction creates new dimensions by combining existing ones — the home turf of PCA and autoencoders — trading interpretability for compression. Reduction for visualization projects data down to two or three dimensions so humans can explore structure by eye; t-SNE and UMAP belong here.

PCA, t-SNE, and UMAP

PCA is over a century old: Jolliffe and Cadima’s review in Philosophical Transactions of the Royal Society A (2016) traces its earliest literature to Pearson (1901) and Hotelling (1933). Per the scikit-learn documentation, it decomposes a dataset into orthogonal components that explain the maximum variance: linear, fast, and inspectable. Autoencoders (Hinton and Salakhutdinov, Science, 2006) learn nonlinear codes that outperform PCA on complex data. t-SNE (van der Maaten and Hinton, JMLR, 2008) was born for visualization: it gives each datapoint a location in a two- or three-dimensional map, preserving local neighborhoods rather than distances. UMAP (McInnes, Healy, and Melville, 2018) matches its visual quality while keeping more global structure, running faster, and putting no limit on the output dimension.

The traps: data leakage and misread maps

First trap, data leakage: fitting the selection or the reduction on the full dataset before splitting train and test contaminates the evaluation. scikit-learn’s common-pitfalls guide shows it: selecting features on complete random data inflates accuracy to 0.76 when the honest figure would sit near 0.5. Transformations must be learned from the training set only — ideally inside a Pipeline — and then applied to the test set; this holds for selection, for PCA, and for nearly all preprocessing.

Second trap, over-reading the maps. Wattenberg, Viégas, and Johnson showed in Distill (2016) that cluster sizes in a t-SNE plot are not comparable and that distances between well-separated clusters may mean nothing. UMAP’s own documentation accepts it as preprocessing but warns that it does not preserve density well, can create false tears in clusters, and that using it before clustering is somewhat controversial and needs different parameters than visualization. These maps are for exploring; as general-purpose features, only with caution and validation.

Where it stands in 2026

Explicit reduction is alive in tabular data, genomics, and exploratory analysis, but today’s most ubiquitous variant is implicit: embeddings, which Google’s machine learning guide defines as lower-dimensional representations of sparse data that capture semantic relationships. Language models and recommender systems compress words, images, or users into learned dense vectors — dimensionality reduction built in. Two questions still lack a universal recipe: how many dimensions to keep, which depends on the task, and how to interpret learned representations, a front that remains open.