Feature Selection

Feature selection is the task of choosing a subset of a dataset's original variables and discarding the rest. Unlike other techniques, the surviving variables are not transformed: they remain the same measurable quantities and stay interpretable. As Guyon and Elisseeff laid out in their standard introduction (JMLR, 2003), its aim is threefold: to improve a predictor's performance, to make the model faster and cheaper, and to better understand the process that generates the data.

Selection versus extraction and dimensionality reduction

Three terms are easily confused. Dimensionality reduction is the general goal: working with fewer variables. It splits into two opposite routes. Selection keeps a subset of the original variables untouched: «worst radius» is still worst radius. Feature extraction, by contrast, builds new variables by combining the originals; principal component analysis (PCA) or an autoencoder yield axes such as «PC1», which are weighted blends of every input variable. The practical difference is interpretability: selection preserves it, extraction trades it for compression. Both cut dimensions, but only selection leaves a model whose inputs still carry a direct physical meaning.

Types: filters, wrappers and embedded methods

Guyon and Elisseeff distinguish three families. Filters score each variable with a statistical criterion —Pearson correlation, chi-squared, mutual information— independently of the model; they are fast but can miss nonlinear interactions. Wrappers use the model itself as a black box to score subsets by their predictive power; they fit the final model better but are costly and prone to overfitting. Embedded methods select during training: Lasso, which zeroes out coefficients through L1 penalization, or the importances that emerge from decision trees. Be wary of «importance» labels: the same authors warn that the most relevant variable is not always the most useful, and that among correlated variables importance is split in unstable ways.

The data-leakage trap and how to validate

The costliest mistake in feature selection is data leakage. If variables are scored and chosen using the whole dataset —test data included— before separating training from evaluation, the model has already «seen» the answer and performance is spuriously inflated. Ambroise and McLachlan showed this on microarray data (PNAS, 2002): cross-validation must be external to the selection, not internal. The rule is to select using the training set only, and to redo the selection inside every fold of a nested cross-validation, so that no decision ever touches the evaluation data. The scikit-learn documentation illustrates the bias with an extreme case: on purely random data, selecting before splitting yields an apparent 76% accuracy, while doing it correctly returns the true 50%. Wrapping selection in a pipeline that is fit on the training data alone is the standard safeguard. Which method performs best in a given case remains empirical: there is no universal winner and no single correct way to measure importance.