Feature Engineering

That bridge between raw data and a useful variable covers cleaning, transforming, combining and selecting. As IBM summarizes it, the process requires data analysis and domain knowledge to turn raw information into useful variables for the model.

Feature engineering is the process of turning raw data into the variables —the features— that a machine learning model can actually use. A sales record, a pixel, or a string of text rarely helps a model as-is: the job is to distill from them numerical signals that capture what matters for the problem and leave out the noise.

Transform, encode, create, select

The toolkit for transforming, encoding, creating and selecting features is broad. For scale-sensitive models, transformations adjust numerical variables through normalization, standardization or logarithms so that magnitude does not distort learning.

Encoding turns categories into numbers the model can understand: one-hot encoding for a small number of classes, embeddings when there are many. Creating new variables, guided by domain knowledge, derives signals that were not explicit: a date becomes a day of the week or a holiday flag, and two columns combine into a ratio.

This also includes handling missing values through imputation and, at the other end, feature selection and extraction, which reduce the set to the most informative variables—topics covered elsewhere in this glossary under feature selection and dimensionality reduction.

Common techniques include normalisation, which moves numerical values onto a suitable scale; binning, which groups them into ranges; and numerical representations of non-numerical data.

Why it rules classical machine learning

In classical models —decision trees, support vector machines, regression— feature engineering is usually the single factor that most determines performance. Pedro Domingos put it plainly in «A Few Useful Things to Know about Machine Learning»: of all the ingredients in a project, the most important is the features used. The working intuition is that data and features set a model's ceiling; the algorithm only decides how close it gets. It is also, for that reason, the most laborious and time-consuming stage of the work. Documentation: Pedro Domingos's paper.

The deep learning shift: automatic representations

Deep learning rewrote the script. Rather than hand-designing variables, neural networks learn their own representations straight from raw data such as pixels or text —what Bengio, Courville, and Vincent termed representation learning. This has cut manual engineering in vision and language, where a network's layers discover hierarchies of features on their own. The nuance matters: it has not vanished. On tabular data, well-fed tree-based models remain hard to beat, and in production features are versioned and shared through feature stores. What is not settled is how far automation will reach —where human judgment still pays off, and where the machine takes over, remains an open question. Documentation: representation-learning paper.