Feature engineering: methods and good practice

There is a documented example in which a model reaches 76 % accuracy predicting a completely random target. Nobody faked anything. The target was pure noise, unrelated to the inputs, and the expected result was 50 % — a coin flip.

The mistake that produces that 76 % fits on one line of code, is the most common in applied machine learning, and lives precisely in feature engineering. Understanding how it happens is worth more than memorising the catalogue of techniques, because the catalogue changes and this failure has stood intact for decades.

What feature engineering is

It is the work of deciding what the model is shown and in what form. It has two halves. Selection discards variables that add nothing or get in the way; transformation turns raw data into something the algorithm can use: scaling magnitudes into comparable ranges, turning categories into numbers, extracting from a date the day of the week or the distance to a public holiday.

Selection comes in three families. Filter methods score each variable by its statistical properties without training anything; they are fast and cheap. Wrapper methods — such as recursive elimination — train the model many times over, trying subsets; expensive, and usually better. And embedded methods, like Lasso or decision trees, perform selection inside training itself.

The reason this work matters so much is simple: an algorithm cannot discover what is not represented in its input. If the real pattern depends on the gap between two dates and you hand over only the two dates separately, capacity gets spent reconstructing the subtraction. Handing it over ready-made is often worth more than switching algorithms.

Leakage: the definition and the example

The scikit-learn common pitfalls documentation defines the problem with a precision worth copying verbatim:

"Data leakage occurs when information that would not be available at prediction time is used when building the model. This results in overly optimistic performance estimates, for example from cross-validation, and thus poorer performance when the model is used on actually novel data, for example during production."

The 76 % example sits on that same page. Data are generated with random targets, feature selection is run over the whole set, and only afterwards is it split into train and test. The result, in their words: "Using all the data to perform feature selection results in an accuracy score much higher than chance, even though our targets are completely random."

The explanation is that the selection step "sees" the test data. In picking, out of thousands of noise variables, the twenty-five that correlate best with the target across the whole set, those twenty-five are chosen to work on the very portion reserved for evaluation. The evaluation stops being independent. And if instead of selecting variables you normalise by dividing by the mean, the same trap wears a different face: the mean must be computed from the training data alone, because "if the test subset is included in the average calculation, information from the test subset is influencing the model."

The two forms you do not see coming

Preprocessing leakage yields to discipline. Two other variants are harder, because they live not in the code but in the meaning of the data.

Temporal leakage means using information from the future. It happens whenever you predict something that evolves over time and split the data at random rather than by date: the model trains on Tuesday and is evaluated on Monday. Any validation of a time series split randomly is inflated, and it is an error committed daily in demand forecasting, price forecasting and churn prediction.

Consequence leakage is the most elegant and the most expensive. It means including as a predictor something that is actually an effect of what you are trying to predict. The textbook case: predicting which customers will default using, among the variables, "number of calls from the collections department." It performs spectacularly in validation. In production it is useless, because by the time that variable has a value the default has already happened. The model was not predicting. It was remembering.

The technique that is leakage with a good reputation

There is one case where this article's two halves collide, and it is worth dwelling on because it is used constantly and done wrong almost every time.

When a categorical variable has hundreds of distinct values — postcodes, product references, salesperson IDs — turning it into one column per value is unworkable. A popular alternative is target encoding: replacing each category with the mean of the target within that category. Postcode 28004 becomes "0.37" because that is the historical default rate there.

It is powerful, and it is literally putting the target inside an input variable. If that mean is computed over all the data, every row carries information about its own answer baked in, and the model will look magnificent right up to launch day. Done properly — computing the mean only within each training fold, excluding the row being encoded — it works. Done the obvious way, it is the perfect leak: it does not look like a mistake, it looks like engineering.

The remedy, which is procedural

The same documentation boils the rules down to three:

Split before touching anything. "Always split the data into train and test subsets first, particularly before any preprocessing steps."

Do not let preprocessing look at the test set. "Never include test data when using the fit and fit_transform methods." You fit on train and apply to both, never the other way round.

Chain the steps into a pipeline. "The scikit-learn pipeline is a great way to prevent data leakage as it ensures that the appropriate method is performed on the correct data subset." It is the same principle behind any reliable control: turn the rule into something the tool enforces by construction, rather than something you have to remember.

The capability you take away

This entire family of errors yields to a single question, and it works without any programming:

Would I have had this value, with this content, at the exact instant the prediction has to be made?

If the answer is no — because the field gets filled in later, because it summarises the whole period, because it is produced by a process triggered by the very thing you are trying to anticipate — then it cannot be in there.

And the question reaches far past models. It applies to the backtest of an investment strategy, to the dashboard that "predicts" which salespeople will hit target, to the study that finds a disease risk factor using records completed after the fact. Whenever a retrospective analysis comes out suspiciously well, the first hypothesis should not be that something has been discovered: it should be that somebody looked at the future without noticing.

Where to go next, with no middlemen

The scikit-learn common pitfalls page carries the random-target example with complete code: it runs in a minute, and watching 0.76 turn into 0.50 by reordering two lines is worth any explanation. The preprocessing documentation covers the usual transformations along with their assumptions.

The capability you take from this is asking every variable whether it existed at the moment of prediction — because a model that peeks at the future always gets it right, and always only in the lab.