Cross-Validation

Cross-validation is a resampling technique for estimating how well a model will generalize to data it has not seen by systematically reusing the available observations. Instead of judging the model on the examples used to fit it, the procedure takes turns holding out one portion for validation and trains on the rest.

Its immediate advantage is reducing dependence on a single train/test split: every observation can take part in training and validation, though never in both roles within the same fold. That does not make the estimate infallible. Bengio and Grandvalet proved in 2004 that there is no universal unbiased estimator of the variance of k-fold cross-validation; fold errors are correlated because their training sets overlap. Historically, Stone applied a generalized cross-validation criterion in 1974 to choosing and assessing predictions in estimation, regression, and analysis of variance.

K-fold validation and its variants

In k-fold cross-validation, the data is divided into k blocks; the model is fitted on k−1 and evaluated on the remaining block. The process continues until every block has been held out once, after which the k scores are summarized. The number of folds is not a law of nature. In his 1995 experiment, which comprised more than half a million runs of C4.5 and Naive Bayes, Ron Kohavi recommended stratified ten-fold validation for datasets similar to those he studied; the paper's stated scope rules out treating that result as a universal recipe.

In leave-one-out (LOO), k equals the number of observations. It trains on n−1 observations, but requires one fit per observation and does not guarantee a more reliable estimate: Kohavi discusses its high variability. Stratified cross-validation keeps approximately the same class proportions in each fold; in the same study's experiments, it slightly reduced bias and variance relative to unstratified splitting.

What it is for: model and hyperparameter selection

Cross-validation can compare algorithms, choose hyperparameters, and estimate out-of-sample error. Those jobs should not be conflated. If many settings are tested and the best score on those same folds is reported as final performance, the search can overfit the selection criterion. Cawley and Talbot showed that this bias can be comparable to the measured differences between algorithms.

Nested cross-validation separates the jobs: the inner loop chooses hyperparameters and the outer loop evaluates the complete procedure. In simulations with shrunken-centroid classifiers and support vector machines, Varma and Simon found that reusing validation for tuning and assessment produced biased estimates, whereas the nested procedure came close to the error measured on independent test data.

Pitfalls to avoid: data leakage and time series

Data leakage occurs when training uses information that would not be available at prediction time. Scaling, imputing, or selecting features on the full dataset before folds are created lets validation observations influence the model. The official scikit-learn documentation gives the operational rule: fit each transformation only on the training subset; a pipeline repeats that fitting inside every fold.

Time series require a split that represents the intended use and respects relevant dependencies. Careless shuffling can mix future information into the past, but no single scheme is valid for every series. Bergmeir and Benítez compared six procedures and proposed blocked cross-validation as a way to use the available observations without ignoring order and dependence. The right question is not simply “how many folds?”, but “which observations will the model know when it has to predict?”.