Cross-Validation: How to Select a Model Without Training on the Exam

Re-edited on July 30, 2026, this article corrects the image of a deck shuffled at random to produce “true accuracy.” Cross-validation estimates performance under a splitting rule. If that rule does not represent how new cases will arrive—another person, hospital, machine, or date—a flawless average may measure leakage.

It also separates two uses that are often confused. Validation helps choose models, variables, and hyperparameters. A final test set then estimates the chosen procedure. Using the same folds to search and announce performance lets evaluation participate in training even when no gradient touches it.

Which question does a split answer?

The independence unit comes first. With repeated visits per patient, windows from one sensor, or documents by one author, row-wise splitting mixes closely related information. “Does it work on another row?” is not equivalent to “does it work on another person?” The group that will be new must remain entirely on one side.

In k-fold validation, data is divided into k parts; each validates once while the rest train. Metrics are averaged, but results share training data and are not simple independent observations. Kohavi’s classic study compared cross-validation and bootstrap estimation for classifier selection and showed bias-variance tradeoffs among designs.

LOOCV uses one observation in each validation fold. It trains with almost all data but can be computationally expensive and unstable for selection, and it fails when dependent rows are separated. “Small data” does not automatically make it best. The choice of k, repetition, and stratification depends on size, structure, and cost.

Stratification approximately preserves class proportions and can help with rare classes, but it does not repair group or time structure. A split may need both constraints. If a class has too few groups to appear in every fold, report that fact rather than duplicating examples to manufacture stability.

Everything that learns belongs inside the fold

Scaling, imputation, feature selection, deduplication, vocabulary learning, oversampling, and threshold tuning use data. If performed before splitting, validation influences the transformation. Fit the whole pipeline only on the training portion and apply it afterward to validation.

Leakage can be subtler. A label calculated after the event, a variable encoding the hospital, or a near duplicate enables shortcuts. Inspect every field for availability at prediction time and search for relationships among rows. Accuracy cannot distinguish a legitimate pattern from an impossible production clue.

A study of subject cross-validation in activity recognition examined how overlapping windows and subject-mixing splits inflate results. The principle transfers: sessions, devices, households, companies, and authors should be grouped when deployment expects new entities.

The final test set remains untouched during this engineering. Consulting it to decide imputation or model choice makes it nonfinal. An honest practice records decisions and limits looks. When data is insufficient for a separate test, nested validation can be used and the greater uncertainty stated.

Nested validation: select inside, estimate outside

In nested validation, the outer loop reserves one fold for estimation. Inside the remainder, another process chooses variables, hyperparameters, and algorithm. The selected configuration is then trained on all outer training data and evaluated once on the outer fold. Every outer fold repeats selection from scratch.

Varma and Simon showed that selecting through cross-validation and then reporting the same estimate introduces optimistic bias, while nested validation reduced that bias in their experiments. This is not a biomedical ritual; it appears whenever the best of many attempts is selected.

Cawley and Talbot explained that the selection criterion itself can overfit and that its variance matters. Testing more hyperparameters, seeds, variables, and models creates more chances to find favorable noise. The search is part of the evaluated procedure and belongs inside.

The outer result is not the final model ready to serve; it estimates the procedure. After fixing that procedure, fit it on all development data and evaluate on the final test if one exists. Preserve configurations by fold: one candidate winning consistently is informative, while frequent changes reveal instability.

Selection includes more than the algorithm. Choosing columns, missing-value treatment, threshold, calibration, epochs, or stopping rule are learned decisions. Even designing a new feature after inspecting validation errors adapts the procedure. This is not forbidden; it simply belongs inside development and must later face data that did not inspire the idea.

Groups, time, and domain change the rule

Time-series evaluation does not train on the future to predict the past. Expanding or sliding windows train through one date and validate later. Work on cross-validation for time series analyzes when CV procedures apply to autoregressive models. The use hypothesis must match the intended horizon and update schedule.

Spatial and center-based problems reserve a region, hospital, or store. This may produce fewer folds and greater variation, but it answers the difficult question. If the goal is a new organization, random splitting within known organizations estimates something else. Both experiments can be reported under different names.

Temporal validation also reveals degradation. Plot performance by date rather than only the average. A decline may come from the population, policy, sensor, or label. Mixing periods again to raise the score erases the deployment finding.

Repeating a random split reduces dependence on one lucky partition under the same assumed population; it does not simulate domain shift. One hundred repetitions that mix hospitals still do not answer how a new hospital will perform. Choose the repetition count in advance and pair it with a structural test. More computation cannot repair the wrong question.

Label availability also shapes design. When an outcome arrives months later, production validation will be delayed. Establish an early process metric without confusing it with the final outcome and reserve a cohort for updating the estimate. A model may appear stable only because its recent errors are not yet known.

Stock-price prediction illustrates the danger of casual claims: autocorrelation, costs, data availability, multiple tests, and regime shifts can turn a simulation into a mirage. A strategy requires temporal backtesting, transaction costs, benchmarks, and a held-out future period. Cross-validation does not turn historical correlation into profit.

Compare models without turning noise into a winner

The average comes with dispersion, per-fold results, and denominators. A naive interval treating folds as independent may be too narrow. Repeating splits reveals sensitivity but does not create new data. Reports also include cost, latency, calibration, subgroups, and operating complexity.

The bootstrap is another estimation tool. Efron’s work studied prediction-rule error estimation and improvements over the apparent error rate. Choosing bootstrap or CV depends on the estimand and assumptions; neither repairs a wrongly defined unit.

When many algorithms are compared across multiple datasets, between-dataset variation matters. Demšar reviewed statistical tests for comparing classifiers over datasets. The practical lesson is to avoid declaring universal superiority from a small difference on one split; publish tasks, ranks, and practical relevance.

AIC and BIC operate in likelihood frameworks and penalize complexity under assumptions; they do not replace a test of transport to a future population and cannot be compared blindly across every pipeline. Interpretability is not a number added to accuracy either: specify who needs to understand what for which decision.

The transferable skill is designing the split from the deployment question and separating selection, estimation, and final testing. The protocol has five questions: which unit will be new, which steps learn, what is selected inside, what remains outside, and which uncertainty is reported? Answering them matters more than reciting k-fold or LOOCV.