Model Evaluation

Model evaluation is the process of measuring how well a trained model generalizes to new, unseen data, and not how well it reproduces the data it was trained on. Its purpose is to estimate future performance in order to judge the model's quality, compare it with others and decide whether it is fit. Evaluating a model on the same data it was trained on is a methodological error: it may look perfect and fail on real data.

Data splitting and cross-validation

That is why data are split into three sets: training, to fit the model; validation, to tune the hyperparameters and monitor generalization; and test, reserved for a final, unbiased estimate. When data are scarce, cross-validation helps: it splits the training set into several parts, trains and validates by rotating which one is left out, and averages the result, giving a more robust estimate.

The metrics

The metrics depend on the task. In classification, accuracy—misleading with imbalanced classes—precision, recall, their harmonic mean (the F1), the confusion matrix and the area under the ROC curve are used. In regression, the mean squared error, the mean absolute error and the coefficient of determination R².

Concepts worth keeping in mind

Evaluating well requires understanding the bias-variance trade-off and distinguishing underfitting (a model that is too simple) from overfitting (a model that memorizes the training set). It also helps to separate the loss function, which is optimized, from the metric, which is reported, and to avoid data leakage: using, when evaluating, information that would not be available at prediction time, which yields deceptively optimistic estimates.