Evaluation Metrics

Evaluation metrics are the quantitative measures used to judge how well a machine learning model performs: they turn its hits and misses into comparable numbers. There is no single metric that fits every task. The right choice depends on the type of problem —classification or regression— and, above all, on how much each kind of mistake costs in the real world.

The scikit-learn documentation recommends choosing a scoring function from the ultimate goal and application; Google's Machine Learning Crash Course adds that the task, error costs, class balance, and threshold all matter. A high number under a misaligned metric can describe a model that is useless for the real decision.

Classification: from the confusion matrix to ROC-AUC

In classification, almost everything starts with the confusion matrix, which crosses predictions against reality in four cells: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). The classic metrics follow from it. Accuracy is the share of correct predictions over the total. Precision asks, of everything the model flagged as positive, how much really was positive: TP / (TP + FP). Recall (also called sensitivity) asks, of all the actual positives, how many it caught: TP / (TP + FN). The F1 score is the harmonic mean of precision and recall, a single figure that balances the two. The ROC curve plots the true-positive rate against the false-alarm rate as the decision threshold moves, and its area under the curve (AUC) sums up in one value how well the model separates the classes.

Why accuracy misleads on imbalanced data

Accuracy is intuitive, but it betrays you when the classes are imbalanced. If only 1% of transactions are fraud, a detector that always said «no fraud» would be right 99% of the time… without catching a single fraud. Its accuracy would look brilliant and its usefulness would be zero. That is why precision and recall matter: they actually look at the rare class. And they involve a trade-off: pushing for more recall (missing no fraud) usually lowers precision (more false alarms), and vice versa. Where to sit on that balance is a business decision, not just a statistical one.

Regression: MAE, MSE, RMSE and R²

When the model predicts a continuous number rather than a label, other measures apply. The mean absolute error (MAE) averages the size of the errors regardless of sign. The mean squared error (MSE) squares each error, punishing large ones harshly; its square root, the RMSE, brings the result back to the original units and is easier to read. The coefficient of determination (R²) compares the model's error with always predicting the target mean: 1 corresponds to perfect predictions, while 0 matches that constant predictor when the target is non-constant. R² can also be negative when the model is worse than that reference, so it is not a universal percentage of quality. The underlying lesson is to choose the metric for the problem and the cost of each error. Cross-validation repeats the estimate over different splits to expose variation and reduce dependence on a single partition; the split scheme must still respect groups, time, and any dependency in the data.