Evaluation Metrics

Evaluation metrics are the quantitative measures used to judge how well a machine learning model performs: they turn its hits and misses into comparable numbers. There is no single metric that fits every task. The right choice depends on the type of problem —classification or regression— and, above all, on how much each kind of mistake costs in the real world.

As both the scikit-learn documentation and Google's Machine Learning Crash Course stress, measuring well matters as much as training well: a poorly chosen metric can make a useless model look excellent.

Classification: from the confusion matrix to ROC-AUC

In classification, almost everything starts with the confusion matrix, which crosses predictions against reality in four cells: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). The classic metrics follow from it. Accuracy is the share of correct predictions over the total. Precision asks, of everything the model flagged as positive, how much really was positive: TP / (TP + FP). Recall (also called sensitivity) asks, of all the actual positives, how many it caught: TP / (TP + FN). The F1 score is the harmonic mean of precision and recall, a single figure that balances the two. The ROC curve plots the true-positive rate against the false-alarm rate as the decision threshold moves, and its area under the curve (AUC) sums up in one value how well the model separates the classes.

Why accuracy misleads on imbalanced data

Accuracy is intuitive, but it betrays you when the classes are imbalanced. If only 1% of transactions are fraud, a detector that always said «no fraud» would be right 99% of the time… without catching a single fraud. Its accuracy would look brilliant and its usefulness would be zero. That is why precision and recall matter: they actually look at the rare class. And they involve a trade-off: pushing for more recall (missing no fraud) usually lowers precision (more false alarms), and vice versa. Where to sit on that balance is a business decision, not just a statistical one.

Regression: MAE, MSE, RMSE and R²

When the model predicts a continuous number rather than a label, other measures apply. The mean absolute error (MAE) averages the size of the errors regardless of sign. The mean squared error (MSE) squares each error, punishing large ones harshly; its square root, the RMSE, brings the result back to the original units and is easier to read. The coefficient of determination (R²) tells you what share of the data's variability the model explains: 1 is a perfect fit and 0 is no better than always predicting the mean. The underlying lesson never changes: choosing the metric that fits the problem and the cost of each error matters as much as the model itself. And for those numbers to be trustworthy, estimate them with cross-validation rather than on a single split of the data.