Choosing the wrong metric is one of the most common and costly ML mistakes: you optimise for a number that doesn't match the real goal. This cheat sheet covers the metrics that matter for regression, classification and ranking, and — crucially — when to reach for each.
Regression
MAE (mean absolute error) is in the units of your target and treats all errors equally — intuitive and robust to outliers. RMSE punishes large errors more heavily, so use it when big misses are especially bad. R² tells you the share of variance explained, useful for comparison but easy to over-read.
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score mae = mean_absolute_error(y_true, y_pred) rmse = mean_squared_error(y_true, y_pred, squared=False) r2 = r2_score(y_true, y_pred)
Classification — beyond accuracy
Accuracy lies on imbalanced data: predict "no fraud" every time and you're 99.9% accurate and useless. Reach for:
- Precision — of what you flagged, how much was right. Optimise when false positives are costly.
- Recall — of what was actually positive, how much you caught. Optimise when false negatives are costly (disease, fraud).
- F1 — the harmonic mean, when you need to balance both.
- ROC-AUC — ranking quality across thresholds; PR-AUC is better on heavy imbalance.
Pick the metric that matches the cost of being wrong — not the one that looks best on a slide.
The confusion matrix is your friend
from sklearn.metrics import classification_report, confusion_matrix print(confusion_matrix(y_true, y_pred)) print(classification_report(y_true, y_pred))
One classification_report gives precision, recall and F1 per class — read it before you trust a single headline number.
Ranking & recommendation
For search and recommenders, position matters: Precision@k, Recall@k and NDCG reward putting the right items near the top, which is what users actually experience.
Wrap up
Start from the decision your model informs and the cost of each type of error, then choose the metric that reflects it. The math is easy; the judgement is the job.