Intermediate · 04
Is my model any good?
Accuracy can lie. Meet the confusion matrix, precision, recall and F1.
6 min read · Tap underlined words for a definition
Swipe up to begin
A model that predicts "no fraud" for every transaction is 99.9% accurate if only 0.1% of transactions are fraud, and completely useless. To judge classifiers properly, you need better tools than accuracy.
The four boxes
True Positive (TP): said yes, was yes. 🎉
True Negative (TN): said no, was no. 👍
False Positive (FP): said yes, was actually no (false alarm).
False Negative (FN): said no, was actually yes (missed it).
Precision
Precision = TP / (TP + FP). Of everything I flagged, how much was right?
Matters when false alarms are costly, e.g. a spam filter binning your job offer.
Recall
Recall = TP / (TP + FN). Of all the real positives, how many did I catch?
Matters when misses are costly, e.g. screening for cancer.
For an airport weapon scanner, which metric matters most?
F1 score & the trade-off
Raising your threshold usually boosts but lowers , and vice versa.
F1 is the harmonic mean of the two, a single number that's only high when both are.
For regression
MAE: average absolute error, easy to read ("off by £12k on average").
RMSE: like MAE but punishes big misses more.
R²: how much of the variation your model explains (1.0 is perfect).
Your model flagged 100 emails as spam; 90 really were spam. What's the precision?
Before you drift off
- Accuracy misleads on imbalanced data.
- Precision: how many flagged were right. Recall: how many real ones were caught.
- F1 balances both; pick metrics based on the cost of each mistake.
Terms in this lesson
Go deeper, free and open
- ML Crash Course: Classification metrics · Google · CC BY 4.0
- scikit-learn: Model evaluation guide · scikit-learn developers · BSD-3
That's the lesson
Is my model any good? · 6 min
Each lesson you finish lights a star.
Spotted a mistake? Suggest an edit
Previous: Overfitting & the Goldilocks model