# Is my model any good? · AI in Bed

Intermediate level · 6 min · https://www.aiinbed.com/learn/intermediate/evaluating-models

*Accuracy can lie. Meet the confusion matrix, precision, recall and F1.*

A model that predicts "no fraud" for every transaction is 99.9% accurate if only 0.1% of transactions are fraud, and completely useless. To judge classifiers properly, you need better tools than accuracy.

*Figure: The confusion matrix: every prediction lands in one of four boxes.*

**Precision and recall by hand**

```python
# 1 = spam, 0 = not spam
truth     = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
predicted = [1, 1, 1, 0, 1, 0, 0, 0, 0, 0]

tp = sum(t == 1 and p == 1 for t, p in zip(truth, predicted))
fp = sum(t == 0 and p == 1 for t, p in zip(truth, predicted))
fn = sum(t == 1 and p == 0 for t, p in zip(truth, predicted))

precision = tp / (tp + fp)
recall = tp / (tp + fn)
accuracy = sum(t == p for t, p in zip(truth, predicted)) / len(truth)
print(f"accuracy  {accuracy:.0%}")
print(f"precision {precision:.0%}  (of emails flagged, how many were spam)")
print(f"recall    {recall:.0%}  (of real spam, how much we caught)")
```

### The four boxes

**True Positive (TP)**: said yes, was yes. 🎉

**True Negative (TN)**: said no, was no. 👍

**False Positive (FP)**: said yes, was actually no (false alarm).

**False Negative (FN)**: said no, was actually yes (missed it).

### Precision

`Precision = TP / (TP + FP)`. Of everything I flagged, how much was right?

Matters when false alarms are costly, e.g. a spam filter binning your job offer.

### Recall

`Recall = TP / (TP + FN)`. Of all the real positives, how many did I catch?

Matters when misses are costly, e.g. screening for cancer.

**Quick check:** For an airport weapon scanner, which metric matters most?

A. Precision
B. Recall
C. Neither
D. Training speed

Answer: B. Missing a real weapon (false negative) is far worse than an extra bag check (false positive), so you want very high recall.

### F1 score & the trade-off

Raising your threshold usually boosts precision but lowers recall, and vice versa.

**F1** is the harmonic mean of the two, a single number that's only high when both are.

### For regression

**MAE**: average absolute error, easy to read ("off by £12k on average").

**RMSE**: like MAE but punishes big misses more.

**R²**: how much of the variation your model explains (1.0 is perfect).

**Quick check:** Your model flagged 100 emails as spam; 90 really were spam. What's the precision?

A. 10%
B. 90%
C. 100%
D. Can't tell without TN

Answer: B. Precision = TP / (TP + FP) = 90 / 100 = 90%. (We'd need the missed spam count to compute recall.)

**Key takeaways**

- Accuracy misleads on imbalanced data.
- Precision: how many flagged were right. Recall: how many real ones were caught.
- F1 balances both; pick metrics based on the cost of each mistake.

**Go deeper (free sources)**

- [ML Crash Course: Classification metrics](https://developers.google.com/machine-learning/crash-course/classification) by Google
- [scikit-learn: Model evaluation guide](https://scikit-learn.org/stable/modules/model_evaluation.html) by scikit-learn developers
