# Decision trees and random forests · AI in Bed

Intermediate level · 7 min · https://www.aiinbed.com/learn/intermediate/decision-trees

*A game of twenty questions that learns from data, and why a whole forest of them still beats neural networks on spreadsheets.*

Not all machine learning is neural networks. A **decision tree** makes predictions by asking a chain of yes or no questions about the data: "Is the customer under 30?" Then "Have they bought before?" Each answer sends you down a branch, until you reach a **leaf** that gives the prediction.

### How a tree picks its questions

Training a tree means choosing good questions. At each step, the algorithm tries many possible splits and keeps the one that best separates the answers.

A good split leaves each side as "pure" as possible: mostly yeses on one side, mostly noes on the other. Then it repeats the process on each side.

### Gini impurity, gently

**Gini impurity** measures how mixed a group is. Pick two items at random from the group: Gini is the chance they have different labels.

A group of all yeses has a Gini of 0, perfectly pure. A 50/50 mix of two labels has a Gini of 0.5, as messy as it gets. The tree picks the split that lowers impurity the most.

**Finding the best first question**

```python
# Should we play tennis? (outlook, windy) -> yes/no
data = [("sunny", False, "yes"), ("sunny", True, "no"), ("cloudy", False, "yes"),
        ("cloudy", True, "yes"), ("rainy", False, "yes"), ("rainy", True, "no"),
        ("sunny", False, "yes"), ("rainy", True, "no")]

def gini(rows):
    if not rows:
        return 0.0
    p = sum(r[2] == "yes" for r in rows) / len(rows)
    return 1 - p**2 - (1 - p)**2

print(f"Gini before any split: {gini(data):.3f}")
questions = {"outlook is sunny?": lambda r: r[0] == "sunny",
             "outlook is cloudy?": lambda r: r[0] == "cloudy",
             "is it windy?": lambda r: r[1]}
for name, ask in questions.items():
    left = [r for r in data if ask(r)]
    right = [r for r in data if not ask(r)]
    score = (len(left) * gini(left) + len(right) * gini(right)) / len(data)
    print(f"{name:20} -> weighted Gini {score:.3f}")
```

The windy question splits the days most cleanly, so a tree would ask it first. Try changing a label and see which question wins.

**Quick check:** A group at a leaf contains 10 "yes" examples and 0 "no" examples. What is its Gini impurity?

A. 0
B. 0.5
C. 1
D. 10

Answer: A. Every item has the same label, so two random picks can never disagree. The group is perfectly pure: Gini is 0.

### Deep trees overfit

Let a tree keep asking questions and it can carve out a leaf for every single training example. It scores perfectly on data it has seen, then stumbles on new data.

The usual fixes are limiting the **depth** (how many questions in a row) or requiring a minimum number of examples per leaf.

### Many trees beat one

A **random forest** trains hundreds of trees, each on a random resample of the data and each looking at a random subset of features at every split. Training on resamples like this is called **bagging**. To predict, the trees vote.

Each tree is a bit wrong in its own way. Averaged together, the mistakes cancel out. Combining models like this is called an **ensemble**.

**Gradient-boosted trees** take another route: they add small trees one after another, each trained to fix the errors of the trees so far. Libraries like XGBoost and LightGBM do this.

> A random forest is like asking a crowd to guess the number of sweets in a jar. Individuals are off, but the average guess is often remarkably close.

- **Tabular data** means rows and columns, like a spreadsheet of customers or loans.
- On this kind of data, forests and boosted trees often match or beat neural networks, and train in minutes on a laptop.
- They cope with mixed columns (numbers and categories) and need little tuning.
- You can also see which features mattered most, which helps explain the model.

**Quick check:** Why does a random forest usually generalise better than a single deep tree?

A. It uses a neural network inside each tree
B. Its trees see different samples and features, so their errors average out when they vote
C. It always uses fewer features
D. It memorises the training data more completely

Answer: B. Each tree overfits in a slightly different way. Voting across many varied trees smooths out those individual quirks.

**Key takeaways**

- A decision tree predicts by asking a chain of questions, choosing splits that make groups purer.
- Single deep trees overfit; random forests and boosted trees combine many trees to fix this.
- On spreadsheet-style data, tree ensembles are still often the best tool.

**Go deeper (free sources)**

- [Decision Trees](https://scikit-learn.org/stable/modules/tree.html) by scikit-learn
- [Ensembles: gradient boosting, random forests, bagging](https://scikit-learn.org/stable/modules/ensemble.html) by scikit-learn
- [Decision Forests course](https://developers.google.com/machine-learning/decision-forests) by Google
- [Why do tree-based models still outperform deep learning on tabular data?](https://arxiv.org/abs/2207.08815) by Grinsztajn, Oyallon and Varoquaux (2022)