Intermediate · 11

Decision trees and random forests

A game of twenty questions that learns from data, and why a whole forest of them still beats neural networks on spreadsheets.

7 min read · Tap underlined words for a definition

Not all is . A decision tree makes predictions by asking a chain of yes or no questions about the data: "Is the customer under 30?" Then "Have they bought before?" Each answer sends you down a branch, until you reach a leaf that gives the prediction.

How a tree picks its questions

Training a tree means choosing good questions. At each step, the algorithm tries many possible splits and keeps the one that best separates the answers.

A good split leaves each side as "pure" as possible: mostly yeses on one side, mostly noes on the other. Then it repeats the process on each side.

Gini impurity, gently

Gini impurity measures how mixed a group is. Pick two items at random from the group: Gini is the chance they have different labels.

A group of all yeses has a Gini of 0, perfectly pure. A 50/50 mix of two labels has a Gini of 0.5, as messy as it gets. The tree picks the split that lowers impurity the most.

Run itFinding the best first question

# Should we play tennis? (outlook, windy) -> yes/no
data = [("sunny", False, "yes"), ("sunny", True, "no"), ("cloudy", False, "yes"),
        ("cloudy", True, "yes"), ("rainy", False, "yes"), ("rainy", True, "no"),
        ("sunny", False, "yes"), ("rainy", True, "no")]

def gini(rows):
    if not rows:
        return 0.0
    p = sum(r[2] == "yes" for r in rows) / len(rows)
    return 1 - p**2 - (1 - p)**2

print(f"Gini before any split: {gini(data):.3f}")
questions = {"outlook is sunny?": lambda r: r[0] == "sunny",
             "outlook is cloudy?": lambda r: r[0] == "cloudy",
             "is it windy?": lambda r: r[1]}
for name, ask in questions.items():
    left = [r for r in data if ask(r)]
    right = [r for r in data if not ask(r)]
    score = (len(left) * gini(left) + len(right) * gini(right)) / len(data)
    print(f"{name:20} -> weighted Gini {score:.3f}")

Edit anything, then run. Python runs in your browser.

The windy question splits the days most cleanly, so a tree would ask it first. Try changing a label and see which question wins.

Quick check

A group at a leaf contains 10 "yes" examples and 0 "no" examples. What is its Gini impurity?

Deep trees overfit

Let a tree keep asking questions and it can carve out a leaf for every single training example. It scores perfectly on data it has seen, then stumbles on new data.

The usual fixes are limiting the depth (how many questions in a row) or requiring a minimum number of examples per leaf.

Many trees beat one

A random forest trains hundreds of trees, each on a random resample of the data and each looking at a random subset of features at every split. Training on resamples like this is called bagging. To predict, the trees vote.

Each tree is a bit wrong in its own way. Averaged together, the mistakes cancel out. Combining models like this is called an ensemble.

Gradient-boosted trees take another route: they add small trees one after another, each trained to fix the errors of the trees so far. Libraries like XGBoost and LightGBM do this.

Tabular data means rows and columns, like a spreadsheet of customers or loans.
On this kind of data, forests and boosted trees often match or beat neural networks, and train in minutes on a laptop.
They cope with mixed columns (numbers and categories) and need little tuning.
You can also see which features mattered most, which helps explain the model.

Quick check

Why does a random forest usually generalise better than a single deep tree?

Before you drift off

A decision tree predicts by asking a chain of questions, choosing splits that make groups purer.
Single deep trees overfit; random forests and boosted trees combine many trees to fix this.
On spreadsheet-style data, tree ensembles are still often the best tool.

Terms in this lesson

Go deeper, free and open

Decision Trees · scikit-learn
Ensembles: gradient boosting, random forests, bagging · scikit-learn
Decision Forests course · Google · Free
Why do tree-based models still outperform deep learning on tabular data? · Grinsztajn, Oyallon and Varoquaux (2022)

That's the lesson

Decision trees and random forests · 7 min

Finish & continue · Up nextFinding groups: clusteringIntermediate · 6 min

Was this lesson clear?

Spotted a mistake? Suggest an edit

Previous: Open-weight models and running AI yourself