Advanced · 13

Distillation and the rise of small models

How a big model can teach a small one, why small models keep getting better, and when a little model is the right choice.

7 min read · Tap underlined words for a definition

Big models are clever but slow and costly. Often we want something that fits on a phone or answers in a blink. The trick is to let a big model pass on what it knows to a small one.

Teacher and student

In knowledge distillation, a large trained model, the teacher, produces answers. A smaller model, the student, is trained to copy them.

The idea was popularised by Hinton and colleagues in 2015. With language models, the student often learns from the teacher's predicted probabilities for the next , or simply from lots of text the teacher has written.

Why soft labels help

A normal training label is hard: this photo is a cat, full stop. The teacher gives soft labels: mostly cat, a bit dog, a little fox, almost certainly not a car.

Those small numbers carry hidden knowledge about how things relate. Raising the softmax spreads them out so the student can see them more clearly.

Run itSoft labels at two temperatures

import math

def softmax(scores, temperature=1.0):
    exps = [math.exp(s / temperature) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

labels = ["cat", "dog", "fox", "car"]
teacher_scores = [6.0, 4.0, 3.0, -2.0]   # the teacher's raw scores for one photo

for t in [1, 4]:
    probs = softmax(teacher_scores, t)
    row = "  ".join(f"{name} {p:.0%}" for name, p in zip(labels, probs))
    print(f"T={t}:  {row}")

print("Hard label: cat 100%, everything else 0%")

Edit anything, then run. Python runs in your browser.

At a higher temperature the teacher's view that a cat looks more like a dog than a car shows up clearly. That is the extra signal the student learns from.

Quick check

What does a student model gain from a teacher's soft labels that a hard label lacks?

Other ways to shrink a model

Pruning removes or whole parts of the network that contribute little, then often retrains briefly to recover. Quantisation stores each weight with fewer bits.

These combine well: distil a small model, prune it, then quantise it to run on modest hardware.

Why small models keep improving

Today's small models often beat much larger models from a few years ago. Reasons include training on far more and better-filtered data, learning from stronger teachers, and better training recipes.

This makes on-device AI practical: models that run on your phone or laptop, working offline and keeping your data on the device.

Choose small when speed, cost, privacy or offline use matter, and the task is narrow, such as sorting emails or pulling out dates.
Choose large for open-ended reasoning, tricky writing, broad knowledge, or tasks you cannot predict in advance.
A common pattern: let a small model handle the easy majority and pass hard cases up to a large one.

Quick check

Which job is the best fit for a small on-device model?

Before you drift off

Distillation trains a small student to copy a big teacher's soft answers.
Pruning and quantisation shrink models further.
Pick small for narrow, fast, private tasks and large for open-ended ones.

Terms in this lesson

Go deeper, free and open

Distilling the Knowledge in a Neural Network · Hinton, Vinyals and Dean
DistilBERT, a distilled version of BERT · Sanh et al. · From Hugging Face
Learning both Weights and Connections for Efficient Neural Networks · Han et al. · Classic pruning paper

That's the lesson

Distillation and the rise of small models · 7 min

Finish & continue · Up nextAI with a body: robots that learnAdvanced · 8 min

Was this lesson clear?

Spotted a mistake? Suggest an edit

Previous: Mixture of experts: big models, small bills