# Distillation and the rise of small models · AI in Bed

Advanced level · 7 min · https://www.aiinbed.com/learn/advanced/distillation-and-small-models

*How a big model can teach a small one, why small models keep getting better, and when a little model is the right choice.*

Big models are clever but slow and costly. Often we want something that fits on a phone or answers in a blink. The trick is to let a big model pass on what it knows to a small one.

### Teacher and student

In **knowledge distillation**, a large trained model, the **teacher**, produces answers. A smaller model, the **student**, is trained to copy them.

The idea was popularised by Hinton and colleagues in 2015. With language models, the student often learns from the teacher's predicted probabilities for the next token, or simply from lots of text the teacher has written.

### Why soft labels help

A normal training label is hard: this photo is a cat, full stop. The teacher gives **soft labels**: mostly cat, a bit dog, a little fox, almost certainly not a car.

Those small numbers carry hidden knowledge about how things relate. Raising the softmax temperature spreads them out so the student can see them more clearly.

**Soft labels at two temperatures**

```python
import math

def softmax(scores, temperature=1.0):
    exps = [math.exp(s / temperature) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

labels = ["cat", "dog", "fox", "car"]
teacher_scores = [6.0, 4.0, 3.0, -2.0]   # the teacher's raw scores for one photo

for t in [1, 4]:
    probs = softmax(teacher_scores, t)
    row = "  ".join(f"{name} {p:.0%}" for name, p in zip(labels, probs))
    print(f"T={t}:  {row}")

print("Hard label: cat 100%, everything else 0%")
```

At a higher temperature the teacher's view that a cat looks more like a dog than a car shows up clearly. That is the extra signal the student learns from.

**Quick check:** What does a student model gain from a teacher's soft labels that a hard label lacks?

A. A faster chip to run on
B. Information about which wrong answers are close and which are far off
C. A larger context window
D. Permission to skip training

Answer: B. Soft labels show how similar the classes look to the teacher, which is richer than a single right answer.

### Other ways to shrink a model

**Pruning** removes weights or whole parts of the network that contribute little, then often retrains briefly to recover. **Quantisation** stores each weight with fewer bits.

These combine well: distil a small model, prune it, then quantise it to run on modest hardware.

### Why small models keep improving

Today's small models often beat much larger models from a few years ago. Reasons include training on far more and better-filtered data, learning from stronger teachers, and better training recipes.

This makes **on-device AI** practical: models that run on your phone or laptop, working offline and keeping your data on the device.

- Choose small when speed, cost, privacy or offline use matter, and the task is narrow, such as sorting emails or pulling out dates.
- Choose large for open-ended reasoning, tricky writing, broad knowledge, or tasks you cannot predict in advance.
- A common pattern: let a small model handle the easy majority and pass hard cases up to a large one.

**Quick check:** Which job is the best fit for a small on-device model?

A. Writing a long legal analysis of a novel case
B. Suggesting the next word on a phone keyboard, offline
C. Planning a multi-step research project
D. Answering any question on any subject

Answer: B. Keyboard suggestions are narrow, need to be instant, and benefit from staying private on the device.

**Key takeaways**

- Distillation trains a small student to copy a big teacher's soft answers.
- Pruning and quantisation shrink models further.
- Pick small for narrow, fast, private tasks and large for open-ended ones.

**Go deeper (free sources)**

- [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) by Hinton, Vinyals and Dean
- [DistilBERT, a distilled version of BERT](https://arxiv.org/abs/1910.01108) by Sanh et al.
- [Learning both Weights and Connections for Efficient Neural Networks](https://arxiv.org/abs/1506.02626) by Han et al.