Intermediate · 05

Backpropagation: blame, passed backwards

How a network with millions of weights figures out which ones to adjust.

6 min read · Tap underlined words for a definition

needs the slope for every weight. With millions of , how do we get them all efficiently? Backpropagation: one pass forward to make a prediction, one pass backward to share out the blame.

forward: make a predictionLosshow wrong?backward: share out the blame (gradients)inputhiddenoutput
Forward pass computes the prediction and loss. Backward pass sends error signals back through each layer to get every weight's gradient.

Forward pass

Input flows through the layers, each doing its weighted sums and activations, until we get a prediction.

Compare to the truth → loss.

Backward pass

Starting from the loss, ask each layer: "how much did you contribute to this error?"

The chain rule from calculus lets us compute this layer by layer, reusing work from the layer above. That reuse is why it's fast.

Quick check

What mathematical rule makes backpropagation work?

Vanishing & exploding gradients

Multiplying many small numbers through deep networks can shrink to almost zero (vanishing), so early layers stop learning. Multiplying big ones can blow up (exploding).

Fixes: ReLU activations, careful initialisation, normalisation layers, and residual connections (shortcuts that let gradients skip layers).

You rarely write it by hand

Frameworks like PyTorch and JAX do automatic differentiation: you write the forward pass and they compute all the gradients for you.

Still, building it once yourself (see Karpathy's micrograd below) makes everything click.

Quick check

Residual (skip) connections mainly help with…

Before you drift off

  • Forward pass → prediction & loss; backward pass → gradients.
  • The chain rule spreads blame layer by layer efficiently.
  • Residual connections and normalisation tame vanishing gradients.

Terms in this lesson

Go deeper, free and open

That's the lesson

Backpropagation: blame, passed backwards · 6 min

Each lesson you finish lights a star.

Finish & continue · Up nextSeeing and remembering: CNNs & RNNsIntermediate · 7 min

Spotted a mistake? Suggest an edit

Previous: Is my model any good?