# Backpropagation: blame, passed backwards · AI in Bed

Intermediate level · 6 min · https://www.aiinbed.com/learn/intermediate/backpropagation

*How a network with millions of weights figures out which ones to adjust.*

Gradient descent needs the slope for *every* weight. With millions of weights, how do we get them all efficiently? **Backpropagation**: one pass forward to make a prediction, one pass backward to share out the blame.

*Figure: Forward pass computes the prediction and loss. Backward pass sends error signals back through each layer to get every weight's gradient.*

### Forward pass

Input flows through the layers, each doing its weighted sums and activations, until we get a prediction.

Compare to the truth → loss.

### Backward pass

Starting from the loss, ask each layer: "how much did you contribute to this error?"

The **chain rule** from calculus lets us compute this layer by layer, reusing work from the layer above. That reuse is why it's fast.

> Pillow thought: it's like a restaurant complaint. The customer tells the waiter, the waiter tells the chef, the chef checks which ingredient was off. Each person passes back a share of the blame.

**Quick check:** What mathematical rule makes backpropagation work?

A. Pythagoras' theorem
B. The chain rule
C. Bayes' theorem
D. The law of large numbers

Answer: B. The chain rule lets us multiply local derivatives layer by layer to get each weight's effect on the final loss.

### Vanishing & exploding gradients

Multiplying many small numbers through deep networks can shrink gradients to almost zero (**vanishing**), so early layers stop learning. Multiplying big ones can blow up (**exploding**).

Fixes: ReLU activations, careful initialisation, normalisation layers, and **residual connections** (shortcuts that let gradients skip layers).

### You rarely write it by hand

Frameworks like **PyTorch** and **JAX** do automatic differentiation: you write the forward pass and they compute all the gradients for you.

Still, building it once yourself (see Karpathy's *micrograd* below) makes everything click.

**Quick check:** Residual (skip) connections mainly help with…

A. Making images prettier
B. Letting gradients flow through very deep networks
C. Reducing the dataset size
D. Encrypting the weights

Answer: B. Skip connections give gradients a shortcut path, so very deep networks (100+ layers) can still train.

**Key takeaways**

- Forward pass → prediction & loss; backward pass → gradients.
- The chain rule spreads blame layer by layer efficiently.
- Residual connections and normalisation tame vanishing gradients.

**Go deeper (free sources)**

- [Neural Networks: Zero to Hero (micrograd lecture)](https://karpathy.ai/zero-to-hero.html) by Andrej Karpathy
- [What is backpropagation really doing?](https://www.3blue1brown.com/lessons/backpropagation) by 3Blue1Brown
- [CS231n: Backpropagation notes](https://cs231n.github.io/optimization-2/) by Stanford University
