Intermediate · 02
Gradient descent: rolling downhill
How models find good weights: feel the slope, take a small step down, repeat.
6 min read · Tap underlined words for a definition
Swipe up to begin
Picture yourself on a foggy hillside at night. You can't see the valley, but you can feel which way the ground slopes under your feet. Step downhill. Feel again. Step again. That's gradient descent, the algorithm that trains almost every model in existence.
Set learning_rate to 0.1 and run it. The steps overshoot and w blows up, exactly like the ball in the explainer above.
The gradient
The gradient is the slope of the loss with respect to each weight. It points uphill.
So we move in the opposite direction: w_new = w_old − learning_rate × gradient.
The learning rate
The learning rate is the step size.
Too small: you'll get there, eventually, in a few weeks. Too big: you leap over the valley and bounce around or fly off entirely.
Picking it well is one of the most important practical skills in .
Training loss is jumping up and down wildly and getting worse. What's the most likely fix?
Mini-batches (SGD)
Computing the on the whole dataset each step is slow. Instead we use a small random mini-batch (say 32 or 256 examples).
This is stochastic gradient descent (SGD). The noisy steps are faster, and the noise even helps escape bad spots.
Fancier optimisers
Momentum keeps rolling in a consistent direction, like a ball gathering speed.
Adam adapts the step size for every weight individually. It's the default choice for most deep learning today.
Which way does gradient descent step?
Before you drift off
- Gradient = slope of the loss; it points uphill.
- Update: w ← w − learning_rate × gradient.
- Learning rate too high → bouncing; too low → crawling.
- Mini-batches (SGD) and Adam make it fast in practice.
Terms in this lesson
Go deeper, free and open
- Gradient descent, how neural networks learn · 3Blue1Brown
- Why Momentum Really Works · Gabriel Goh, Distill · CC BY 4.0, interactive
- Dive into Deep Learning, ch. 12: Optimisation · Zhang, Lipton, Li & Smola · CC BY-SA 4.0
That's the lesson
Gradient descent: rolling downhill · 6 min
Each lesson you finish lights a star.
Spotted a mistake? Suggest an edit
Previous: Lines and curves: regression