Intermediate · 02

Gradient descent: rolling downhill

How models find good weights: feel the slope, take a small step down, repeat.

6 min read · Tap underlined words for a definition

Picture yourself on a foggy hillside at night. You can't see the valley, but you can feel which way the ground slopes under your feet. Step downhill. Feel again. Step again. That's gradient descent, the algorithm that trains almost every model in existence.

loss ↑weight →start← minimum ✓steep slope → big stepsgentle slope → tiny steps
The loss curve is the hill. Each step moves the weight a little in the downhill direction until we settle near the bottom.
Try different learning rates and starting points.
Run itGradient descent in ten lines
Edit anything, then run. Python runs in your browser.

Set learning_rate to 0.1 and run it. The steps overshoot and w blows up, exactly like the ball in the explainer above.

The gradient

The gradient is the slope of the loss with respect to each weight. It points uphill.

So we move in the opposite direction: w_new = w_old − learning_rate × gradient.

The learning rate

The learning rate is the step size.

Too small: you'll get there, eventually, in a few weeks. Too big: you leap over the valley and bounce around or fly off entirely.

Picking it well is one of the most important practical skills in .

Quick check

Training loss is jumping up and down wildly and getting worse. What's the most likely fix?

Mini-batches (SGD)

Computing the on the whole dataset each step is slow. Instead we use a small random mini-batch (say 32 or 256 examples).

This is stochastic gradient descent (SGD). The noisy steps are faster, and the noise even helps escape bad spots.

Fancier optimisers

Momentum keeps rolling in a consistent direction, like a ball gathering speed.

Adam adapts the step size for every weight individually. It's the default choice for most deep learning today.

Quick check

Which way does gradient descent step?

Before you drift off

  • Gradient = slope of the loss; it points uphill.
  • Update: w ← w − learning_rate × gradient.
  • Learning rate too high → bouncing; too low → crawling.
  • Mini-batches (SGD) and Adam make it fast in practice.

Terms in this lesson

Go deeper, free and open

That's the lesson

Gradient descent: rolling downhill · 6 min

Each lesson you finish lights a star.

Finish & continue · Up nextOverfitting & the Goldilocks modelIntermediate · 5 min

Spotted a mistake? Suggest an edit

Previous: Lines and curves: regression