# Gradient descent: rolling downhill · AI in Bed

Intermediate level · 6 min · https://www.aiinbed.com/learn/intermediate/gradient-descent

*How models find good weights: feel the slope, take a small step down, repeat.*

Picture yourself on a foggy hillside at night. You can't see the valley, but you can feel which way the ground slopes under your feet. Step downhill. Feel again. Step again. That's **gradient descent**, the algorithm that trains almost every model in existence.

*Figure: The loss curve is the hill. Each step moves the weight a little in the downhill direction until we settle near the bottom.*

*Interactive explainer: Try different learning rates and starting points.*

**Gradient descent in ten lines**

```python
# Learn w in y = w * x from data, one small step at a time.
data = [(1, 3), (2, 6), (3, 9), (4, 12)]   # the true w is 3
w = 0.0
learning_rate = 0.02

for step in range(30):
    # slope of the mean squared error with respect to w
    grad = sum(2 * (w * x - y) * x for x, y in data) / len(data)
    w -= learning_rate * grad
    if step % 5 == 0:
        loss = sum((w * x - y) ** 2 for x, y in data) / len(data)
        print(f"step {step:2d}  w = {w:.3f}  loss = {loss:.3f}")

print("final w:", round(w, 3))
```

Set learning_rate to 0.1 and run it. The steps overshoot and w blows up, exactly like the ball in the explainer above.

### The gradient

The **gradient** is the slope of the loss with respect to each weight. It points *uphill*.

So we move in the **opposite** direction: `w_new = w_old − learning_rate × gradient`.

### The learning rate

The **learning rate** is the step size.

Too small: you'll get there, eventually, in a few weeks. Too big: you leap over the valley and bounce around or fly off entirely.

Picking it well is one of the most important practical skills in deep learning.

**Quick check:** Training loss is jumping up and down wildly and getting worse. What's the most likely fix?

A. Increase the learning rate
B. Decrease the learning rate
C. Remove the loss function
D. Add more test data

Answer: B. Wild bouncing usually means steps are too big and keep overshooting the minimum. Smaller steps calm it down.

### Mini-batches (SGD)

Computing the gradient on the whole dataset each step is slow. Instead we use a small random **mini-batch** (say 32 or 256 examples).

This is **stochastic gradient descent** (SGD). The noisy steps are faster, and the noise even helps escape bad spots.

### Fancier optimisers

**Momentum** keeps rolling in a consistent direction, like a ball gathering speed.

**Adam** adapts the step size for every weight individually. It's the default choice for most deep learning today.

> Pillow thought: real loss landscapes have millions of dimensions, not one. Nobody can picture that, but the foggy hillside rule still works: feel the slope, step down.

**Quick check:** Which way does gradient descent step?

A. In the direction of the gradient
B. Opposite to the gradient
C. In a random direction every time
D. It doesn't move, it just measures

Answer: B. The gradient points uphill (towards higher loss), so we step the opposite way to reduce loss.

**Key takeaways**

- Gradient = slope of the loss; it points uphill.
- Update: w ← w − learning_rate × gradient.
- Learning rate too high → bouncing; too low → crawling.
- Mini-batches (SGD) and Adam make it fast in practice.

**Go deeper (free sources)**

- [Gradient descent, how neural networks learn](https://www.3blue1brown.com/lessons/gradient-descent) by 3Blue1Brown
- [Why Momentum Really Works](https://distill.pub/2017/momentum/) by Gabriel Goh, Distill
- [Dive into Deep Learning, ch. 12: Optimisation](https://d2l.ai/chapter_optimization/index.html) by Zhang, Lipton, Li & Smola
