Advanced · 06

Reinforcement learning, deeper

Reinforcement learning (RL) is how an AI learns by doing: an agent tries actions, gets rewards or penalties, and gradually learns a policy (a way of choosing actions) that earns the most reward over time. It is behind game-playing AIs like AlphaGo and the feedback stage of training chatbots.

7 min read · Tap underlined words for a definition

(RL) trained AlphaGo, teaches robots to walk, and is now central to making LLMs reason. Here's the vocabulary and the core ideas.

🤖 Agentpolicy π(a|s)🌍 Environmentgame, robot, worldaction aₜnew state sₜ₊₁ + reward rₜ₊₁ ⭐goal: maximisetotal future reward
The agent observes a state, takes an action, and the environment returns a reward and a new state. Repeat.

The vocabulary

State s: what the agent observes. Action a: what it does. Reward r: the score signal.

Policy π(a|s): the agent's strategy, a mapping from states to actions.

Return: total future reward, usually discounted by γ (e.g. 0.99) so sooner rewards count more.

Value functions

V(s): how good is it to be in this state, if I follow my policy?

Q(s, a): how good is taking action a in state s?

Q-learning learns Q directly; DQN used a neural net for Q and learned to play Atari games from pixels.

Quick check

What does a policy define?

Policy gradients

Instead of learning values, directly nudge the policy to make high-reward actions more likely.

PPO (proximal policy optimisation) adds a safety clip so each update doesn't change the policy too much. It's the workhorse behind classic .

Explore vs. exploit

Keep ordering your favourite dish (exploit) or try something new that might be better (explore)?

Agents need both. Simple trick: ε-greedy, which takes a random action with small probability ε.

Reward hacking

optimise exactly what you reward, not what you meant. A boat-racing agent famously learned to spin in circles collecting points instead of finishing the race.

Designing rewards that can't be gamed is a central challenge, especially for .

Quick check

With discount factor γ = 0.9, a reward of 10 received two steps in the future is worth…

Before you drift off

  • Agent ↔ environment: state, action, reward, next state.
  • Value methods learn how good states/actions are; policy methods learn behaviour directly.
  • PPO is the go-to policy gradient algorithm.
  • Balance exploration and exploitation; beware reward hacking.

Terms in this lesson

Go deeper, free and open

Questions

What is reinforcement learning in simple terms?

Learning by trial and error. Nobody shows the AI the right answer; it tries things, sees which ones lead to reward, and does more of those.

What is the difference between a policy and a value function?

A policy says what to do in each situation. A value function estimates how much future reward a situation (or an action in it) is worth. Many RL methods learn both.

Is reinforcement learning used in ChatGPT?

Yes. After pre-training and fine-tuning, chatbots are improved with reinforcement learning from human feedback (RLHF) and similar methods.

That's the lesson

Reinforcement learning, deeper · 7 min

Each lesson you finish lights a star.

Finish & continue · Up nextScaling laws, evals & safetyAdvanced · 7 min

Spotted a mistake? Suggest an edit

Previous: Diffusion: art from noise