Advanced · 06
Reinforcement learning, deeper
Reinforcement learning (RL) is how an AI learns by doing: an agent tries actions, gets rewards or penalties, and gradually learns a policy (a way of choosing actions) that earns the most reward over time. It is behind game-playing AIs like AlphaGo and the feedback stage of training chatbots.
7 min read · Tap underlined words for a definition
Swipe up to begin
(RL) trained AlphaGo, teaches robots to walk, and is now central to making LLMs reason. Here's the vocabulary and the core ideas.
The vocabulary
State s: what the agent observes. Action a: what it does. Reward r: the score signal.
Policy π(a|s): the agent's strategy, a mapping from states to actions.
Return: total future reward, usually discounted by γ (e.g. 0.99) so sooner rewards count more.
Value functions
V(s): how good is it to be in this state, if I follow my policy?
Q(s, a): how good is taking action a in state s?
Q-learning learns Q directly; DQN used a neural net for Q and learned to play Atari games from pixels.
What does a policy define?
Policy gradients
Instead of learning values, directly nudge the policy to make high-reward actions more likely.
PPO (proximal policy optimisation) adds a safety clip so each update doesn't change the policy too much. It's the workhorse behind classic .
Explore vs. exploit
Keep ordering your favourite dish (exploit) or try something new that might be better (explore)?
Agents need both. Simple trick: ε-greedy, which takes a random action with small probability ε.
Reward hacking
optimise exactly what you reward, not what you meant. A boat-racing agent famously learned to spin in circles collecting points instead of finishing the race.
Designing rewards that can't be gamed is a central challenge, especially for .
With discount factor γ = 0.9, a reward of 10 received two steps in the future is worth…
Before you drift off
- Agent ↔ environment: state, action, reward, next state.
- Value methods learn how good states/actions are; policy methods learn behaviour directly.
- PPO is the go-to policy gradient algorithm.
- Balance exploration and exploitation; beware reward hacking.
Terms in this lesson
Go deeper, free and open
- Reinforcement Learning: An Introduction (2nd ed., free) · Richard Sutton & Andrew Barto
- Spinning Up in Deep RL · OpenAI · MIT licensed code
- Hugging Face Deep RL Course · Hugging Face · Apache 2.0
- Specification gaming examples · Google DeepMind
Questions
What is reinforcement learning in simple terms?
Learning by trial and error. Nobody shows the AI the right answer; it tries things, sees which ones lead to reward, and does more of those.
What is the difference between a policy and a value function?
A policy says what to do in each situation. A value function estimates how much future reward a situation (or an action in it) is worth. Many RL methods learn both.
Is reinforcement learning used in ChatGPT?
Yes. After pre-training and fine-tuning, chatbots are improved with reinforcement learning from human feedback (RLHF) and similar methods.
That's the lesson
Reinforcement learning, deeper · 7 min
Each lesson you finish lights a star.
Spotted a mistake? Suggest an edit
Previous: Diffusion: art from noise