# Reinforcement learning, deeper · AI in Bed

Advanced level · 7 min · https://www.aiinbed.com/learn/advanced/reinforcement-learning

*Policies, values and rewards: the maths of learning by doing.*

Reinforcement learning (RL) trained AlphaGo, teaches robots to walk, and is now central to making LLMs reason. Here's the vocabulary and the core ideas.

*Figure: The agent observes a state, takes an action, and the environment returns a reward and a new state. Repeat.*

### The vocabulary

**State** s: what the agent observes. **Action** a: what it does. **Reward** r: the score signal.

**Policy** π(a|s): the agent's strategy, a mapping from states to actions.

**Return**: total future reward, usually **discounted** by γ (e.g. 0.99) so sooner rewards count more.

### Value functions

**V(s)**: how good is it to be in this state, if I follow my policy?

**Q(s, a)**: how good is taking action a in state s?

**Q-learning** learns Q directly; **DQN** used a neural net for Q and learned to play Atari games from pixels.

**Quick check:** What does a policy define?

A. The reward the environment gives
B. Which action the agent takes in each state
C. The size of the neural network
D. The discount factor

Answer: B. The policy is the agent's behaviour: given a state, which action (or distribution over actions) to take.

### Policy gradients

Instead of learning values, directly nudge the policy to make high-reward actions more likely.

**PPO** (proximal policy optimisation) adds a safety clip so each update doesn't change the policy too much. It's the workhorse behind classic RLHF.

### Explore vs. exploit

Keep ordering your favourite dish (exploit) or try something new that might be better (explore)?

Agents need both. Simple trick: ε-greedy, which takes a random action with small probability ε.

### Reward hacking

Agents optimise exactly what you reward, not what you meant. A boat-racing agent famously learned to spin in circles collecting points instead of finishing the race.

Designing rewards that can't be gamed is a central challenge, especially for LLMs.

**Quick check:** With discount factor γ = 0.9, a reward of 10 received two steps in the future is worth…

A. 10
B. 9
C. 8.1
D. 0.81

Answer: C. 10 × 0.9 × 0.9 = 8.1. Each step of delay multiplies by γ.

**Key takeaways**

- Agent ↔ environment: state, action, reward, next state.
- Value methods learn how good states/actions are; policy methods learn behaviour directly.
- PPO is the go-to policy gradient algorithm.
- Balance exploration and exploitation; beware reward hacking.

**Go deeper (free sources)**

- [Reinforcement Learning: An Introduction (2nd ed., free)](http://incompleteideas.net/book/the-book-2nd.html) by Richard Sutton & Andrew Barto
- [Spinning Up in Deep RL](https://spinningup.openai.com/) by OpenAI
- [Hugging Face Deep RL Course](https://huggingface.co/learn/deep-rl-course) by Hugging Face
- [Specification gaming examples](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/) by Google DeepMind
