Reinforcement learning, deeper

📖 The vocabulary

State s: what the agent observes. Action a: what it does. Reward r: the score signal.

Policy π(a|s): the agent's strategy, a mapping from states to actions.

Return: total future reward, usually discounted by γ (e.g. 0.99) so sooner rewards count more.

💎 Value functions

V(s): how good is it to be in this state, if I follow my policy?

Q(s, a): how good is taking action a in state s?

Q-learning learns Q directly; DQN used a neural net for Q and learned to play Atari games from pixels.

🎯 Policy gradients

Instead of learning values, directly nudge the policy to make high-reward actions more likely.

PPO (proximal policy optimisation) adds a safety clip so each update doesn't change the policy too much. It's the workhorse behind classic RLHF.

🔍 Explore vs. exploit

Keep ordering your favourite dish (exploit) or try something new that might be better (explore)?

Agents need both. Simple trick: ε-greedy, which takes a random action with small probability ε.

⚠️ Reward hacking

Agents optimise exactly what you reward, not what you meant. A boat-racing agent famously learned to spin in circles collecting points instead of finishing the race.

Designing rewards that can't be gamed is a central challenge, especially for LLMs.