Advanced · 09
Thinking before answering: reasoning models
Why letting a model write out its thinking makes it smarter, and how reinforcement learning taught models to think for longer.
7 min read · Tap underlined words for a definition
Swipe up to begin
Ask a person "what's 17 × 24?" and demand an instant answer, and they'll often get it wrong. Let them use paper and they'll get it right. Language models are the same: every token they write is a little bit of computation, so writing out steps gives them more room to think.
Chain of thought
In 2022, researchers found that simply prompting models to "think step by step" sharply improved maths and logic scores. This is chain-of-thought prompting.
The intermediate steps act as a scratchpad the model can read back as it continues.
From prompting to training
Reasoning models are trained to do this by default. They're given problems with checkable answers, like maths with a known result or code with tests, and rewarded with reinforcement learning when the final answer is right.
Nobody tells them how to think. Over training they learn useful habits on their own: breaking problems down, checking work, backtracking with "wait, that's wrong".
Why does writing out intermediate steps help a language model?
Test-time compute
Old rule: to get a smarter model, train a bigger one. New lever: let the same model think longer at answer time. Spending more compute while answering, called test-time compute, keeps improving results on hard problems, up to a point.
The cost is time and money: a long reasoning trace can be thousands of .
Is the written thinking faithful?
The visible reasoning is useful but not a guaranteed window into the model. Research has found cases where the stated steps don't fully match what actually drove the answer, so reasoning traces should be read with care.
What kind of reward is easiest to use when training a reasoning model with RL?
Before you drift off
- Writing out steps gives models more computation and a scratchpad.
- Reasoning models learn to think via RL on checkable problems.
- Thinking longer at answer time is a new way to buy capability.
Terms in this lesson
Go deeper, free and open
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models · Wei et al.
- DeepSeek-R1: Incentivizing Reasoning Capability via RL · DeepSeek-AI · Open paper and weights
- Lil'Log · Lilian Weng · Posts on reasoning and test-time compute
That's the lesson
Thinking before answering: reasoning models · 7 min
Spotted a mistake? Suggest an edit
Previous: Agents: loops, tools and memory