Advanced · 02
How LLMs are trained
Large language models are trained in three stages. Pre-training teaches the model to predict the next token on a huge amount of text; supervised fine-tuning teaches it to follow instructions from example conversations; and learning from feedback (like RLHF) nudges it towards answers people prefer.
7 min read · Tap underlined words for a definition
Swipe up to begin
A chat assistant isn't trained in one go. It's built in stages, each with a different goal and very different kinds of data.
This bigram model only looks one letter back, so it babbles. LLMs do the same job, predicting the next token, but look back over thousands of tokens with billions of parameters.
Pre-training
Next- prediction on trillions of tokens of web pages, books, code and more.
Takes thousands of GPUs for weeks or months. Produces a base model: vast knowledge, but it just continues text. Ask it a question and it might reply with more questions.
Supervised fine-tuning (SFT)
Train on high-quality example conversations: a and an ideal response, written or curated by people (and increasingly by other models).
The model learns the format and style of being a helpful assistant.
Learning from feedback
RLHF: humans compare pairs of answers; a reward model learns to predict their preferences; RL (classically PPO) optimises the against it.
DPO skips the separate reward model and learns directly from preference pairs.
Constitutional AI / RLAIF uses a written set of principles and AI feedback to scale this up.
What does a base model (only pre-trained) typically do?
Reasoning models
Newer models are also trained with RL on tasks with checkable answers (maths, code) to produce long chains of thought before answering.
Spending more compute at inference time (thinking longer) turns out to boost accuracy on hard problems.
Data is still king
Pre- is filtered, de-duplicated and mixed carefully (how much code? how many languages?).
Open datasets like FineWeb and The Pile let researchers study this in public.
In RLHF, what does the reward model learn?
Before you drift off
- Pre-training → base model with broad knowledge.
- SFT → assistant format and style.
- RLHF / DPO / RLAIF → aligned with preferences.
- RL on verifiable tasks produces stronger reasoning.
Terms in this lesson
Go deeper, free and open
- Training language models to follow instructions (InstructGPT) · Ouyang et al., 2022
- Direct Preference Optimization · Rafailov et al., 2023
- Constitutional AI: Harmlessness from AI Feedback · Bai et al. (Anthropic), 2022
- Illustrating RLHF · Hugging Face
- FineWeb dataset · Hugging Face · ODC-By 1.0
Questions
What is pre-training an LLM?
Pre-training is the first and most expensive stage: the model reads trillions of tokens of text and learns to predict the next one. That is where its knowledge and fluency come from.
What is RLHF?
Reinforcement learning from human feedback. People rank the model's answers, a reward model learns those preferences, and the LLM is trained to score well on it. It makes chatbots more helpful and polite.
How long does it take to train an LLM?
Frontier models train for weeks to months on thousands of GPUs. Fine-tuning an existing open model for a task can take hours on a single GPU with methods like LoRA.
That's the lesson
How LLMs are trained · 7 min
Each lesson you finish lights a star.
Spotted a mistake? Suggest an edit
Previous: Attention is all you need