Advanced · 02

How LLMs are trained

Large language models are trained in three stages. Pre-training teaches the model to predict the next token on a huge amount of text; supervised fine-tuning teaches it to follow instructions from example conversations; and learning from feedback (like RLHF) nudges it towards answers people prefer.

7 min read · Tap underlined words for a definition

A chat assistant isn't trained in one go. It's built in stages, each with a different goal and very different kinds of data.

data size1 · Pre-trainingnext-token prediction, web scaletrillions of tokens2 · Supervised fine-tuninglearn from ideal example answers~100k examples3 · Preference tuningRLHF / DPO / AI feedbackcomparisons🤖 Assistantbase model is madein stage 1; stages2–3 shape behaviour
The typical recipe: a huge pre-training run, then much smaller but carefully curated post-training stages.
Run itA tiny next-letter model
Edit anything, then run. Python runs in your browser.

This bigram model only looks one letter back, so it babbles. LLMs do the same job, predicting the next token, but look back over thousands of tokens with billions of parameters.

Pre-training

Next- prediction on trillions of tokens of web pages, books, code and more.

Takes thousands of GPUs for weeks or months. Produces a base model: vast knowledge, but it just continues text. Ask it a question and it might reply with more questions.

Supervised fine-tuning (SFT)

Train on high-quality example conversations: a and an ideal response, written or curated by people (and increasingly by other models).

The model learns the format and style of being a helpful assistant.

Learning from feedback

RLHF: humans compare pairs of answers; a reward model learns to predict their preferences; RL (classically PPO) optimises the against it.

DPO skips the separate reward model and learns directly from preference pairs.

Constitutional AI / RLAIF uses a written set of principles and AI feedback to scale this up.

Quick check

What does a base model (only pre-trained) typically do?

Reasoning models

Newer models are also trained with RL on tasks with checkable answers (maths, code) to produce long chains of thought before answering.

Spending more compute at inference time (thinking longer) turns out to boost accuracy on hard problems.

Data is still king

Pre- is filtered, de-duplicated and mixed carefully (how much code? how many languages?).

Open datasets like FineWeb and The Pile let researchers study this in public.

Quick check

In RLHF, what does the reward model learn?

Before you drift off

  • Pre-training → base model with broad knowledge.
  • SFT → assistant format and style.
  • RLHF / DPO / RLAIF → aligned with preferences.
  • RL on verifiable tasks produces stronger reasoning.

Terms in this lesson

Go deeper, free and open

Questions

What is pre-training an LLM?

Pre-training is the first and most expensive stage: the model reads trillions of tokens of text and learns to predict the next one. That is where its knowledge and fluency come from.

What is RLHF?

Reinforcement learning from human feedback. People rank the model's answers, a reward model learns those preferences, and the LLM is trained to score well on it. It makes chatbots more helpful and polite.

How long does it take to train an LLM?

Frontier models train for weeks to months on thousands of GPUs. Fine-tuning an existing open model for a task can take hours on a single GPU with methods like LoRA.

That's the lesson

How LLMs are trained · 7 min

Each lesson you finish lights a star.

Finish & continue · Up nextRAG & AI agentsAdvanced · 7 min

Spotted a mistake? Suggest an edit

Previous: Attention is all you need