# How LLMs are trained · AI in Bed

Advanced level · 7 min · https://www.aiinbed.com/learn/advanced/training-llms

*Pre-training, supervised fine-tuning, and learning from feedback: the three-stage recipe.*

A chat assistant isn't trained in one go. It's built in stages, each with a different goal and very different kinds of data.

*Figure: The typical recipe: a huge pre-training run, then much smaller but carefully curated post-training stages.*

**A tiny next-letter model**

```python
import random
from collections import Counter, defaultdict

text = "the cat sat on the mat. the dog sat on the log. the cat saw the dog."

# "Training": count which letter follows each letter.
counts = defaultdict(Counter)
for a, b in zip(text, text[1:]):
    counts[a][b] += 1

def next_letter(c):
    options = counts[c]
    return random.choices(list(options), weights=options.values())[0]

random.seed(4)
out = "t"
for _ in range(60):
    out += next_letter(out[-1])
print(out)
```

This bigram model only looks one letter back, so it babbles. LLMs do the same job, predicting the next token, but look back over thousands of tokens with billions of parameters.

### Pre-training

Next-token prediction on trillions of tokens of web pages, books, code and more.

Takes thousands of GPUs for weeks or months. Produces a **base model**: vast knowledge, but it just continues text. Ask it a question and it might reply with more questions.

### Supervised fine-tuning (SFT)

Train on high-quality example conversations: a prompt and an ideal response, written or curated by people (and increasingly by other models).

The model learns the *format* and *style* of being a helpful assistant.

### Learning from feedback

**RLHF**: humans compare pairs of answers; a **reward model** learns to predict their preferences; RL (classically PPO) optimises the LLM against it.

**DPO** skips the separate reward model and learns directly from preference pairs.

**Constitutional AI / RLAIF** uses a written set of principles and AI feedback to scale this up.

**Quick check:** What does a base model (only pre-trained) typically do?

A. Follow instructions perfectly
B. Continue text in a plausible way, not necessarily answer helpfully
C. Refuse everything
D. Only output code

Answer: B. Pre-training teaches prediction, not helpfulness. Post-training (SFT + feedback) shapes it into an assistant.

### Reasoning models

Newer models are also trained with RL on tasks with checkable answers (maths, code) to produce long chains of thought before answering.

Spending more compute *at inference time* (thinking longer) turns out to boost accuracy on hard problems.

### Data is still king

Pre-training data is filtered, de-duplicated and mixed carefully (how much code? how many languages?).

Open datasets like FineWeb and The Pile let researchers study this in public.

**Quick check:** In RLHF, what does the reward model learn?

A. The next token
B. To predict which responses humans prefer
C. To compress the training data
D. To translate languages

Answer: B. The reward model is trained on human preference comparisons, then used as the signal for reinforcement learning.

**Key takeaways**

- Pre-training → base model with broad knowledge.
- SFT → assistant format and style.
- RLHF / DPO / RLAIF → aligned with preferences.
- RL on verifiable tasks produces stronger reasoning.

**Go deeper (free sources)**

- [Training language models to follow instructions (InstructGPT)](https://arxiv.org/abs/2203.02155) by Ouyang et al., 2022
- [Direct Preference Optimization](https://arxiv.org/abs/2305.18290) by Rafailov et al., 2023
- [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073) by Bai et al. (Anthropic), 2022
- [Illustrating RLHF](https://huggingface.co/blog/rlhf) by Hugging Face
- [FineWeb dataset](https://huggingface.co/datasets/HuggingFaceFW/fineweb) by Hugging Face
