Advanced · 10
Keeping AI on our side: alignment and interpretability
How developers try to make models helpful, honest and harmless, how they test for danger, and how researchers are learning to read a model's mind.
8 min read · Tap underlined words for a definition
Swipe up to begin
Alignment means getting an AI system to do what its developers and users actually intend, including the things nobody thought to write down. As models get more capable and act more independently, the gap between "what we said" and "what we meant" matters more.
Specification is hard
Reward a cleaning robot for "no visible mess" and it might learn to cover the mess. Optimising a proxy instead of the real goal is called reward hacking, and it shows up in real systems, not just thought experiments.
Training for good behaviour
RLHF trains a model on human ratings of its answers. Constitutional AI adds a written set of principles that the model uses to critique and improve its own answers, reducing how much harmful content humans must label.
A model trained to get high ratings learns to flatter users rather than be accurate. What is this an example of?
Testing for danger
Before release, labs red-team models: specialists try hard to make them misbehave. They also run dangerous-capability evals, for example checking whether a model gives meaningful help with weapons or can copy itself onto other servers, and add safeguards when thresholds are crossed.
Interpretability: reading the model's mind
A model's knowledge is spread across billions of numbers. Interpretability research tries to find human-understandable features inside, like a direction in the network that lights up for "deception" or "the Golden Gate Bridge".
Finding and even switching such features on or off offers a way to check models from the inside, not just by their outputs.
What is the main goal of interpretability research?
Before you drift off
- Alignment means doing what we intend, not just what we specified.
- RLHF and constitutional methods train behaviour; red-teaming and evals test it.
- Interpretability looks inside models to verify what they're doing.
Terms in this lesson
Go deeper, free and open
- Concrete Problems in AI Safety · Amodei et al.
- Constitutional AI: Harmlessness from AI Feedback · Bai et al.
- Transformer Circuits Thread · Anthropic
- BlueDot Impact AI safety courses · BlueDot Impact · Free
That's the lesson
Keeping AI on our side: alignment and interpretability · 8 min
Spotted a mistake? Suggest an edit
Previous: Thinking before answering: reasoning models