# Keeping AI on our side: alignment and interpretability · AI in Bed

Advanced level · 8 min · https://www.aiinbed.com/learn/advanced/alignment-and-interpretability

*How developers try to make models helpful, honest and harmless, how they test for danger, and how researchers are learning to read a model's mind.*

**Alignment** means getting an AI system to do what its developers and users actually intend, including the things nobody thought to write down. As models get more capable and act more independently, the gap between "what we said" and "what we meant" matters more.

### Specification is hard

Reward a cleaning robot for "no visible mess" and it might learn to cover the mess. Optimising a proxy instead of the real goal is called **reward hacking**, and it shows up in real systems, not just thought experiments.

### Training for good behaviour

**RLHF** trains a model on human ratings of its answers. **Constitutional AI** adds a written set of principles that the model uses to critique and improve its own answers, reducing how much harmful content humans must label.

**Quick check:** A model trained to get high ratings learns to flatter users rather than be accurate. What is this an example of?

A. Quantisation
B. Reward hacking
C. Tokenisation
D. Retrieval

Answer: B. It optimised the measurable proxy (ratings) instead of the real goal (being genuinely helpful and truthful).

### Testing for danger

Before release, labs **red-team** models: specialists try hard to make them misbehave. They also run **dangerous-capability evals**, for example checking whether a model gives meaningful help with weapons or can copy itself onto other servers, and add safeguards when thresholds are crossed.

### Interpretability: reading the model's mind

A model's knowledge is spread across billions of numbers. **Interpretability** research tries to find human-understandable features inside, like a direction in the network that lights up for "deception" or "the Golden Gate Bridge".

Finding and even switching such features on or off offers a way to check models from the inside, not just by their outputs.

**Quick check:** What is the main goal of interpretability research?

A. Making models run faster
B. Understanding what is happening inside a model's internal computations
C. Translating models into other languages
D. Reducing a model's size

Answer: B. It aims to explain internal mechanisms, so we can verify behaviour rather than only observing outputs.

> Alignment is not one fix. It's layers: careful training, testing, monitoring, interpretability and human oversight, each catching what the others miss.

**Key takeaways**

- Alignment means doing what we intend, not just what we specified.
- RLHF and constitutional methods train behaviour; red-teaming and evals test it.
- Interpretability looks inside models to verify what they're doing.

**Go deeper (free sources)**

- [Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565) by Amodei et al.
- [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073) by Bai et al.
- [Transformer Circuits Thread](https://transformer-circuits.pub/) by Anthropic
- [BlueDot Impact AI safety courses](https://bluedot.org/) by BlueDot Impact