# Attention is all you need · AI in Bed

Advanced level · 8 min · https://www.aiinbed.com/learn/advanced/transformers

*The transformer architecture and the attention mechanism that powers every modern LLM.*

In 2017 a Google paper titled *Attention Is All You Need* introduced the **transformer**. It threw away recurrence entirely and let every token look directly at every other token. Nearly every frontier model since, from GPT to Claude to Gemini, is built on it.

### The intuition

In "The animal didn't cross the street because **it** was too tired", what does "it" refer to?

Attention lets the model, while processing "it", look back and put most of its focus on "animal".

*Figure: Attention weights for the token "it": the model attends most strongly to "animal". Darker = more attention.*

*Interactive explainer: See which words each word attends to.*

**Attention scores in pure Python**

```python
import math

# Tiny 2-number "query" and "key" vectors for each word.
words = ["the", "cat", "sat", "it"]
keys  = {"the": [0.1, 0.0], "cat": [0.9, 0.8], "sat": [0.3, 0.9], "it": [0.5, 0.4]}
query = [1.0, 0.7]   # what "it" is looking for: a noun, an animal

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

scores = [dot(query, keys[w]) for w in words]
exps = [math.exp(s) for s in scores]
weights = [e / sum(exps) for e in exps]   # softmax

for w, s, a in zip(words, scores, weights):
    print(f"{w:>4}  score {s:.2f}  attention {a:.0%}  " + "#" * round(a * 40))
```

Change the query vector and see the attention shift. In a real transformer, queries and keys are learned and have hundreds of numbers.

### Queries, keys and values

Each token produces three vectors: a **query** (what am I looking for?), a **key** (what do I contain?), and a **value** (what will I share?).

Score = query · key for every pair, scaled and softmaxed into weights. Output = weighted sum of values.

`Attention(Q,K,V) = softmax(QKᵀ / √d) · V`

**Quick check:** In attention, what is the softmax applied to?

A. The values directly
B. The query–key similarity scores, turning them into weights that sum to 1
C. The final output text
D. The learning rate

Answer: B. Softmax turns raw query·key scores into a probability-like distribution over which tokens to attend to.

### Multi-head attention

Run several attention operations in parallel, each with its own Q/K/V projections.

Different heads can specialise: one tracks grammar, another tracks which noun a pronoun refers to, another tracks position.

### The transformer block

Each block = multi-head attention → feed-forward network (an MLP applied to each token), with **residual connections** and **layer normalisation** around both.

Stack dozens of blocks. Add **positional encodings** so the model knows word order, since attention itself is order-blind.

### Decoder-only & causal masking

GPT-style LLMs use only the decoder half, with a **causal mask**: each token may only attend to earlier tokens, never peek at the future.

That's what makes next-token prediction a fair game during training.

> Pillow thought: attention compares every token with every other one, so its cost grows with the square of the context length. A lot of current research (FlashAttention, sparse and linear attention) is about taming that.

**Quick check:** Why do transformers need positional encodings?

A. To make the model larger
B. Because attention alone has no sense of word order
C. To encrypt the input
D. They don't, it's optional decoration

Answer: B. Attention treats the input as a set. Without position information, "dog bites man" and "man bites dog" would look the same.

**Key takeaways**

- Attention lets each token pull information from any other token.
- softmax(QKᵀ/√d)·V: score, normalise, mix.
- Multi-head attention + MLP + residuals + norms = a transformer block.
- LLMs are decoder-only transformers with causal masking.

**Go deeper (free sources)**

- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) by Vaswani et al., 2017
- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/) by Jay Alammar
- [Transformer Explainer (interactive)](https://poloclub.github.io/transformer-explainer/) by Georgia Tech Polo Club
- [Let's build GPT: from scratch, in code](https://karpathy.ai/zero-to-hero.html) by Andrej Karpathy
