Advanced · 01

Attention is all you need

The transformer architecture and the attention mechanism that powers every modern LLM.

8 min read · Tap underlined words for a definition

In 2017 a Google paper titled Attention Is All You Need introduced the transformer. It threw away recurrence entirely and let every token look directly at every other token. Nearly every frontier model since, from GPT to Claude to Gemini, is built on it.

The intuition

In "The animal didn't cross the street because it was too tired", what does "it" refer to?

lets the model, while processing "it", look back and put most of its focus on "animal".

Where does “it” look?The2%animal46%didn't3%cross4%the1%street14%because5%it12%was4%too2%tired7%
Attention weights for the token "it": the model attends most strongly to "animal". Darker = more attention.
See which words each word attends to.
Run itAttention scores in pure Python
Edit anything, then run. Python runs in your browser.

Change the query vector and see the attention shift. In a real transformer, queries and keys are learned and have hundreds of numbers.

Queries, keys and values

Each produces three vectors: a query (what am I looking for?), a key (what do I contain?), and a value (what will I share?).

Score = query · key for every pair, scaled and softmaxed into . Output = weighted sum of values.

Attention(Q,K,V) = softmax(QKᵀ / √d) · V

Quick check

In attention, what is the softmax applied to?

Multi-head attention

Run several attention operations in parallel, each with its own Q/K/V projections.

Different heads can specialise: one tracks grammar, another tracks which noun a pronoun refers to, another tracks position.

The transformer block

Each block = multi-head attention → feed-forward network (an MLP applied to each token), with residual connections and layer normalisation around both.

Stack dozens of blocks. Add positional encodings so the model knows word order, since attention itself is order-blind.

Decoder-only & causal masking

GPT-style use only the decoder half, with a causal mask: each token may only attend to earlier tokens, never peek at the future.

That's what makes next-token prediction a fair game during training.

Quick check

Why do transformers need positional encodings?

Before you drift off

  • Attention lets each token pull information from any other token.
  • softmax(QKᵀ/√d)·V: score, normalise, mix.
  • Multi-head attention + MLP + residuals + norms = a transformer block.
  • LLMs are decoder-only transformers with causal masking.

Terms in this lesson

Go deeper, free and open

That's the lesson

Attention is all you need · 8 min

Each lesson you finish lights a star.

Finish & continue · Up nextHow LLMs are trainedAdvanced · 7 min

Spotted a mistake? Suggest an edit