AI glossary
What is attention?
Short answer
A mechanism that lets each token weigh and pull information from other tokens. The heart of the transformer.
For example
In "The cat didn't eat because it was full", attention lets the word "it" look back and pull most of its meaning from "cat" rather than from "eat".
Learn it in a free lesson
- Attention is all you needAdvanced · 8 min
- Diffusion: art from noiseAdvanced · 6 min
- Seeing and remembering: CNNs & RNNsIntermediate · 7 min
- How models are served: prefill, decode and the KV cacheAdvanced · 8 min
Related terms
Questions
What is attention in simple terms?
A mechanism that lets each token weigh and pull information from other tokens. The heart of the transformer.
What is an example of attention?
In "The cat didn't eat because it was full", attention lets the word "it" look back and pull most of its meaning from "cat" rather than from "eat".
Where can I learn about attention for free?
AI in Bed explains it in the free lesson "Attention is all you need" (Advanced level, 8 minutes), with no sign-up.