AI glossary
What is token?
Short answer
A chunk of text (a word, part of a word or punctuation) that language models read and write.
For example
"Unbelievable!" might be split into the tokens "Un", "believ", "able" and "!". English text averages roughly three quarters of a word per token.
Learn it in a free lesson
- Meet the chatbots: LLMsBeginner · 6 min
- How text becomes tokensIntermediate · 7 min
- How models are served: prefill, decode and the KV cacheAdvanced · 8 min
- Mixture of experts: big models, small billsAdvanced · 8 min
- Attention is all you needAdvanced · 8 min
- Thinking before answering: reasoning modelsAdvanced · 7 min
Related terms
Questions
What is token in simple terms?
A chunk of text (a word, part of a word or punctuation) that language models read and write.
What is an example of token?
"Unbelievable!" might be split into the tokens "Un", "believ", "able" and "!". English text averages roughly three quarters of a word per token.
Where can I learn about token for free?
AI in Bed explains it in the free lesson "Meet the chatbots: LLMs" (Beginner level, 6 minutes), with no sign-up.