# How text becomes tokens · AI in Bed

Intermediate level · 7 min · https://www.aiinbed.com/learn/intermediate/tokenisation

*Language models don't read letters or words: they read tokens. How text gets chopped up, and why it explains some odd model mistakes.*

A language model only works with numbers. Before it sees your message, a **tokeniser** chops the text into pieces called tokens and swaps each one for an ID number from a fixed list, the **vocabulary**. How you chop matters more than you might think.

### Three ways to chop

**Characters**: a tiny vocabulary, but sequences get very long, and single letters carry little meaning.

**Whole words**: short sequences, but the vocabulary explodes, and any new word, typo or name is unknown.

**Subwords**: the middle path used by modern models. Common words stay whole; rare words split into familiar pieces, like "un" + "believ" + "able".

### Byte-pair encoding

**Byte-pair encoding** (BPE) builds a subword vocabulary from data. Start with single characters. Find the pair that appears side by side most often and glue it into a new token. Repeat thousands of times.

Frequent chunks like "ing", "the" and "tion" become single tokens. Many modern tokenisers start from raw bytes instead of characters, so any text, even emoji, can be encoded.

**A mini BPE merge loop**

```python
from collections import Counter
# Each word starts as a list of single characters
words = ["low", "low", "lower", "lowest", "newest", "newest", "widest"]
corpus = [list(w) for w in words]

for merge in range(5):
    pairs = Counter()
    for w in corpus:
        for a, b in zip(w, w[1:]):
            pairs[(a, b)] += 1
    (a, b), count = pairs.most_common(1)[0]
    print(f"merge {merge + 1}: '{a}' + '{b}' -> '{a + b}'  (seen {count} times)")
    new_corpus = []
    for w in corpus:  # glue every occurrence of the pair together
        out, i = [], 0
        while i < len(w):
            if i < len(w) - 1 and (w[i], w[i + 1]) == (a, b):
                out.append(a + b); i += 2
            else:
                out.append(w[i]); i += 1
        new_corpus.append(out)
    corpus = new_corpus
print("tokens now:", sorted({" ".join(w) for w in corpus}))
```

Watch "low" and "est" become tokens because they appear often. Add more words and see which merges win.

**Quick check:** In byte-pair encoding, which pair gets merged next?

A. The pair that appears side by side most often
B. The rarest pair, to save space
C. A random pair
D. Whichever pair comes first in the alphabet

Answer: A. BPE always glues the most frequent neighbouring pair into a new token, so common chunks of text become single tokens.

### Why models fumble letter counting

Ask a model how many r's are in "strawberry" and it may get it wrong. It never saw the letters: it saw perhaps two or three tokens, each just an ID number.

Spelling, reversing words and counting characters are hard for the same reason. Models that reason step by step, or write the word out letter by letter first, do much better.

> It is a bit like being handed a word as a few Scrabble tiles with whole syllables printed on them, then being asked how many times one letter appears. You'd have to stop and spell it out.

- In English, one token is roughly four characters, or about three quarters of a word.
- AI services charge per token, and the context window is measured in tokens.
- Tokenisers trained mostly on English split other languages into more pieces. The same sentence in Hindi, Burmese or Amharic can take several times as many tokens.
- That means non-English users can pay more, wait longer and fit less into the context window.

**Quick check:** Why might the same message cost more to process in Thai than in English?

A. Thai text is always longer to read
B. The tokeniser, trained mostly on English, splits Thai into more tokens
C. Providers charge extra for non-English languages by policy
D. Models translate everything into English first

Answer: B. Costs follow token counts. A tokeniser that learned mostly from English has fewer ready-made chunks for other scripts, so it needs more tokens for the same meaning.

**Key takeaways**

- Models read tokens, usually subword pieces, not letters or whole words.
- Byte-pair encoding builds the vocabulary by repeatedly merging the most common pair.
- Tokens explain letter-counting slips, pricing, and why some languages cost more.

**Go deeper (free sources)**

- [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/abs/1508.07909) by Sennrich, Haddow and Birch (2016)
- [Byte-Pair Encoding tokenization](https://huggingface.co/learn/nlp-course/chapter6/5) by Hugging Face course
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://arxiv.org/abs/2305.15425) by Petrov et al. (2023)