Intermediate · 13

How text becomes tokens

Language models don't read letters or words: they read tokens. How text gets chopped up, and why it explains some odd model mistakes.

7 min read · Tap underlined words for a definition

A language model only works with numbers. Before it sees your message, a tokeniser chops the text into pieces called and swaps each one for an ID number from a fixed list, the vocabulary. How you chop matters more than you might think.

Three ways to chop

Characters: a tiny vocabulary, but sequences get very long, and single letters carry little meaning.

Whole words: short sequences, but the vocabulary explodes, and any new word, typo or name is unknown.

Subwords: the middle path used by modern models. Common words stay whole; rare words split into familiar pieces, like "un" + "believ" + "able".

Byte-pair encoding

Byte-pair encoding (BPE) builds a subword vocabulary from data. Start with single characters. Find the pair that appears side by side most often and glue it into a new token. Repeat thousands of times.

Frequent chunks like "ing", "the" and "tion" become single tokens. Many modern tokenisers start from raw bytes instead of characters, so any text, even emoji, can be encoded.

Run itA mini BPE merge loop

from collections import Counter
# Each word starts as a list of single characters
words = ["low", "low", "lower", "lowest", "newest", "newest", "widest"]
corpus = [list(w) for w in words]

for merge in range(5):
    pairs = Counter()
    for w in corpus:
        for a, b in zip(w, w[1:]):
            pairs[(a, b)] += 1
    (a, b), count = pairs.most_common(1)[0]
    print(f"merge {merge + 1}: '{a}' + '{b}' -> '{a + b}'  (seen {count} times)")
    new_corpus = []
    for w in corpus:  # glue every occurrence of the pair together
        out, i = [], 0
        while i < len(w):
            if i < len(w) - 1 and (w[i], w[i + 1]) == (a, b):
                out.append(a + b); i += 2
            else:
                out.append(w[i]); i += 1
        new_corpus.append(out)
    corpus = new_corpus
print("tokens now:", sorted({" ".join(w) for w in corpus}))

Edit anything, then run. Python runs in your browser.

Watch "low" and "est" become tokens because they appear often. Add more words and see which merges win.

Quick check

In byte-pair encoding, which pair gets merged next?

Why models fumble letter counting

Ask a model how many r's are in "strawberry" and it may get it wrong. It never saw the letters: it saw perhaps two or three tokens, each just an ID number.

Spelling, reversing words and counting characters are hard for the same reason. Models that reason step by step, or write the word out letter by letter first, do much better.

In English, one token is roughly four characters, or about three quarters of a word.
AI services charge per token, and the is measured in tokens.
Tokenisers trained mostly on English split other languages into more pieces. The same sentence in Hindi, Burmese or Amharic can take several times as many tokens.
That means non-English users can pay more, wait longer and fit less into the context window.

Quick check

Why might the same message cost more to process in Thai than in English?

Before you drift off

Models read tokens, usually subword pieces, not letters or whole words.
Byte-pair encoding builds the vocabulary by repeatedly merging the most common pair.
Tokens explain letter-counting slips, pricing, and why some languages cost more.

Terms in this lesson

Go deeper, free and open

Neural Machine Translation of Rare Words with Subword Units · Sennrich, Haddow and Birch (2016) · The paper that brought BPE to language models
Byte-Pair Encoding tokenization · Hugging Face course · Free
Language Model Tokenizers Introduce Unfairness Between Languages · Petrov et al. (2023)

That's the lesson

How text becomes tokens · 7 min

Finish & continue · Up nextHow recommendations workIntermediate · 7 min

Was this lesson clear?

Spotted a mistake? Suggest an edit

Previous: Finding groups: clustering