Advanced · 11

How models are served: prefill, decode and the KV cache

What happens on the computer when you press send. Why replies stream out word by word, why output tokens cost more, and the tricks that make it all faster.

8 min read · Tap underlined words for a definition

Training a model happens once. Answering questions happens millions of times a day. Running a trained model to get answers is called inference, and doing it quickly and cheaply for lots of people at once is called serving. It is its own craft, with its own clever tricks.

Two phases: prefill and decode

Prefill: the model reads your whole in one go. All the prompt tokens can be processed side by side, so the chip stays busy and this part is fast per token.

Decode: the model writes its reply one token at a time. Each new token depends on the one before, so the steps cannot run in parallel. This is why answers stream out word by word.

The KV cache

Inside , every produces a key and a value: small lists of numbers that later tokens look back at. Without care, the model would recompute these for the whole conversation at every step.

The KV cache stores the keys and values once and reuses them. Each decode step then only does fresh work for the one new token. The catch is memory: long conversations make the cache large.

Run itCounting the work a KV cache saves

prompt = 1000   # tokens in the prompt
reply = 500     # tokens the model writes

# Without a cache: each new token re-processes everything so far.
no_cache = sum(prompt + i for i in range(1, reply + 1))

# With a KV cache: process the prompt once, then one new token per step.
with_cache = prompt + reply

print(f"Token passes without cache: {no_cache:,}")
print(f"Token passes with cache:    {with_cache:,}")
print(f"Roughly {no_cache // with_cache}x less work")

Edit anything, then run. Python runs in your browser.

A simplified count of token passes, not exact maths, but it shows why no serious system runs without a cache.

Quick check

Why do output tokens usually cost more than input tokens?

Batching, latency and throughput

Latency is how long one person waits. Throughput is how many tokens the whole system produces per second across everyone.

A single decode step barely uses a modern chip, so servers group many users' requests into one . Throughput rises a lot, but each person may wait slightly longer. Good servers add new requests to the batch as old ones finish, which is called continuous batching.

Speculative decoding

A small, fast draft model guesses the next few tokens. The big model then checks all those guesses in a single pass, which is cheap because checking can run in parallel, like prefill.

Correct guesses are kept, and at the first wrong one the big model supplies its own token. When the draft is often right, replies come out noticeably faster, and the final text matches what the big model would have written anyway.

Quick check

In speculative decoding, what does the big model do with the draft model's guesses?

Before you drift off

Prefill reads the prompt in parallel; decode writes one token at a time.
The KV cache trades memory for a huge saving in repeated work.
Batching boosts throughput, and speculative decoding cuts waiting time.

Terms in this lesson

Go deeper, free and open

Fast Inference from Transformers via Speculative Decoding · Leviathan et al.
Efficient Memory Management for LLM Serving with PagedAttention · Kwon et al. · The vLLM paper
Assisted Generation: a new direction toward low-latency text generation · Hugging Face

That's the lesson

How models are served: prefill, decode and the KV cache · 8 min

Finish & continue · Up nextMixture of experts: big models, small billsAdvanced · 8 min

Was this lesson clear?

Spotted a mistake? Suggest an edit

Previous: Keeping AI on our side: alignment and interpretability