# How models are served: prefill, decode and the KV cache · AI in Bed

Advanced level · 8 min · https://www.aiinbed.com/learn/advanced/inference-and-serving

*What happens on the computer when you press send. Why replies stream out word by word, why output tokens cost more, and the tricks that make it all faster.*

Training a model happens once. Answering questions happens millions of times a day. Running a trained model to get answers is called **inference**, and doing it quickly and cheaply for lots of people at once is called **serving**. It is its own craft, with its own clever tricks.

### Two phases: prefill and decode

**Prefill**: the model reads your whole prompt in one go. All the prompt tokens can be processed side by side, so the chip stays busy and this part is fast per token.

**Decode**: the model writes its reply one token at a time. Each new token depends on the one before, so the steps cannot run in parallel. This is why answers stream out word by word.

### The KV cache

Inside attention, every token produces a key and a value: small lists of numbers that later tokens look back at. Without care, the model would recompute these for the whole conversation at every step.

The **KV cache** stores the keys and values once and reuses them. Each decode step then only does fresh work for the one new token. The catch is memory: long conversations make the cache large.

**Counting the work a KV cache saves**

```python
prompt = 1000   # tokens in the prompt
reply = 500     # tokens the model writes

# Without a cache: each new token re-processes everything so far.
no_cache = sum(prompt + i for i in range(1, reply + 1))

# With a KV cache: process the prompt once, then one new token per step.
with_cache = prompt + reply

print(f"Token passes without cache: {no_cache:,}")
print(f"Token passes with cache:    {with_cache:,}")
print(f"Roughly {no_cache // with_cache}x less work")
```

A simplified count of token passes, not exact maths, but it shows why no serious system runs without a cache.

**Quick check:** Why do output tokens usually cost more than input tokens?

A. Output tokens are longer words
B. Output is written one token at a time, so the chip does less useful work per step
C. Input tokens are stored for free in the cloud
D. Providers charge extra for the model's opinions

Answer: B. Prefill handles the prompt in parallel and keeps the hardware busy. Decode is sequential, so each output token ties up the hardware for longer.

### Batching, latency and throughput

**Latency** is how long one person waits. **Throughput** is how many tokens the whole system produces per second across everyone.

A single decode step barely uses a modern chip, so servers group many users' requests into one batch. Throughput rises a lot, but each person may wait slightly longer. Good servers add new requests to the batch as old ones finish, which is called continuous batching.

> Think of a lift. Taking one person at a time is quick for them but wastes the lift. Waiting a moment to fill it moves far more people per hour.

### Speculative decoding

A small, fast **draft model** guesses the next few tokens. The big model then checks all those guesses in a single pass, which is cheap because checking can run in parallel, like prefill.

Correct guesses are kept, and at the first wrong one the big model supplies its own token. When the draft is often right, replies come out noticeably faster, and the final text matches what the big model would have written anyway.

**Quick check:** In speculative decoding, what does the big model do with the draft model's guesses?

A. Ignores them unless the user asks
B. Checks several of them in one parallel pass and keeps the ones it agrees with
C. Retrains itself on them
D. Sends them straight to the user without checking

Answer: B. Verifying a run of tokens at once is cheap. Accepted guesses save whole decode steps, and the output quality is unchanged.

**Key takeaways**

- Prefill reads the prompt in parallel; decode writes one token at a time.
- The KV cache trades memory for a huge saving in repeated work.
- Batching boosts throughput, and speculative decoding cuts waiting time.

**Go deeper (free sources)**

- [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192) by Leviathan et al.
- [Efficient Memory Management for LLM Serving with PagedAttention](https://arxiv.org/abs/2309.06180) by Kwon et al.
- [Assisted Generation: a new direction toward low-latency text generation](https://huggingface.co/blog/assisted-generation) by Hugging Face