AI glossary
What is speculative decoding?
Short answer
A speed-up where a small draft model guesses several tokens ahead and the big model checks them all at once, keeping the ones it agrees with.
For example
A small model drafts the next five words; the big model checks all five in one pass, keeps the first four it agrees with and writes the fifth itself.
Learn it in a free lesson
Related terms
Questions
What is speculative decoding in simple terms?
A speed-up where a small draft model guesses several tokens ahead and the big model checks them all at once, keeping the ones it agrees with.
What is an example of speculative decoding?
A small model drafts the next five words; the big model checks all five in one pass, keeps the first four it agrees with and writes the fifth itself.
Where can I learn about speculative decoding for free?
AI in Bed explains it in the free lesson "How models are served: prefill, decode and the KV cache" (Advanced level, 8 minutes), with no sign-up.