AI glossary
What is mixture of experts?
Short answer
A model design where a router sends each token to only a few of many sub-networks (experts), so it can store a lot but compute little per token.
For example
A mixture-of-experts model might hold hundreds of billions of weights but use only a small fraction of them for each token, so it runs much faster than its size suggests.
Learn it in a free lesson
Related terms
Questions
What is mixture of experts in simple terms?
A model design where a router sends each token to only a few of many sub-networks (experts), so it can store a lot but compute little per token.
What is an example of mixture of experts?
A mixture-of-experts model might hold hundreds of billions of weights but use only a small fraction of them for each token, so it runs much faster than its size suggests.
Where can I learn about mixture of experts for free?
AI in Bed explains it in the free lesson "Mixture of experts: big models, small bills" (Advanced level, 8 minutes), with no sign-up.