AI glossary
What is knowledge distillation?
Short answer
Training a small student model to copy the soft probability answers of a larger teacher model.
For example
Many small, fast models are trained by having them copy the answers of a larger model, keeping much of its skill at a fraction of the cost.
Learn it in a free lesson
Questions
What is knowledge distillation in simple terms?
Training a small student model to copy the soft probability answers of a larger teacher model.
What is an example of knowledge distillation?
Many small, fast models are trained by having them copy the answers of a larger model, keeping much of its skill at a fraction of the cost.
Where can I learn about knowledge distillation for free?
AI in Bed explains it in the free lesson "Distillation and the rise of small models" (Advanced level, 7 minutes), with no sign-up.