Model Distillation

Summary

Training a smaller 'student' model to mimic a larger 'teacher' model's outputs — a way to get much of a large model's quality at a fraction of the inference cost.

• 1Read a case study of a distilled model (e.g. DistilBERT) vs. its teacher.
• 2Learn the basic distillation training setup: matching soft labels/logits from the teacher.
• 3Understand when distillation is worth the extra training pipeline complexity.