Stop 13 · Fine-Tuning & Adapting LLMsTopics · 8/9

Model Distillation

In short

Big models are smart but slow and costly. Distillation trains a small 'student' model to imitate a big 'teacher', so you keep much of the skill at a fraction of the cost.

Training a smaller 'student' model to mimic a larger 'teacher' model's outputs — a way to get much of a large model's quality at a fraction of the inference cost.

The blob copies an elderly head chef's stirring movement on its own small pot in a kitchen.

Think of it like this

Learning from the head chef

A new cook doesn't read every cookbook in the world. He stands next to the head chef and copies how he tastes, stirs and adjusts. He'll never know everything the chef knows, but for the restaurant's menu he becomes nearly as good, and much cheaper.

Watch it happen

One round of distillation

Prompts

Teacher

Student

'Classify: late order'

1/5A batch of task inputs goes to the large teacher model.

The big ideas

What you need to know

Teacher and student

The teacher is large and accurate. The student is small and fast. The student learns from the teacher's outputs.

Soft labels carry hints

'85 percent ORDER, 10 percent REFUND' tells the student more than just 'ORDER'. A temperature setting softens these further.

LLM-style distillation

For LLMs, a common shortcut is generating teacher answers and fine-tuning the student on them as text.

Check the licence

Some providers' terms restrict using their outputs to train competing models. Read the terms first.

See it in code

Walk through the code

python

1import torch.nn.functional as F2 3def distill_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.5):4    soft = F.kl_div(5        F.log_softmax(student_logits / T, dim=-1),6        F.softmax(teacher_logits / T, dim=-1),7        reduction="batchmean",8    ) * (T * T)9    hard = F.cross_entropy(student_logits, labels)10    return alpha * soft + (1 - alpha) * hard

1/3T softens both distributions. alpha balances teacher-matching and true-label learning.

The classic distillation loss: match the teacher's softened probabilities, plus normal loss on true labels.

Save yourself the pain

Common mistakes

Oops

Expecting a tiny student to match the teacher on every task.

Do this instead

Distil for a specific task. Narrow students can get close; general ones lose more.

Oops

Generating teacher data with an API without reading its terms.

Do this instead

Check usage policies before training on any provider's outputs.

Your mission

Distil a classifier

~45 min

Quick check

Why do students often learn better from the teacher's soft probabilities than from hard labels alone?

Go deeper

1Read a case study of a distilled model (e.g. DistilBERT) vs. its teacher.
2Learn the basic distillation training setup: matching soft labels/logits from the teacher.
3Understand when distillation is worth the extra training pipeline complexity.