Pretraining vs Fine-tuning

Summary

Pretraining teaches a model general language understanding from massive raw text; fine-tuning adapts a pretrained model to a specific task or behavior with much smaller, targeted data.

• 1Learn roughly how much data/compute separates pretraining from fine-tuning (orders of magnitude).
• 2Understand why almost no individual engineer pretrains an LLM from scratch, but fine-tuning is accessible.
• 3Read about instruction tuning as the step that turns a raw pretrained model into an assistant.