Stop 11 · Large Language ModelsTopics · 3/9

Pretraining vs Fine-tuning

In short

Training an LLM from scratch costs huge amounts of compute. Adapting one that already exists can take an afternoon. Knowing which stage does what tells you what you can realistically build.

Pretraining teaches a model general language understanding from massive raw text; fine-tuning adapts a pretrained model to a specific task or behavior with much smaller, targeted data.

The blob in a doctor's coat studies one small book, sitting among tall piles of finished books.

Think of it like this

MBBS first, then MD

A doctor spends years on MBBS learning the whole body. Only then does she specialise in, say, cardiology with a much smaller, focused course. Pretraining is the MBBS on internet-scale text. Fine-tuning is the short specialisation for your task.

Watch it happen

How a chat model is usually made

Raw text

Pretrain

SFT

Preference

Assistant

1/5Trillions of tokens of web pages, books and code are collected and cleaned.

The big ideas

What you need to know

Pretraining: broad and costly

Huge unlabelled data, huge compute, done by a few labs. It's where general knowledge and language skill come from.

Fine-tuning: narrow and cheap

Hundreds to thousands of examples teach format, tone or a skill. Often done with LoRA on a single GPU.

Base vs instruct models

A base model just continues text. An instruct or chat model has been fine-tuned to answer questions.

Try prompting first

Many tasks need only a good prompt or RAG. Fine-tune when you need consistent style or format at scale.

See it in code

Walk through the code

json

1{"messages": [2  {"role": "system", "content": "You write short, polite replies for a food delivery app."},3  {"role": "user", "content": "My order is 40 minutes late."},4  {"role": "assistant", "content": "Sorry about the wait! Your rider is 5 minutes away. We've added a Rs 50 coupon to your account."}5]}

1/3The system message sets the role the model should play.

One line of a typical chat fine-tuning dataset in JSONL. You'd have hundreds of these, one per line.

Save yourself the pain

Common mistakes

Oops

Fine-tuning to teach the model new facts that change often.

Do this instead

Use RAG for changing knowledge. Fine-tuning is better at teaching behaviour, tone and format.

Oops

Fine-tuning a base model and expecting chat behaviour from a few examples.

Do this instead

Start from an instruct model for chat tasks. It already follows instructions, so your data only adds the specifics.

Your mission

Base vs instruct

~20 min

Quick check

A startup wants its support bot to always reply in a short, friendly, fixed format. What fits best?

Go deeper

1Learn roughly how much data/compute separates pretraining from fine-tuning (orders of magnitude).
2Understand why almost no individual engineer pretrains an LLM from scratch, but fine-tuning is accessible.
3Read about instruction tuning as the step that turns a raw pretrained model into an assistant.