Stop 13 · Fine-Tuning & Adapting LLMsTopics · 3/9

Instruction Tuning

In short

A raw pretrained model will happily continue your question with three more questions. Instruction tuning teaches it to actually answer. It's the step that turned text predictors into helpful assistants.

Fine-tuning a pretrained model on (instruction, response) pairs so it learns to follow directions rather than just complete text — the step that turns a base model into an assistant.

The blob tutor holds up picture flash cards for an eager small student who raises a hand to answer.

Think of it like this

The flash card tutor

A bright kid knows lots of words but doesn't know how to answer exam questions. A tutor drills flash cards: question on one side, model answer on the other. After hundreds of cards, the kid answers in the right style every time. That's instruction tuning.

Watch it happen

Where the model learns in one training example

<user>

Translate

'thank you'

to Hindi

dhanyavaad

<end>

1/5One training example, laid out with the model's chat template.

The big ideas

What you need to know

Instruction and response pairs

Data looks like (instruction, optional input, ideal response). Variety across tasks matters as much as quantity.

Use the chat template

Each model family has special tokens for roles. Format data with tokenizer.apply_chat_template, or results suffer.

Train on answers only

Masking the prompt from the loss focuses learning on responses. Many trainers support this directly.

Quality over volume

A few thousand clean, diverse examples can beat a huge noisy set. Bad answers teach bad habits.

See it in code

Walk through the code

python

1from transformers import AutoTokenizer2 3tok = AutoTokenizer.from_pretrained("<open-instruct-model>")4 5example = [6    {"role": "user", "content": "Translate 'thank you' to Hindi."},7    {"role": "assistant", "content": "Dhanyavaad (धन्यवाद)."},8]9text = tok.apply_chat_template(example, tokenize=False)10print(text)   # shows the special role tokens this model expects

1/3Load the tokenizer of the model you'll fine-tune. It contains the chat template.

Let the tokenizer format your data. Every model family uses different special tokens for roles.

Save yourself the pain

Common mistakes

Oops

Writing your own '### User:' format for a model that has its own chat template.

Do this instead

Use the model's template. Mismatched formats confuse the model and waste training.

Oops

Training on 10,000 near-identical examples.

Do this instead

Diversity matters. Vary topics, lengths and phrasing, and remove duplicates.

Your mission

Build 30 instruction pairs

~30 min

Quick check

Why do many instruction-tuning setups compute loss only on the assistant's tokens?

Go deeper

1Study the format of an instruction-tuning dataset (e.g. Alpaca-style).
2Fine-tune a small model on an instruction dataset and compare its behavior before/after.
3Learn how instruction tuning differs from RLHF as a later alignment step.