Stop 13 · Fine-Tuning & Adapting LLMsTopics · 6/9

Dataset Preparation for Fine-Tuning

In short

Fine-tuning copies whatever is in your data, including typos, wrong answers and bad habits. Most fine-tuning failures are dataset failures. Spend more time here than on hyperparameters.

Cleaning, formatting and structuring examples for fine-tuning — dataset quality matters more than dataset size, and this step is where most fine-tuning projects succeed or fail.

The blob picks small stones out of rice spread on a wide steel plate.

Think of it like this

Cleaning rice before cooking

Before cooking, Amma spreads rice on a steel plate and picks out every tiny stone and husk. Skip it, and the best recipe still gives you a crunchy surprise. Your dataset is the rice. Clean it before you cook a model with it.

Watch it happen

From raw logs to a training set

Collect

Clean

Dedupe

Format

Split

1/5Gather real examples: support chats, expert answers, or carefully written samples.

The big ideas

What you need to know

Quality beats quantity

A few hundred to a few thousand excellent examples often beat many thousands of noisy ones.

Cover real variety

Include short and long inputs, typos, Hinglish, edge cases and examples where the right answer is 'I can't help with that'.

Strip personal data

Fine-tuned models can memorise training text. Remove names, numbers and IDs before training.

Read your data

Open 50 random rows and read them. You'll find problems no script catches.

See it in code

Walk through the code

python

1import json, re2 3PHONE = re.compile(r"\b[6-9]\d{9}\b")        # Indian mobile numbers4seen, rows = set(), []5 6for line in open("raw.jsonl"):7    ex = json.loads(line)8    q, a = ex["question"].strip(), ex["answer"].strip()9    if len(a) < 20 or q.lower() in seen:     # too short or duplicate10        continue11    seen.add(q.lower())12    a = PHONE.sub("[PHONE]", a)13    rows.append({"messages": [{"role": "user", "content": q},14                              {"role": "assistant", "content": a}]})15 16with open("train.jsonl", "w") as f:17    f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)

1/4A simple pattern for 10-digit Indian mobile numbers starting with 6 to 9.

A tiny cleaning pass: drop short or duplicate rows, mask phone numbers, and write chat-format JSONL.

Save yourself the pain

Common mistakes

Oops

Using raw support logs, including agents' wrong or rude replies.

Do this instead

Filter for answers you'd be proud of. The model will copy the bad ones just as faithfully.

Oops

Testing on examples that also appear in training.

Do this instead

Split before training and check for near-duplicates across the split, or scores will look falsely good.

Your mission

Clean a messy set

~35 min

Quick check

Your fine-tuned bot sometimes replies rudely. The base model never did. What's the most likely cause?

Go deeper

1Build a small fine-tuning dataset from scratch for a specific task you care about.
2Learn to deduplicate, filter low-quality examples, and balance the dataset.
3Split into train/validation sets and keep a held-out test set for final evaluation.