Stop 19 · System Design for AI ApplicationsTopics · 9/9

Choosing Between Fine-Tuning and RAG

In short

'Should we fine-tune or use RAG?' comes up in almost every AI project and interview. The answer depends on one thing: are you missing knowledge, or missing behaviour?

A recurring real-world design decision — fine-tuning bakes in behavior/style, RAG grounds answers in up-to-date external knowledge, and many systems genuinely need both.

The blob sits at an exam desk with an open textbook while a coach in a tracksuit with a whistle stands beside it.

Think of it like this

Open-book exam vs months of coaching

An open-book exam lets you look up facts you never memorised, and the book can be updated anytime. Coaching for months changes how you think and write answers, but you can't easily unlearn it. RAG is the open book. Fine-tuning is the coaching.

Watch it happen

Two tools for two problems

RAG

Adds fresh, private knowledge
Update docs, no retraining
Answers can cite sources

Fine-tuning

Teaches style, format, behaviour
Needs good labelled examples
Retrain to change anything

Missing facts? Use RAG. Model knows enough but behaves wrong? Fine-tune. Many real systems use both, plus good prompting first.

The big ideas

What you need to know

Try prompting first

A clear prompt with a few examples often fixes behaviour without any training. It's the cheapest experiment.

Fresh data means RAG

Prices, policies and news change. Fine-tuning freezes knowledge at training time; RAG reads the latest version.

Format and tone mean fine-tune

Always output a strict schema, match a brand voice, or handle a niche task style: fine-tuning fits well.

Cost and latency

A fine-tuned small model can replace a big model with long prompts, cutting cost per call.

How it works

A quick decision path

1
Write 20 test cases
Real questions with what a good answer looks like. You'll use these to judge every option.
2
Prompt well
Try a strong prompt with examples. Measure. Stop if it's good enough.
3
Missing facts?
If answers fail because the model doesn't know your data, add RAG and measure again.
4
Wrong behaviour?
If it knows the facts but format or style is off, collect examples and fine-tune.
5
Combine if needed
A fine-tuned model can still use RAG. Keep the test set to check each step helped.

Save yourself the pain

Common mistakes

Oops

Fine-tuning on company docs so the model 'learns' the latest policies.

Do this instead

Fine-tuning is poor at storing changing facts. Put policies in a RAG index and update them there.

Oops

Building RAG to fix a model that ignores your output format.

Do this instead

Retrieval adds knowledge, not behaviour. Fix format with prompting, structured outputs or fine-tuning.

Your mission

Decide for three cases

~20 min

Quick check

A bank's chatbot must answer using interest rates that change every week. Best approach?

Go deeper

1Build a decision framework: use RAG for facts/freshness, fine-tuning for style/format/behavior.
2Practice justifying this choice for 2-3 different hypothetical products.
3Learn how combining both (fine-tuned model + RAG) is common in production systems.