Stop 13 · Fine-Tuning & Adapting LLMsTopics · 5/9

DPO (Overview)

In short

RLHF works, but training a reward model and running PPO is fiddly. DPO uses the same 'this answer is better than that one' data and trains the model directly. Simpler, cheaper and now very common.

Direct Preference Optimization — a simpler alternative to RLHF that directly optimizes on preference pairs without training a separate reward model.

The blob looks through a lens testing machine while an optometrist flips between two lenses.

Think of it like this

The eye test

At the eye clinic, the optometrist flips lenses: 'Better with one, or two?' You just say which is clearer. No scoring out of 10, no separate judge. After a few rounds, your prescription is set. DPO learns from pairs exactly like that: preferred vs not preferred.

Watch it happen

RLHF vs DPO

RLHF (PPO)

Trains a separate reward model
Then runs reinforcement learning
Powerful but tricky to tune

DPO

No reward model needed
A simple loss on preference pairs
Trains like normal fine-tuning

For most teams with preference data, DPO is the practical first choice. RLHF with PPO is used when you need the extra control and have the expertise.

The big ideas

What you need to know

Data: chosen vs rejected

Each row has a prompt, a chosen answer and a rejected answer. The model learns to prefer the chosen one.

Reference model

A frozen copy of the starting model keeps training anchored, playing the same role as the KL penalty in RLHF.

Beta controls strength

The beta setting decides how far the model may move from the reference. Small values stay closer.

Works with LoRA

DPO combines well with LoRA or QLoRA, so it can run on modest GPUs.

See it in code

Walk through the code

json

1{"prompt": "Explain what a credit score is to a college student.",2 "chosen": "A credit score is a number showing how reliably you repay loans. Paying credit card bills on time raises it.",3 "rejected": "Credit scores are complicated financial metrics computed by bureaus using proprietary algorithms."}

1/3The prompt both answers respond to.

One line of a DPO dataset in JSONL. Libraries like Hugging Face TRL's DPOTrainer read this shape.

Save yourself the pain

Common mistakes

Oops

Making 'rejected' answers obviously terrible.

Do this instead

Pairs that are close in quality teach finer preferences. Very easy pairs teach little.

Oops

Running DPO on a base model with no instruction tuning.

Do this instead

Do SFT first so the model already follows instructions, then DPO to refine preferences.

Your mission

Make a preference set

~30 min

Quick check

What does DPO remove compared to classic RLHF?

Go deeper

1Read how DPO reformulates the RLHF objective into a single supervised loss.
2Compare the practical complexity of setting up DPO vs. full RLHF.
3Try a DPO fine-tuning example on a small open-weight model if hardware allows.