DPO (Overview)

Summary

Direct Preference Optimization — a simpler alternative to RLHF that directly optimizes on preference pairs without training a separate reward model.

• 1Read how DPO reformulates the RLHF objective into a single supervised loss.
• 2Compare the practical complexity of setting up DPO vs. full RLHF.
• 3Try a DPO fine-tuning example on a small open-weight model if hardware allows.