Definition
Direct preference optimization trains a language model from preference triples: a prompt, a chosen response, and a rejected response. Its loss increases the chosen response's log-probability advantage over the rejected response, measured relative to a reference model. The reference term limits how far training moves from the starting policy.
DPO derives a direct classification objective from the same regularized preference model used in a common RLHF formulation. It therefore avoids fitting a separate scalar reward model and avoids an online reinforcement-learning step such as PPO. The method still requires preference data, a reference policy, optimization choices, and evaluation outside the training pairs.
Name and attribution
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning, and Chelsea Finn introduced the name "Direct Preference Optimization" in their 2023 paper. The name belongs to that derived objective, not to every method that fine-tunes on chosen and rejected responses.
Distinguish it from nearby terms
DPO changes model weights. Runtime reranking compares candidates without necessarily changing the generator. Reward-model-based RLHF first learns an explicit scoring model and then optimizes against it. DPO removes those two components from the standard pipeline, but preference quality and distribution shift can still limit the result.
Check your understanding
A preference dataset consistently favors short answers, even when a longer answer is required for correctness. What behavior will DPO likely reinforce, and which evaluation would reveal the problem?