What is DPO in AI? Everything you need to know

What is DPO in AI?

Direct preference optimization (DPO) is an AI training technique that eliminates two stages typically required by reinforcement learning from human feedback (RLHF): training a reward model and running policy-gradient optimization. There's just a prompt, a preferred response, a rejected response, and a loss function that pushes the model toward the response humans preferred.

DPO was first introduced by a team from Stanford in a paper titled "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (Rafailov et al., 2023) and presented at the 2023 Conference on Neural Information Processing Systems.

How does DPO work?

DPO starts with a language model that has already been supervised fine-tuned (SFT). Instead of training a separate reward model and using reinforcement learning to improve the model's outputs, DPO learns directly from pairs of human-preferred and rejected responses.

During training, DPO uses a loss function that makes the preferred response more likely than the rejected one, while comparing both against a fixed reference model. Because it optimizes directly from human preference data, DPO skips the reward model and reinforcement learning loop used in RLHF.

By comparison, a standard RLHF pipeline trains a separate reward model on human preference data before using reinforcement learning, typically with proximal policy optimization (PPO), to update the policy.

The training process can be broken down into six stages.

Stage 1: Build a preference dataset

DPO trains on triplets:

• A prompt
• A preferred response
• A rejected response

For example:

Prompt: How do I reset my password?

Preferred response: "Click 'Forgot Password' on the login page and follow the instructions sent to your email." Rejected response: "Passwords are important for keeping your account secure."

Unlike RLHF, DPO doesn't need numerical reward scores or absolute quality ratings. It only needs a relative preference such as, this response is better than that one for a given prompt.

Stage 2: Calculate response probabilities

For each preference pair, the model calculates how likely it is to generate both the preferred and rejected responses.

If the model is currently more likely to generate the rejected response, it has identified a mistake. If it already favors the preferred response, only a small correction is needed.

Stage 3: Compare against a reference model

DPO doesn't evaluate these probabilities in isolation. It also compares them against a frozen reference model, which is typically the supervised fine-tuned (SFT) model before DPO training begins.

The reference model acts as a baseline, ensuring the model doesn't drift too far from the capabilities it learned during supervised fine-tuning. A hyperparameter, β (beta), controls this tradeoff. A lower beta allows the model to move further away from the reference model to better match human preferences, while a higher beta keeps it closer, prioritizing stability over larger behavioral changes.

Stage 4: Compute the loss function

DPO uses a loss function to determine how much the model should learn from each preference pair.

The loss is larger when the model assigns a higher probability to the rejected response than the preferred one, resulting in a larger update. If the model already favors the preferred response, the loss is smaller because less correction is needed.

Rather than optimizing a reward score, the loss function directly encourages the model to increase the probability of preferred responses while decreasing the probability of rejected ones.

Stage 5: Adjust the model's weights

Using the loss, DPO updates the model's weights—the billions of parameters that determine how it generates text.

Each update increases the likelihood of producing responses humans preferred while reducing the likelihood of producing rejected responses. Repeating this process across millions of preference pairs gradually teaches the model the behaviors humans consistently value, such as being more helpful, accurate, and safe.

Stage 6: Learn an implicit reward function

In RLHF, a separate reward model evaluates every response before reinforcement learning updates the language model. DPO removes that intermediate step.

Instead of explicitly learning a reward model, DPO learns an implicit reward function through the language model itself. Because preferred responses become increasingly more probable than rejected ones, the model effectively internalizes the same preference signal that an external reward model would have provided.

This is DPO's key innovation. By directly optimizing from human preference data, it achieves the same alignment objective as RLHF while eliminating the need for a separate reward model and reinforcement learning pipeline.

DPO variants: IPO, KTO, and online methods

Vanilla DPO has several known failure modes, and a handful of variants have been developed to patch specific ones. None of them make standard DPO obsolete. Many teams still start with vanilla DPO and switch to a variant only when a specific problem emerges in their runs.

Identity preference optimization

Identity preference optimization (IPO), from Azar et al. (2023), targets a specific DPO failure mode: When preference labels are close to deterministic (annotators agree almost every time), the KL constraint effectively weakens. The model can overfit, pushing the implicit reward gap between preferred and rejected responses beyond what the data actually supports.

IPO adds a regularization term that keeps this gap bounded. Use it when your dataset has very strong, very clean preference signals and vanilla DPO shows signs of overfitting.

Kahneman-Tversky optimization

Kahneman-Tversky optimization (KTO), from Ethayarajh et al. (2024), drops the paired-comparison requirement entirely. Instead of "response A beats response B," it trains on standalone binary labels indicating whether an individual response is good or bad. That's meaningfully cheaper to collect, since annotators judge one output at a time instead of ranking two.

The trade-off is information density. A direct comparison tells you more about relative quality than two independent good-or-bad judgments. KTO is most useful when collecting paired preference data is operationally difficult or too slow to run at the monthly volume you need.

Online and iterative DPO

Standard DPO is offline: It trains once on a fixed, pre-collected dataset of preference pairs. Online DPO instead generates new preference pairs during training, using the model's current outputs, which keeps the training distribution closer to what the model actually produces as it improves.

Iterative DPO alternates between collecting fresh preferences from the current policy and running DPO updates on them. This rhythm starts to resemble how PPO continuously adapts to the policy it's training.

Both approaches cost more than offline DPO in data collection overhead, but they reduce the distribution shift that is DPO's main structural weakness.

DPO vs. RLHF and PPO: when to use each

Neither method has replaced the other.

DPO is simpler, cheaper, and more stable to train for a lot of alignment work. But it's fundamentally an offline method, learning from a fixed snapshot of human preferences rather than adapting as the policy changes.

PPO is more expensive to run and harder to tune. It adapts to the current policy during training and can encode reward structures that are too complex for simple preference pairs to capture, such as assigning partial credit on a multi-step proof.

Choose DPO when you have:

• A clear, stable preference signal
• A high-quality dataset of preferred and rejected responses
• Limited compute or engineering resources for a full RLHF pipeline

OpenAI now supports DPO as a fine-tuning method, reflecting how common it’s become for standard alignment workflows. For many instruction-following and conversational models, DPO is the practical place to start before considering the additional complexity of RLHF.

When RLHF and PPO are the better fit

RLHF and PPO justify their added sophistication when DPO's offline approach becomes a limitation.

Choose RLHF or PPO when you need:

• Online adaptation as the model evolves during training
• Complex reward structures, such as mathematical reasoning or code generation
• Better handling of distribution shift between training and deployment

The choice ultimately depends on whether your task can be captured with simple preference pairs or requires richer reward signals and continuous adaptation.

What are the DPO limitations and trade-offs?

The DPO algorithm simplifies alignment training, but it also comes with important trade-offs. Most stem from the fact that it's an offline optimization method that learns directly from preference data instead of relying on a separate reward model.

Key limitations include:

• Offline training can become outdated: DPO learns from a fixed set of human preference examples. If the types of prompts or responses change after training, it can't adapt unless it's trained again. RLHF methods that collect feedback during training can adjust more easily to changing conditions.
• High-quality preference data is essential: DPO learns directly from human preference pairs. If those labels are inconsistent or inaccurate, the model learns the wrong behavior because there isn't a separate reward model to compensate for poor-quality data.
• DPO only learns relative preferences: DPO teaches the model which of two responses is better for the same prompt. It doesn't measure whether either response is objectively good, only which one humans preferred.
• Some tasks still benefit from RLHF: Research has shown that PPO-based RLHF can outperform DPO on tasks such as mathematical reasoning and code generation, where learning from ongoing feedback and more detailed reward signals can produce better results.

Removing the reward model doesn't eliminate the need for human judgment. It shifts even more responsibility onto the quality of the preference data itself.

Teams using DPO benefit from reviewers with genuine domain expertise who can consistently distinguish the best response from one that merely sounds convincing. That makes expert preference data a critical part of a successful DPO pipeline, often the area where organizations need the most support.

How direct preference optimization changes your alignment pipeline

DPO lowers the engineering barrier to alignment training, but what it doesn't lower is the judgment requirement.

The bottleneck it exposes, whether a given response is actually better than its alternative, was always the hard part of RLHF, too. DPO simply removes the reward model that used to sit between that judgment and the policy update. That change means the judgment itself now has to be right the first time.

In practice, this shows up as a sourcing problem before it shows up as a modeling problem. nThe preference pairs that a team collects matter as much as the loss function they choose, since there's no reward model left to smooth over a bad label.