DPO vs. RLHF: Comparison and when to use each

What is DPO (Direct Preference Optimization)?

Direct preference optimization (DPO) is a method for aligning a language model to human preferences without training a separate reward model or running reinforcement learning (RL). It takes pairs of responses, one preferred and one rejected, and adjusts the model directly so the preferred response becomes more likely.

What is RLHF (Reinforcement Learning From Human Feedback)?

Reinforcement learning from human feedback (RLHF) aligns a language model by first training a reward model on human preference comparisons, then using RL to push the model toward responses that score well according to that reward.

RLHF became the default approach for instruction tuning after OpenAI used it for InstructGPT and later ChatGPT, while Anthropic applied it to turn base models into helpful and harmless assistants. This method enables a model to follow instructions and refuse the wrong requests instead of just predicting the next token.

What's the difference between DPO and RLHF?

The main difference between DPO vs. RLHF is the presence of a reward model. RLHF trains one as a separate stage and then optimizes the policy against it with RL. DPO consolidates both steps into a single loss based on preference pairs. Everything else flows from that one structural decision.

The following table compares RLHF and DPO across several key axes:

Cost

RLHF keeps three models active during training: the policy you're tuning, a frozen reference model, and a separate reward model.

DPO drops the reward model and the sampling loop, which means less compute and far less engineering scaffolding. It also means that DPO is cheaper to run, with the savings being structural rather than incremental.

For smaller teams without dedicated RL infrastructure, this cost gap often influences project decisions. It's worth noting that RLHF itself is relatively inexpensive compared to pretraining. OpenAI reported that its InstructGPT fine-tuning required less than 2% of the pretraining compute used for GPT-3. The real cost is often in the human expertise needed, not the technology.

Stability

DPO is typically easier to stabilize because it behaves much like supervised fine-tuning (SFT), while RLHF inherits the challenges of RL. Proximal policy optimization (PPO), the algorithm most RLHF setups use, is sensitive to its hyperparameters, and reward models can sometimes be exploited by the policy in ways that are hard to detect.

Hugging Face's Transformer Reinforcement Learning library, which now ships a dedicated DPO trainer, openly describes RLHF as a complex and often unstable procedure. While DPO removes a whole category of failure, it introduces its own challenges.

Data

Both methods rely on preference data, but they use it differently. RLHF can sample new responses from the policy during training and have them scored by the reward model, so it keeps learning from the model's current distribution.

In contrast, DPO trains on a fixed offline set of chosen and rejected pairs. This approach is simpler, but it can lead to drift if the model moves away from the data it was originally tuned on.

Neither method generates the human signal on its own. Someone has to decide which response is better, and the quality of that judgment sets the ceiling for both.

Performance

Performance is the axis where the straightforward narrative falls apart and where claims of "DPO wins" often overreach.

The original DPO paper reported that DPO matched or exceeded PPO-based RLHF on tasks such as sentiment control, summarization, and single-turn dialogue, while also being simpler to train. Those results, however, came from specific tasks using specific data.

Independent comparisons since then have found that success can vary depending on task type, data quality, and model scale. For most open-model workflows, DPO is a strong default, while RLHF offers a stronger performance ceiling if you invest in it.

If you want to understand how those success rates are actually measured, it's worth taking a closer look at how you evaluate alignment.

How does RLHF and DPO work?

Both pipelines begin with SFT on demonstration data and end with a model that reflects human preferences. The difference lies in what happens in between.

The RLHF approach: using reward model and PPO

RLHF operates in 3 stages:

1. Supervised fine-tuning (SFT): The base language model first learns from high-quality demonstrations so it can generate reasonable responses.
2. Reward model training: Humans compare pairs of model responses and choose the better one. The reward model learns those preferences so it can score future responses automatically.
3. Reinforcement learning (usually PPO): The language model generates new responses, the reward model evaluates them, and PPO adjusts future behavior based on those scores. A Kullback–Leibler penalty keeps the policy close to the original SFT model.

This approach is useful when you need to generate fresh responses during training, reuse the reward model across experiments, or optimize for more complex objectives than simple preference pairs.

The trade-off is added complexity: RLHF requires multiple models and a set of hyperparameters that can be difficult to tune.

The DPO approach: optimizing preferences directly

DPO trains directly on the same chosen and rejected response pairs without first building a separate reward model.

During training, it compares how the policy and a frozen reference model score each response, then adjusts the policy so the preferred response becomes more likely and the rejected response becomes less likely.

DPO's key insight is that it can learn directly from human preference data without first training a separate reward model. Using the Bradley-Terry model of preferences, it combines reward modeling and policy optimization into a single training objective.

DPO still depends on high-quality human preference data and a frozen reference model during training. What it removes is the separate reward model and the RL loop, resulting in a simpler training pipeline.

DPO is therefore a good choice when you want a simpler stack, have offline preference data already, or lack the RL expertise to manage PPO. Plus, it slots cleanly into a standard fine-tuning workflow.

For a closer look, explore how DPO works.

When to use DPO vs. RLHF

The choice between RLHF vs. DPO should be based on your constraints rather than current trends. Match the optimizer to your budget, data, model scale, risk tolerance, and the RL skill on your team. Sometimes the best option is a combination of both.

When to use DPO

Reach for DPO when you have a fixed set of high-quality preference pairs, a modest compute budget, and no appetite for maintaining RL infrastructure. It's the right call for most teams shipping an open model where simplicity and reproducibility matter more than maximizing reward performance.

The trade-off is in DPO's offline nature. Because it trains on a static dataset, it's susceptible to distribution shift as the model changes over time. In this case, the preference data may describe a model that no longer exists, and DPO can overfit to those frozen comparisons.

If your data is narrow or outdated, DPO will faithfully learn those limitations.

When RLHF is the better choice

RLHF justifies its complexity when you need to keep generating new samples during training, want a reward model you can reuse across experiments, or need richer reward shaping than a single preference can express.

Frontier labs rely on RLHF for exactly these reasons. Notably, InstructGPT, which was trained with RLHF, was preferred by human raters over a larger GPT-3 model with 100x more parameters. This is the kind of gain that justifies the overhead. RLHF also handles safety work well, where iterative preference modeling matters and you want to refine the reward signal over time rather than freeze it.

Whichever method you pick, the ceiling is set by the quality of the preference data, not the optimizer. Mercor connects domain experts who produce the chosen and rejected comparisons both RLHF and DPO consume. If you're a domain expert, you can explore opportunities directly with Mercor.

How does DPO and RLHF compare to newer LLM alignment methods?

DPO and RLHF remain the 2 anchor points in a crowded landscape of post-training methods, and most newer approaches are variations on the same theme rather than outright replacements. These methods can be broadly grouped into a few families:

• Reward-free and ranking variants: These include Kahneman-Tversky optimization, which learns from simple thumbs-up or thumbs-down signals instead of preference pairs, and odds ratio preference optimization, which incorporates preference tuning into the SFT stage, eliminating the reference model.
• Efficiency-focused DPO descendants: These include simple preference optimization, which removes the reference model, and identity preference optimization, which addresses overfitting in the DPO objective.
• Group-relative RL methods: Group relative policy optimization, popularized by DeepSeek, estimates advantages across groups of sampled responses instead of training a separate value network and has driven much of the recent progress in reasoning work.

None of these approaches is an outright winner. Each changes the training approach but still leaves you with the same upstream decision: what kind of feedback signal fits your use case and who produces it.

Choosing an alignment method is only part of the equation. High-quality feedback data and strong performance on real-world tasks ultimately determine how successful the model will be.

How Mercor supports AI training

Whether you use DPO, RLHF, or another alignment method, success ultimately depends on the quality of your preference data. Better human feedback leads to better model behavior.

That's where Mercor helps. We connect AI teams with vetted domain experts who generate high-quality preference data and evaluations across specialized fields, helping improve alignment datasets.