SFT vs. RLHF: Compare LLM training approaches

What is SFT?

Supervised fine-tuning (SFT) trains a pretrained language model on labeled input-output pairs, teaching it to reproduce demonstrated behavior for specific tasks, formats, or domains.

What is RLHF?

RLHF (reinforcement learning from human feedback) is a multi-stage training process that fine-tunes a language model using human preference judgments, typically through a reward model and policy optimization, to align outputs with what people actually prefer.

What is the difference between SFT and RLHF?

The mechanical differences between supervised fine-tuning vs reinforcement learning cluster around 4 dimensions, and each one has practical consequences for how you build.

1. Data format

SFT uses labeled pairs: a prompt and a target completion. You show the model what a good answer looks like, and it learns to reproduce that pattern. RLHF uses preference data instead, asking humans to rank which of two outputs is better. Rather than defining a single correct answer, those rankings capture subtler qualities like tone, helpfulness, and how well the model handles ambiguity.

2. Loss function

SFT optimizes cross-entropy loss on token prediction, teaching the model to reproduce patterns from the training examples. RLHF's optimization stage instead maximizes a reward signal based on human preferences. Rather than matching a specific answer, the model learns to produce responses people are more likely to prefer.

3. Training stability

SFT is predictable and relatively easy to debug using standard ML tools. RLHF is much less stable because the optimization stage can introduce problems like reward hacking, hyperparameter sensitivity, and training collapse. Classic PPO-based pipelines are especially prone to these challenges, making them harder to tune in production.

4. Optimization Targets

SFT optimizes format adherence and task completion, teaching the model to reproduce the behavior shown in its training examples. RLHF instead optimizes for human preferences like helpfulness, safety, and tone, qualities that labeled examples alone often can't fully capture.

That extra alignment comes with a tradeoff. Research shows RLHF improves out-of-distribution generalization compared to SFT, but it also reduces output diversity (Kirk et al., 2023). For customer-facing assistants, that tradeoff is often worthwhile. Creative and research-oriented applications, however, may benefit from a wider range of responses.

When to use SFT, RLHF, or both

The decision isn't binary. Most production teams doing serious alignment work use both. The question is whether the RLHF overhead, preference data collection, reward model training, RL instability, is justified for your specific use case.

When to use SFT

SFT is often the best choice when your task is narrow, deterministic, and easy to verify. If you can define clear success criteria and have high-quality labeled examples, SFT alone can deliver excellent results. Common use cases include:

• Code generation with unit tests
• Structured data extraction
• Translation
• Classification
• Medical record summarization into a fixed template

Research supports this approach. Studies have shown that carefully curated SFT data can match RLHF performance on narrow, well-defined tasks ((arXiv:2501.07886). The LIMA paper found that fine-tuning a 65B model on just 1,000 high-quality prompt-response pairs produced strong instruction-following without RLHF (Zhou et al., 2023).

Google's FLAN work similarly demonstrated that instruction tuning alone can substantially improve zero-shot generalization (Wei et al., 2022).

SFT-only is not a compromise. For many well-defined applications, it's the most practical and effective architectural choice.

When to use RLHF

RLHF is worth the extra investment when your model needs to handle open-ended tasks where human judgment matters. It shines when labeled examples alone can't fully capture what makes one response better than another. Common use cases include:

• Chat assistants where tone and helpfulness matter as much as accuracy
• Safety-critical applications that must handle risky requests appropriately
• Tasks where there isn't a single correct answer and nuance is important

Anthropic used RLHF to optimize for qualities like helpfulness and harmlessness that labeled demonstrations alone couldn't fully specify (Bai et al., 2022).

The biggest cost isn't compute. It's collecting high-quality preference data from people who can make consistent, informed judgments. Platforms like Mercor provide domain experts for this work, and the difference between expert feedback and crowd-sourced rankings shows up directly in model behavior.

When using both is the better approach

Most production alignment pipelines start with SFT and then add preference optimization. SFT teaches the model format, task structure, and instruction following, while RLHF, DPO, or another preference method refines outputs based on what humans actually prefer.

Leading AI labs follow this same pattern. OpenAI's InstructGPT used SFT, followed by reward model training and PPO optimization (Ouyang et al., 2022). Meta's Llama 2-Chat combined iterative SFT with RLHF to improve helpfulness and safety (Touvron et al., 2023).

For many teams, SFT delivers most of the value. Preference optimization helps close the remaining gap on subjective qualities like safety, helpfulness, and instruction nuance, making it worthwhile when those improvements justify the added cost and complexity.

Are SFT and RLHF alternatives?

No. The biggest misconception in the SFT vs. RLHF discussion is that you're choosing one or the other. You're not. The standard RLHF pipeline includes three stages:

• SFT to teach instruction following and task format
• Reward model training on human preference rankings
• Policy optimization using PPO or another preference optimization method

Both InstructGPT and Llama 2-Chat follow this basic sequence. So does Anthropic's work on reinforcement learning from human feedback. The pattern is now well established across major labs building RLHF-trained models.

The reward model is the piece most people overlook. It turns human preferences into a training signal, so inconsistent or low-quality preference data leads to a weaker reward model and weaker downstream behavior.

You aren't choosing between SFT and RLHF. You're deciding whether to stop after SFT or continue through the full alignment pipeline.

How DPO, GRPO, and RFT are changing the picture

The RL stage of the classic SFT-to-RLHF pipeline is being simplified, not abandoned.

The direction is clear: keep the SFT foundation, keep the preference optimization concept, but reduce the engineering complexity and instability that PPO introduced. The paradigm remains SFT plus preference optimization. What's changing is how that second stage works.

DPO (Direct Preference Optimization)

DPO achieves similar alignment results to RLHF without training a separate reward model. It optimizes preference pairs directly against a reference policy, which means fewer moving parts, lower compute, and less surface area for reward hacking (Rafailov et al., 2023).

For most alignment tasks in DPO fine-tuning, this is now the dominant alternative to PPO. The tradeoff is that DPO can be less expressive than a full RL loop on complex, multi-turn preference signals, but for the majority of use cases, the simplicity wins.

GRPO (Group Relative Policy Optimization)

GRPO keeps the RL loop but reduces compute by estimating reward baselines from group scores rather than training a separate critic model. DeepSeek used GRPO in their math-focused models because PPO's compute cost was a genuine barrier to iteration speed.

GRPO addresses that without giving up online RL entirely, which matters for tasks where the reward signal benefits from exploration rather than static preference pairs.

RFT and verifiable-reward methods

RFT (rejection fine-tuning) and related methods bypass human preference collection entirely. They use verifiable rewards: correct/incorrect signals from math proofs, code execution results, or formal verification.

An RFT LLM pipeline generates multiple candidate outputs, keeps the ones that pass a ground-truth check, and fine-tunes on those. The limitation is obvious, this only works for SFT vs RFT comparisons where ground truth is checkable. You can verify whether code compiles. You can't verify whether a therapy chatbot's response was empathetic.

SFT remains foundational across all of these methods. What's evolving is the preference optimization layer on top of it.

What this means for teams building with LLMs

Most teams should start with SFT and only add preference optimization when they hit a ceiling that labeled data alone can't fix.

That ceiling is real, and you'll know it when you see it: the model follows instructions correctly but its outputs feel flat, unsafe, or weirdly unhelpful in ways that more training data doesn't solve.

The real cost driver in RLHF isn't compute. It's the quality of human judgment feeding the reward model.

Expert preference data from professionals who understand the domain, whether that's medicine, law, or software engineering, produces measurably better reward models than generic crowd-sourced rankings. If you're building models that need to perform on real professional tasks, the APEX benchmark measures exactly that gap.

Whether you're an ML professional evaluating your post-training pipeline or an enterprise team sourcing expert feedback, the next step is the same: figure out where your alignment ceiling is, then decide what kind of human judgment you need to break through it.