Aligning a model to human preferences using pairwise judgments ("response A is better than B"). RLHF trains a reward model then optimizes with RL (PPO). DPO skips the reward model and optimizes the preference objective directly.
How it works
Reinforcement Learning from Human Feedback (RLHF) starts by collecting a dataset of ranked outputs, where human raters indicate which response is preferred between pairs. This data trains a Reward model that predicts the quality of a response based on human preferences. Then, Proximal Policy Optimization (PPO) fine-tunes the model's policy, nudging it to generate responses that maximize the predicted reward. A KL penalty ensures that the new policy doesn't stray too far from the original model, maintaining its foundational knowledge while adapting to human preferences.
In contrast, Direct Preference Optimization (DPO) simplifies this process by directly reframing the preference data into a classification-style loss function. Instead of relying on a reward model and the complexities of reinforcement learning, DPO treats pairs of chosen and rejected responses as binary classifications. This approach eliminates the need for sampling, which can introduce instability in RL methods. Variants like IPO, KTO, and ORPO offer different ways to implement preference tuning, while tools like TRL and Axolotl provide the necessary frameworks for both RLHF and DPO.
For example, suppose we have two responses to a query: Response A and Response B. In RLHF, we would first train a reward model on many such comparisons, then use PPO to adjust the model's outputs based on the learned rewards. In DPO, we would directly optimize the model to classify Response A as better than Response B without building a reward model, which streamlines the process and reduces the potential for reward hacking.
- Collect ranked outputs for RLHF.
- Fit a reward model for RLHF.
- Use PPO for RLHF optimization.
- Use preference data for DPO.
- Optimize directly with classification loss in DPO.
The software-engineering analogue
This process is similar to tuning a ranking system based on A/B testing data, where you adjust the model based on user interactions rather than relying on predefined rules. In software, you might collect click data to determine which version of a webpage performs better. However, the key difference here is that the objective in DPO and RLHF is not just about maximizing clicks but learning a nuanced model of human taste and preference. This means that while traditional A/B testing optimizes for direct outcomes, these AI methods can explore deeper layers of preference that might lead to unintended consequences if not managed carefully.
Why not the obvious alternative
A newcomer might first consider supervised fine-tuning (SFT) as the go-to method for aligning models with human preferences. However, SFT only mimics good examples from the training data and lacks the capability to express preferences like 'avoid this' or to rank two plausible answers. This limitation means that while SFT can produce valid outputs, it often fails to capture the relative quality of responses or the nuances of safety and alignment that preference tuning can address.
This shortfall in SFT leads to a model that may generate acceptable responses but lacks the discernment to prioritize one response over another when both seem valid. Preference tuning, through DPO or RLHF, addresses this by capturing the relative quality of responses, allowing for a more sophisticated understanding of what makes one answer preferable to another. This is crucial for developing models that align closely with human values and expectations.
When to use which
Start with DPO when you want a simpler, more stable, and cost-effective solution. It provides strong results and has become the industry default for preference tuning. DPO is particularly useful when you have a clear preference dataset and want to optimize quickly without the complexities of reinforcement learning. However, if your application requires online reward signals, involves complex multi-objective shaping, or if you’re working at the frontier of model capabilities with a mature reward-modeling team, then transitioning to RLHF with PPO may be necessary to achieve the desired outcomes.
In production: when it breaks
Why it happens. The drop in user satisfaction can occur if the model is overfitting to the preferences learned during training, failing to generalize well to unseen scenarios. This can happen particularly with DPO if the preference data is not diverse enough, leading to biased outputs that do not align with broader user expectations.
- Check the distribution of training preferences for diversity.
- Log user satisfaction scores across different types of queries.
- A/B test the model with a subset of users using both RLHF and DPO approaches.
What it delivers
The end result of using either DPO or RLHF is a model that better understands helpfulness, tone, and safety in its responses. These methods allow the model to select the better answer from a set of plausible options, rather than simply producing a valid response. This alignment with human preferences leads to outputs that are not only accurate but also more aligned with user expectations and needs.