What is Reinforcement Learning from Human Feedback?
Also called RLHF, preference tuning.
Reinforcement learning from human feedback is a training method that shapes a model's behavior using human judgments about which of several candidate responses is better. Those comparisons train a reward model, which then guides further optimization of the language model itself. It is the main technique behind the helpful, cautious tone that most commercial assistants share.
The classic pipeline has three steps. An instruction-tuned model generates several answers to the same prompt, human annotators rank them from best to worst, and a reward model learns to predict those rankings. The language model is then optimized to score well under the reward model, while a penalty term keeps it from drifting too far from its starting behavior.
The method targets qualities that are easy to recognize but hard to specify: tone, tact, refusing harmful requests, admitting uncertainty, and knowing when to ask a clarifying question. Writing an explicit rule for a good answer is impractical, but people can compare two answers quickly and fairly consistently, and that comparison signal turns out to be enough to steer behavior.
Optimizing against a learned proxy invites reward hacking. Models can become sycophantic, agreeing with a user's incorrect premise because agreement was rated favorably, or excessively hedged and quick to refuse. Whose preferences get collected also matters a great deal, since annotator guidelines and demographics are baked into whatever the finished model treats as good.
Variants now compete with the original recipe. Direct preference optimization skips the separate reward model and trains on the comparisons directly, and constitutional approaches replace some human labels with model-generated critiques against a written set of principles. Practitioners often use the acronym loosely to mean this whole family of preference-based post-training methods.
Key points
- Humans rank candidate responses; a reward model learns those preferences.
- Targets qualities that are easy to judge but hard to define.
- Produces the cautious, helpful tone common to commercial assistants.
- Can cause sycophancy, over-refusal, and reward hacking.
- Newer variants train on preferences directly, without a reward model.
In practice
An annotator sees one prompt and two answers to a question about a medication. The first is confident and slightly wrong; the second is accurate and adds a note to consult a pharmacist. The annotator picks the second. Repeated across many thousands of such comparisons, this teaches a general pattern: prefer accurate, appropriately cautious answers over confident-sounding ones.