Definition
Reinforcement learning from human feedback learns a reward signal from human judgments and then uses reinforcement learning to optimize a policy against that signal. For language models, people commonly rank pairs or sets of candidate responses. A reward model learns to predict those preferences, and a policy optimizer changes the language model to earn a higher predicted reward.
A common LLM pipeline starts with supervised fine-tuning on demonstrations, collects rankings of outputs, fits the reward model, and optimizes the policy while penalizing large departures from a reference model. The supervised stage improves instruction following before reinforcement learning begins. The exact optimizer, comparison format, and reference penalty are design choices rather than parts of the definition.
Human feedback does not become a direct rule inside the model. The reward model generalizes from a finite group of judgments and can miss disagreement, rare harms, factual errors, or behavior outside the comparison distribution. Policy optimization can then exploit those gaps. Independent evaluations and enforceable runtime controls remain necessary.
From control tasks to language models
Paul Christiano and colleagues' 2017 paper trained agents for simulated robotics and Atari tasks from human preferences between trajectory segments. Daniel Ziegler and colleagues applied learned human-preference rewards to pretrained language models in 2019. OpenAI's 2022 InstructGPT paper then documented the supervised, reward-model, and policy-optimization pipeline that became a common reference for LLM RLHF. These papers trace the method's adoption; they do not establish a single inventor of every technique grouped under the current label.
Distinguish it from nearby terms
Instruction tuning learns from desired demonstrations. RLHF learns a reward predictor from judgments and runs reinforcement learning against it. DPO trains directly on preference pairs without a separate reward model or online reinforcement-learning phase. Human-in-the-loop describes human participation in an operating workflow, which may happen without changing model weights.
Check your understanding
Annotators prefer polished answers in the training comparisons, and the policy learns to sound certain even when wrong. Which proxy did the optimizer exploit, and which evaluation or runtime control would detect the missed goal?