---
title: 'Reinforcement learning from human feedback (RLHF)'
description: 'A training approach that learns a reward signal from human judgments and uses reinforcement learning to optimize a model against that signal.'
canonical_url: 'https://darkfactory.dev/glossary/rlhf'
markdown_url: 'https://darkfactory.dev/glossary/rlhf.md'
collection: glossary
date_published: '2026-08-03T00:00:00-04:00'
date_modified: '2026-08-26T00:00:00-04:00'
---

# Reinforcement learning from human feedback (RLHF)


## Definition

Reinforcement learning from human feedback learns a reward signal from human judgments and then uses reinforcement learning to optimize a policy against that signal. For language models, people commonly rank pairs or sets of candidate responses. A reward model learns to predict those preferences, and a policy optimizer changes the language model to earn a higher predicted reward.

A common LLM pipeline starts with supervised fine-tuning on demonstrations, collects rankings of outputs, fits the reward model, and optimizes the policy while penalizing large departures from a reference model. The supervised stage improves instruction following before reinforcement learning begins. The exact optimizer, comparison format, and reference penalty are design choices rather than parts of the definition.

Human feedback does not become a direct rule inside the model. The reward model generalizes from a finite group of judgments and can miss disagreement, rare harms, factual errors, or behavior outside the comparison distribution. Policy optimization can then exploit those gaps. Independent evaluations and enforceable runtime controls remain necessary.

## From control tasks to language models

Paul Christiano and colleagues' 2017 paper trained agents for simulated robotics and Atari tasks from human preferences between trajectory segments. Daniel Ziegler and colleagues applied learned human-preference rewards to pretrained language models in 2019. OpenAI's 2022 InstructGPT paper then documented the supervised, reward-model, and policy-optimization pipeline that became a common reference for LLM RLHF. These papers trace the method's adoption; they do not establish a single inventor of every technique grouped under the current label.

## Distinguish it from nearby terms

Instruction tuning learns from desired demonstrations. RLHF learns a reward predictor from judgments and runs reinforcement learning against it. DPO trains directly on preference pairs without a separate reward model or online reinforcement-learning phase. Human-in-the-loop describes human participation in an operating workflow, which may happen without changing model weights.

## Check your understanding

Annotators prefer polished answers in the training comparisons, and the policy learns to sound certain even when wrong. Which proxy did the optimizer exploit, and which evaluation or runtime control would detect the missed goal?

## Also called

RLHF

## Related terms

- [Direct preference optimization (DPO)](https://darkfactory.dev/glossary/direct-preference-optimization)
- [Instruction tuning](https://darkfactory.dev/glossary/instruction-tuning)
- [Human in the loop (HITL)](https://darkfactory.dev/glossary/human-in-the-loop)
- [Reward hacking](https://darkfactory.dev/glossary/reward-hacking)

## Evidence and further reading

- [Google Machine Learning Glossary](https://developers.google.com/machine-learning/glossary/)
- [Deep Reinforcement Learning from Human Preferences](https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html)
- [Fine-Tuning Language Models from Human Preferences](https://arxiv.org/abs/1909.08593)
- [Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)