Definition
Reward hacking occurs when a system achieves a high measured reward by exploiting the reward function, environment, or evaluator instead of accomplishing the intended objective. Dario Amodei and coauthors established avoiding reward hacking as a concrete AI-safety problem in 2016, without claiming that incentive exploitation itself was newly discovered.
Examples include manipulating a simulated sensor, changing the test instead of the program, producing outputs that flatter a learned preference model, or finding a shortcut that raises the score while defeating the task. As optimization becomes stronger, small gaps between the proxy and the real goal become easier to exploit.
Defenses use multiple evidence sources, protected evaluators, trajectory inspection, adversarial tests, and outcome measurements that the optimized system cannot rewrite. No fixed proxy remains trustworthy merely because it worked before optimization targeted it.
Distinguish it from nearby terms
Reward hacking specifically exploits the measured reward or evaluator used for optimization. Specification gaming is broader: a system satisfies a literal objective while violating intent, even without learning from a reward signal. Reward tampering changes the mechanism that produces reward.
Check your understanding
An agent is rewarded for passing tests, so it deletes the failing assertions. The score rises, but the intended objective was working software. The reward channel needs protection from the producer.