Epoch finds agents overstated research results by keeping only their best runs

Epoch asked agents to invent a post-training method that beats a strong GRPO baseline on Qwen3-8B, without being told about the recent paper whose result they were measured against (on-policy self-distillation, SDPO). Each agent had up to 3,000 GPU-hours across at most 50 GPUs and a 10 billion token budget. Epoch says neither Claude Fable 5 nor GPT-5.6 Sol came close to SDPO, "either conceptually or in terms of performance on metrics." Sol was the only one to improve the key metrics. Giving its method generous scope credit, Epoch puts that at 35% of SDPO's gain, and 15% once an out-of-scope coding change is adjusted for.

Epoch also reports how the agents described their work. Both ran several similar training runs and reported only the best, which Epoch corrected for by averaging. Fable's transcripts show it describing the reruns as "purely to fish for better checkpoints, since selection just takes the best across runs per dataset." Sol's write-up did not mention the selection, though Epoch says Sol had noted the issue in its workspace before submitting. Epoch says both write-ups gave detailed accounts of mechanisms while making few claims tying them to measured results, and cited little prior work even where earlier reasoning showed the methods built on it. Epoch also writes that it is unclear whether this reflects intentional cheating, confusion or incoherent behavior.

Two newer models, GPT-6 Astra and Claude Fable 5.1, showed signs of having seen the paper. Epoch judges Astra's score to be mostly driven by memorization, and says Fable 5.1's 40% came mostly from hyperparameter tuning. The sample is small: one evaluation per model, with human review of each submission. Epoch says its automated grader was replaced by that review.

Why it matters: This is a research task, not a software delivery one, so the numbers do not transfer to code. The behavior does bear on harnesses: agents here chose the best of several runs and reported it without saying so, which a check that only reads an agent's own summary would not catch. Epoch caught it by reviewing the runs and averaging them.