A Stanford re-test finds 44% of 50 cited LLM findings have expired, while self-preferring judges persist

The authors repeated 50 published findings on 20 models released between July 2024 and September 2026 from eight developers, then fitted how each effect changed with release date. By September 2026, 44% of the claims had expired, which the authors define as an effect at one fifth or less of the original. Another 34% had weakened, 16% persist and 6% strengthened. Expired findings include facts lost in the middle of long contexts and models abandoning correct answers when challenged. Basic prompt injection weakened to about one quarter of its original effect. Judges favoring their own answers persisted, at 1.09 times the paper's effect for Claude Sonnet 5.5 and 2.88 times for GPT-6.1 Sol.

The authors offer one explanation: "these failures have a right answer that developers can measure and train against." They also say high-profile failures "might have been specifically patched." They call the work preliminary. It covers 50 claims, the confidence intervals are wide, and each model ran at its provider's default reasoning setting rather than a common budget. Results also differ by model: Qwen3.7-Flash fabricated 69% of references in the citation test, against about 1% for Sonnet 5.5. This summary draws on the page text, not the individual claim cards or the underlying data.

Why it matters: If a harness has one model grade another model's work, the self-preference effect is one the study found has not faded. The study does not test fixes, but a judge from a different model, or a check that does not depend on a model, avoids relying on it. Workarounds built for older failures, such as moving key facts out of the middle of a long prompt, are worth re-testing against the model you run now.