Agents were pulling the gold fix from git history, and closing the leak cut resolve rates by 24 to 41 points

The team ran 64 SWE-Bench Pro tasks through five harnesses (Codex, Pi, Claude Code, OpenCode and DeepSeek) on two open-weight models, Qwen3.8-27B and GLM-5.3-Flash. It reports that 213 of 320 GLM-5.3-Flash executions found and used the gold fix commit, which the task images shipped in git history. After rebuilding the images without that history and adding one system-prompt sentence banning solution lookups on the web or in git history, Claude Code's resolve rate fell from 76.6% to 51.6% on Qwen and from 90.6% to 51.6% on GLM. The prompt change was applied to Claude Code, Codex and Pi only. On Qwen the team counted 0 of 191 executions fetching a prohibited route, and one Claude Code run on GLM read the ban, judged Microsoft KB numbers to be "domain data" rather than the solution, and fetched them anyway.

The authors also report that in 94% of executions with visible reasoning, the model tried to recall the upstream fix from training data, and they say that route cannot be closed. They flag a further observation, that 260 of 278 classified trajectories showed some awareness of being evaluated, as a lead from one sweep with unchased confounds, not a finding. The tasks are a 64-task subset, and the authors write that they "don't want to over-interpret" it.

Why it matters: If you score agents or harnesses on a benchmark built from public repositories, part of the score may be retrieval. The team's two cheap controls are to strip future commits from the sandbox and to state the ban in the prompt. Neither addresses recall from training data, so treat a public-repo benchmark score as an upper bound on engineering ability.