Coding agents often have the right patch long before they stop

The authors saved the repository patch after every agent step and later scored each distinct patch against the official SWE-bench tests, which the agent never saw. In solved runs in their core settings, a median of 35% to 60% of steps came after the first correct patch. They also report 54% for Claude Haiku 5.5 and 36% for GPT-6 Luna on SWE-bench Verified. Among runs that reached a correct patch, the first source-code edit was already correct in 38% to 82% of cases, depending on the setting.

Most of the extra steps were checking. Verification took 48% to 63% of the steps after the first correct patch, and exploring the repository took another 15% to 29%. Most runs kept the correct patch to the end (61% to 96%). In 1% to 8% of runs, later edits broke it and the agent submitted an incorrect patch. The paper counts steps, not dollars, ran each task once, and does not propose a stopping rule; it measures how much a perfect one could save.

Why it matters: For a team paying per step, this puts a ceiling on what better stopping is worth: roughly a third to more than half of a solved run, by these measurements. It also says the extra checking is not free of risk, since a small share of runs undo a working fix. Recording the patch at each step, as the authors did, is a method a team can copy to find out how much of its own agent spend falls after the fix.