An evidence gate that blocks premature commitment lifts SWE-bench Pass@1 by up to 11.8 points

The paper names a specific failure mode: an agent edits code or submits a patch "before examining enough repository evidence to justify the change," which it calls premature commitment, distinct from violating a safety or governance policy. Its fix, ECLoop, compiles a list of what the agent should inspect before each type of edit or submission, tracks which of those conditions the agent's own trajectory has satisfied, and holds any edit or submission whose relevant conditions remain unmet, feeding the gap back to the agent rather than blocking it outright. Across all 500 SWE-bench Verified instances, two models (MiniMax-M2.5, GPT-5-mini) and two agent scaffolds, ECLoop raises Pass@1 by 4.8 to 11.8 percentage points; GPT-5-mini under the mini-swe-agent v2 scaffold goes from 56.2 percent to 68.0 percent. Token use falls by up to 12.1 percent and cost by up to 10.2 percent, because the gate keeps the agent from chasing unsupported edits into longer, unproductive trajectories. The paper was also integrated into Codex CLI through its existing hook mechanism, with no change to the underlying model, tools or action-selection policy.

Self-Refine, where the model reviews and revises its own patch after the fact, made results slightly worse (down 1.4 and 1.8 percentage points across the two models) rather than better: post-hoc review cannot recover from a decision made on evidence that was never gathered. And in an ablation on a 100-instance subset, replacing the structured evidence conditions with an equivalent natural-language summary dropped solves from 68 to 58, below the 63 solved with no guidance at all. The format of the check is carrying the result; a reminder alone is not.

Why it matters: if you are building a verification loop for a coding agent, this is evidence that a structured, trajectory-checked gate on commitment actions outperforms both of the two things teams commonly reach for instead, asking the model to review its own output afterward, or just telling it to look harder first. Neither substitutes for tracking, per task, what has actually been observed.