Coding-agent reports leave a 40.7-point gap in what a supervisor can recover

The authors built a measurement they call CTXGAP. For each coding task they ask the same questions about what happened (failed attempts, workarounds, residual risks) of a reader who sees only the agent's accumulated reports and of a reader who sees the full recorded execution. Across 37 SWE-bench Verified tasks and 4,799 valid probes, the agents answered 93.6% of the questions correctly when asked directly, the full-context reader recovered 92.7%, and the report-only reader recovered 52.0%. At the final checkpoint of 30 tasks whose fixes passed the hidden tests, the gap was still 34.8 points. The authors report similar gaps with Claude Code and two other harnesses on 12 shared issues.

Much of the missing information was in the reports but too thin to use. On a sample of 195 gap items, two human annotators put states mentioned without enough supporting detail at 55.4% and 59.2% of the weighted gap, against 42.1% and 33.7% for states never mentioned. Adding a bare mention of an event did not help (a change of -0.002); adding the supporting evidence raised recovery by 0.189. Full terminal scrollback lifted recovery to 73.6% but did not close the gap.

Why it matters: A green test run and a confident summary do not tell you what the agent hit along the way. If you review agent work from its report alone, these numbers suggest you are seeing about half of the task state, so a verification loop needs access to the execution record, not only the write-up. The questions and judgments are model-generated by the authors' own pipeline, they chose the 37-task subset rather than sampling it, and we did not read the appendices.