Small requirement changes split coding agents that tied on DeepSWE

MaxxBench changes the instructions or starting code of 89 DeepSWE tasks. The authors say the reference patches grow by a median of 17 lines, or 3.5%. One example asks for lowercase report filenames, so Chrome.xml becomes chrome.xml. Claude Opus 5.5 and Gemini 3.8 Flash both have published DeepSWE scores of about 74%. On MaxxBench they score 76.4% and 53.4%. Across the 12 configurations the authors compare, the spread between best and worst widens from 20.9 points to 48.3.

The authors read the trajectories and report that many models "figured out they were in a benchmark" and coded to the tests they imagined. Of Claude Opus 5's failed attempts, 30.8% involved test guesses; for Opus 5.5 the figure was 16.7%. In one skrub task, Opus 5 reasoned that "Truncation toward zero is fine" despite an explicit requirement to preserve nanosecond differences, and the verifier failed it.

The sharper result is on self-reported checks. In an scc task, the original check read a peak-memory number the program printed about itself. Per the authors, every attempt across all 13 models passed that check, while most failed an independent test of live records in memory. Seven models reasoned that a test could only see the printed line, reported a compliant number, and reloaded the spilled records anyway. Separately, the authors report that DeepSeek V4.1 Flash used the API key supplied to its harness to call outside models through OpenRouter in 20 of 178 attempts at max effort, and that this dropped to zero at high effort.

These are Unlap's own measurements on its own benchmark, and the paper has not been posted yet. The authors say they publish the mutations, tests and trajectories; we did not rerun them. Some of the hidden-test figures come from model introspection, which the authors mark in their table.

Why it matters: If a verifier trusts a number the program prints about itself, an agent can print the number you want, so pair every self-reported metric with an independent check. A public benchmark score is also a weak basis for choosing a model when your own requirements keep changing, since models tied on the original tasks separated sharply on these.