HumanLayer's SlopCodeBench rerun tops out at 16.3% strict pass, and xhigh effort mostly cost more without passing more

Horthy ran the full benchmark catalog, 36 problems and 196 checkpoints per setting, and reports strict pass rates of 12.24% to 16.33%. A strict pass means every test category passes, regression tests included. The top setting, GPT-6 Astra at xhigh effort, passed 32 of 196 at an estimated cost of $758.62. In May, he writes, gpt-5.5 scored 14.8% on the same benchmark.

The effort comparison is the useful part. In the three models run at both medium and xhigh, xhigh cost 2.11 to 4.22 times as much, and strict passes rose only for Astra, from 29 to 32. Opus 5.5 passed 28 at medium and 25 at xhigh. GPT-6.1 SOL at medium passed 29 for $178.51, while Fable 5.1 at xhigh passed 26 for $1,550.07. Horthy cautions that each setting was run once, so the results cannot separate effort from run-to-run variation.

Passing checkpoints is not the same as ending with working software. Across the ten runs, between zero and three of the 36 problems finished with a strict final checkpoint, and the final checkpoints still had 1,435 to 2,831 failing test cases, mostly regression tests. In his summary, Horthy writes that "we'll be keeping the lights on for a while to come." On a 28-problem subset, he reports slop-rule coverage of 77.75% of source lines for Opus 5.5 at xhigh and 94.25% at medium. He adds that the scanner's rules may not cover patterns newer models produce, and says a second part with qualitative analysis is still to come.

Why it matters: If you pick a model and effort level for a long-running build, this is a first-party data point against defaulting to the most expensive setting: the top strict scores sit within a few checkpoints of each other while cost differs by up to roughly 8.7 times. It is one run per setting on the authors' own benchmark, so treat it as a reason to measure your own workload, not as a ranking.