SlopCodeBench's author reports less line-level slop and agents that rewrite the libraries they are given

Orlanski compares older and newer model runs on SlopCodeBench, 36 Python problems with 196 checkpoints in which an agent keeps extending its own code as the spec changes. Dex Horthy ran the models; this post is the follow-up to his write-up of pass rates and cost.

The line-level numbers improved. The strict solve rate now peaks at 16.3% (GPT-6 Astra XHigh, 32 of 196 checkpoints). From older to newer Anthropic runs, hits on the benchmark's slop rules per 1,000 lines fell 25%, and the count of functions over 200 lines fell 93%.

Orlanski's summary is that the slop "has moved up the stack to now infest design decisions." Error handling got worse for all three model families he compares. Some flagged patterns were swapped for narrower ones that pass the rule; of one before-and-after pair he writes, "Same bug, narrower exception." Python 2 habits came back: percent formatting is the main style in 14 to 23 of 36 problems for Opus 5.5, Fable and GLM 5.3, against at most 3 for any older run.

The largest numbers come from two new benchmarks. On LibraryUseBench, each model gets a problem, a human-written library and a minimal prompt saying the library is available. Every model hand-wrote something the library already provides in 73% to 100% of its solutions. Under LibraryDesignBench's much more prescriptive prompt, reimplementation fell to about 17% of solutions on human-designed libraries and 40% on agent-written ones. On that benchmark the same three implementer agents also had to work around the library, patching it or rebuilding a piece, in 40% of solutions when an agent wrote it, against 27% for the human-written one.

These are the author's own benchmarks, and he says so: "I might be biased". LibraryUseBench is 15 tasks, scored on a first attempt, with subagents judging what counts as a rewrite. We read the post in full, with its charts as text. We did not open the LibraryDesignBench launch post or the benchmark sites.

Why it matters: A falling lint count is weaker evidence than it was. By Orlanski's account the flagged pattern can disappear while the bug stays, so a rule-based slop check needs a reviewer who reads what replaced the pattern. If your agents build on shared libraries, look for reimplemented library functions in review and spell out in the prompt which library calls to use. Telling the agent a library is available did not stop it rewriting library functions in most of these runs.