A control layer, not a bigger model, moved a coding benchmark by more than a model generation

StateM is a state-machine runtime that sits outside the model context and holds what a long transcript loses: which phase the agent is in, what it has already persisted, and which checks must pass before it may move on. The agent still reasons freely inside a phase. It just cannot leave one until the configured checks commit. Holding weights fixed, GPT-5.5 xhigh with StateM records 92.1% on Terminal-Bench 2.1 against an 83.1% reference for the same model, a nine-point difference, though the authors note that reference is a published public result rather than a rerun A/B control on a matched agent version. Under the reference harness, moving from GPT-5.5 to GPT-5.6 Sol was worth 1.8 points, from 83.1% to 84.9%. The authors put it plainly: "Model scaling expands what the agent can do; harness scaling determines how reliably that capability becomes a completed job."

The headline 95.3% needs its footnote read. With the GPT-5.5 profile frozen and applied unchanged to GPT-5.6 Sol xhigh, the system records 424 successful trials out of 445, or 95.28%. That is a raw, pre-adjudication score from a public Terminal-Bench submission that is still an open pull request and has not been merged into the leaderboard. The authors say so themselves and then do the arithmetic against their own interest: they agree four rewarded trajectories flagged in review should not count, giving 420/445, or 94.38%, and scoring all nine trajectories currently flagged for possible reward hacking as zero gives 415/445, or 93.26%. "We therefore report 95.28% only as a raw public submission score."

Cost is where a purchasing decision changes. Starting from the same runtime and runbook structure, $37.02 of DeepSeek API spend was enough to adapt the profile to DeepSeek-V4 Flash, which then reached 392/445, or 88.09%, on the full 89 tasks under standard timeouts. Producing that final-score evidence cost $15.20 in realized API charges, and all recorded DeepSeek spend across adaptation and evaluation came to $52.22. The public GPT-5.6 Sol max submission it is measured against reports $574.68 in model cost. Transfer is not free, though, and the paper is careful about where it breaks: the frozen GPT profile applied directly to DeepSeek made things slightly worse, moving 82.7% to 82.0%. Exact control profiles carry across nearby models in one family and stop at the provider boundary.

Then there is the section arguing against the authors' own method, which is the one to read if you are building something similar. Failure-driven tuning can preserve the wrong lesson, and they name three cases from their own runs. Their tuning agent set default frame-precision values for video tasks after watching benchmark behavior, resolving an ambiguity the task never specified and then storing that resolution as though it were general. In the DNA tasks, the verifier picks the left-most valid insertion boundary, a convention absent from the visible task description, and repeated feedback taught the profile to reproduce it: "The resulting behavior agrees with the evaluator, but its semantics come from the evaluator rather than the stated task contract." Their design rule is that "experience must be filtered before it becomes memory." They also state that the results measure the runtime combined with an evolved, benchmark-adapted profile and "should not be interpreted as isolating the effect of the state-machine abstraction alone."

Why it matters: the trade between buying a stronger model and building a better system around a cheaper one now has numbers on both sides, from an artifact whose trials are public enough that reviewers found nine of them worth disputing. If you are budgeting next quarter against a model upgrade, the comparison to run is against roughly $38 of harness adaptation. And the number to plan against is not 95.3.