A harness upgrade matched a model-generation jump, with costs and failures on the record

The authors built StateM, a runtime that wraps a coding agent in durable states, checked transitions, and versioned runbooks without touching model weights. Applied to GPT-5.5 on Terminal-Bench 2.1, it raised accuracy from an 83.1 percent reference to 92.1 percent, close to the newer GPT-5.6 Sol Ultra's 91.9 percent. The same runbook, frozen and unmodified, then took GPT-5.6 Sol to 95.28 percent raw accuracy in a public leaderboard submission, recorded in PR #142 on the harbor-framework repository, open and unmerged as of this writing. The authors disclose their own caveat: under stricter scoring that excludes flagged trajectories, the figure drops to 93.26 percent. Moving to a different model provider broke the exact transfer. The frozen GPT-tuned profile applied to DeepSeek-V4 Flash made results worse, from 82.7 percent to 82.0 percent. Adapting the runbook to DeepSeek cost under $38 and brought accuracy to 88.1 percent, close to the GPT-5.6 reference at a fraction of the price: about $15 in API charges for the evaluation run, against a reported $574.68 for the GPT submission.

Why it matters: These results isolate the harness as a variable separate from model choice and attach costs to both the successful run and the provider adaptation. They also mark a boundary: the general control structure carried across providers, but the concrete rules did not, and adapting them is what the $38 paid for.