---
title: 'In the News: August 19, 2026, Extra'
description: 'A control layer around an unchanged model moved Terminal-Bench 2.1 further than a model generation did, and did it on a cheap model for about $15.'
canonical_url: 'https://darkfactory.dev/news/2026-08-19-extra'
markdown_url: 'https://darkfactory.dev/news/2026-08-19-extra.md'
collection: news
date_published: '2026-08-19T09:15:00-04:00'
date_modified: '2026-08-19T09:15:00-04:00'
---

# In the News: August 19, 2026, Extra


A preprint posted on August 15 puts a price on a choice most teams are making by instinct. Holding the model fixed and rebuilding the execution layer around it moved a coding-agent benchmark by nine points, against 1.8 points for the step from one model generation to the next. It also shows its working on how much of its own headline number is still disputed. Separately, GitHub's Don Syme sets out where his team puts the quality gates and why the reviewer stays.

## 1. A control layer, not a bigger model, moved a coding benchmark by more than a model generation

**[StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling](https://arxiv.org/abs/2608.15089)** · Ziheng Qin, Yaxin Lu, Zhangyang "Atlas" Wang and Kai Wang · arXiv preprint 2608.15089, August 15, 2026

StateM is a state-machine runtime that sits outside the model context and holds what a long transcript loses: which phase the agent is in, what it has already persisted, and which checks must pass before it may move on. The agent still reasons freely inside a phase. It just cannot leave one until the configured checks commit. Holding weights fixed, GPT-5.5 xhigh with StateM records 92.1% on Terminal-Bench 2.1 against an 83.1% reference for the same model, a nine-point difference, though the authors note that reference is a published public result rather than a rerun A/B control on a matched agent version. Under the reference harness, moving from GPT-5.5 to GPT-5.6 Sol was worth 1.8 points, from 83.1% to 84.9%. The authors put it plainly: "Model scaling expands what the agent can do; harness scaling determines how reliably that capability becomes a completed job."

The headline 95.3% needs its footnote read. With the GPT-5.5 profile frozen and applied unchanged to GPT-5.6 Sol xhigh, the system records 424 successful trials out of 445, or 95.28%. That is a raw, pre-adjudication score from a public Terminal-Bench submission that is still an open pull request and has not been merged into the leaderboard. The authors say so themselves and then do the arithmetic against their own interest: they agree four rewarded trajectories flagged in review should not count, giving 420/445, or 94.38%, and scoring all nine trajectories currently flagged for possible reward hacking as zero gives 415/445, or 93.26%. "We therefore report 95.28% only as a raw public submission score."

Cost is where a purchasing decision changes. Starting from the same runtime and runbook structure, $37.02 of DeepSeek API spend was enough to adapt the profile to DeepSeek-V4 Flash, which then reached 392/445, or 88.09%, on the full 89 tasks under standard timeouts. Producing that final-score evidence cost $15.20 in realized API charges, and all recorded DeepSeek spend across adaptation and evaluation came to $52.22. The public GPT-5.6 Sol max submission it is measured against reports $574.68 in model cost. Transfer is not free, though, and the paper is careful about where it breaks: the frozen GPT profile applied directly to DeepSeek made things slightly worse, moving 82.7% to 82.0%. Exact control profiles carry across nearby models in one family and stop at the provider boundary.

Then there is the section arguing against the authors' own method, which is the one to read if you are building something similar. Failure-driven tuning can preserve the wrong lesson, and they name three cases from their own runs. Their tuning agent set default frame-precision values for video tasks after watching benchmark behavior, resolving an ambiguity the task never specified and then storing that resolution as though it were general. In the DNA tasks, the verifier picks the left-most valid insertion boundary, a convention absent from the visible task description, and repeated feedback taught the profile to reproduce it: "The resulting behavior agrees with the evaluator, but its semantics come from the evaluator rather than the stated task contract." Their design rule is that "experience must be filtered before it becomes memory." They also state that the results measure the runtime combined with an evolved, benchmark-adapted profile and "should not be interpreted as isolating the effect of the state-machine abstraction alone."

**Why it matters:** the trade between buying a stronger model and building a better system around a cheaper one now has numbers on both sides, from an artifact whose trials are public enough that reviewers found nine of them worth disputing. If you are budgeting next quarter against a model upgrade, the comparison to run is against roughly $38 of harness adaptation. And the number to plan against is not 95.3.

## 2. GitHub puts the quality gates before the pull request, not after it

**[Every Repo Is a Software Factory Now](https://tessl.io/podcast/119)** · Don Syme, Principal Researcher, GitHub · The AI Native Dev podcast, August 18, 2026 (publisher's edited transcript)

Syme, who designed F#, now leads GitHub's work on what he calls continuous AI: judgment-heavy work such as triage, documentation and performance research, run on the same scheduled, event-driven footing as CI/CD but kept deliberately separate from it. GitHub Agentic Workflows, the product built on this and now in public preview, runs coding agents inside GitHub Actions with a firewall around each run, a fixed set of inputs, and a bounded set of outcomes. His case for constraint is a case for speed: "the better the guardrails are the faster you can run with automation. It's like train tracks, you know, the stronger the train tracks, the faster you can run." One concrete default: issues from untrusted contributors are ignored unless a workflow explicitly opts in.

Where human attention goes is the sharper argument. Syme does not dispute that review is a bottleneck, but he refuses to solve it by removing the reviewer. "There's just no point creating a pull request for the human to review" when the work is low quality, he argues; the job is to build enough automated quality gates upstream that the reviewer only ever sees candidates worth their time. The word he returns to is equipped. "My job is to equip the reviewer with all the information they need" to understand why a change was made, what the risks are, and what the trade-offs were. In GitHub's own CI performance work, that means shipping the pull request alongside a before-and-after run: the reviewer is handed a measurement rather than asked to trust the agent.

Two smaller practices are worth copying. He evaluates non-deterministic workflows by running several models on identical work at exactly the same moment, taking the first result, and comparing: "A room full of models is doing exactly the same work at exactly the same time, setting them in exam." Simultaneity is the point, because these workflows share a ledger, so a model that runs later can find a closed issue that hints at the answer. He also names an absence: "I think harnesses should have cost controls and kind of budgeting." And he declines the metaphor his own product page uses, preferring to describe a repository as "a site of production, a site where things happen, a site where value is kind of produced," on the grounds that the factory analogy only earns its keep when tasks repeat identically. This is a conversation rather than a study and it reports no dataset. The one throughput figure Syme mentions in passing carries no denominator or time window, so it is not repeated here.

**Why it matters:** GitHub's stated position is that "the human is always in the loop" and "no pull request gets merged without the human actually approving that," which sits against the autonomy-maximalist line elsewhere in this field. Read as a design commitment rather than a limitation, it relocates the engineering: if the merge gate stays human, the work moves into the gates upstream of it and into the evidence attached to whatever arrives. That is a different roadmap from the one that ends in auto-merge.