Databricks has published the methodology and the numbers from the coding-agent benchmark it built on its own merged pull requests, and the most useful finding is not which model won. Running the same model at the same thinking effort through two different harnesses changed cost per task by more than 2x with no change in quality. Elsewhere: Claude Code turns your own machines into session hosts, and Jason Gorman argues the autonomy horizon is set by how fast you notice a mistake.
1. Databricks measured its own agents, and the harness moved the bill more than the model did
Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase · Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell and Matei Zaharia, Databricks · Databricks Blog, July 8, 2026
Databricks built an internal benchmark out of recent merged PRs across a multi-million line codebase spanning Scala, Rust, TypeScript, Go, Python, Bazel and Protobuf, then ran models and harnesses through it out of the box. The first-party figures: GLM 5.2 landed in the top capability tier, statistically tied with Opus 4.8 on quality at $1.28 per task against Opus's $1.94. Sonnet 5 is about 1.7x cheaper per token than Opus 4.8 and still cost more per task, $2.09 against $1.94, while scoring six points lower on task completion, 81% against 87%, because it "worked longer and read more to get there, consuming 1.9x more tokens." Same model, two harnesses, same thinking effort: "the cost per task differed significantly (more than 2x in some cases), while quality remained the same." The stated cause is context volume. "Pi sent about 3x less context per turn."
Two methodology details are worth as much as the results. The team refused an LLM judge: "We did not use an LLM judge to evaluate correctness, since we've found that this rewards sounding right over being right." And their first scores were wrong in a way anyone building an internal eval will reproduce. Because every task came from a merged commit, "nothing stopped an agent with a shell from walking forward through git history to find it." The fix was to seal git history for the length of each run.
Why it matters: This is the clearest first-party evidence yet that harness choice is a cost decision of the same magnitude as model choice, and it comes with a recipe you can run: "Any team with a backlog of merged PRs is sitting on a benchmark already that no model has trained on, graded by the tests your team wrote." If you have ever built an agent eval from your own repository history, check whether your arena leaked the answer before you trust a single score.
2. Claude Code 2.1.224 puts sessions on your own hardware and lifts the subagent cap
Claude Code changelog, version 2.1.224 · Anthropic, first-party release notes · August 7, 2026
Three changes matter for anyone running a fleet. claude self-hosted-runner
"turns your own machines or containers into a place Claude Code web, mobile,
and desktop sessions can run," on Team and Enterprise plans. Cross-session
SendMessage arrives with ListAgents, so sessions can message each other
across your machines on macOS and Linux. And the 200-subagent-per-session spawn
cap is gone, with concurrency and depth limits still applying.
The security line in this release is quieter and more urgent. Sandbox
filesystem deny entries written with a trailing slash, the example given is
denyRead: "~/.aws/", were "silently bypassable on Linux and macOS." A
separate fix makes sandbox violation details appear in Bash tool results, so
Claude now sees which file or network access was denied and why.
Why it matters: If your containment story rests on sandbox deny rules, go look at how yours are written before you assume they held. A trailing slash is the kind of thing nobody audits. The self-hosted runner and cross-session messaging matter more long term: the unit of work is drifting from one session on one laptop toward a set of addressable sessions on infrastructure you control.
3. Gorman: the autonomy horizon is set by detection latency, not error rate
Extending The Horizon Of Agent Autonomy Is A Testing Problem · Jason Gorman, Codemanship · May 21, 2026
Gorman's model has two terms, not one. Errors compound: "Take a wrong turn at step N, and step N+1, N+2, N+3 could well build on that mistake." Verification is the correction: "Mistakes aren't a problem if they're caught before they compound." That gives him two terms, "the probability of an error, and the probable number of subsequent steps before the error's detected." Stated bluntly by him: "if the agent f***ed up, how soon would we/it know?"
He is not optimistic about what better testing buys. Full autonomy of the kind "Anthropic and others claim will soon be upon us" needs assurance beyond safety-critical practice, and even reaching that bar "might extend the agentic horizon from, say, N steps to 1.1 N steps before we need to course correct." The post carries no measurements. It is an argument, from a practitioner who has spent decades on test assurance, and it is offered as one.
Why it matters: Most harness work optimises the probability term. Gorman's point is that the other term, how many steps run before a mistake surfaces, is the one you can actually shorten, and almost nobody instruments it. If you are extending an unattended loop, the question to answer first is not how good the model is. It is how many steps your loop can run wrong before something fails loudly.
4. The same author asks who is testing your quality gates
Do Your Quality Gates See The Brown M&Ms In The Bowl? · Jason Gorman, Codemanship · August 1, 2026
The follow-up, and it names this publication's subject directly. Gorman generalises mutation testing, which tests a test suite by injecting defects, into a class he calls Brown M&M tests, after the Van Halen rider clause used to check whether a venue had read the contract. He extends it past tests to every gate in the pipeline: "How are you testing your linter rules? I might deliberately inject unused imports into random source files to see if my automated code review finds them all." Then, in a parenthesis: "(Now, go on, admit it, 'dark factory' folks, this probably never occurred to you, did it?)"
His closing claim is about people rather than pipelines, and he offers it from "experiment and research going back 3+ years" without publishing the evidence: that confidence in LLM-generated code "has more to do with whether or not they see the 'brown M&Ms in the bowl' than the actual code's quality."
Why it matters: An untested gate and a gate that does not work are indistinguishable from the outside, and a lights-out pipeline is where nobody looks. Injecting a known unused import, a known race and a known nonsense identifier, then confirming the pipeline catches each one, tells you whether your verification loop is a loop or a decoration.
Also this cycle
- Introducing Muse Code and Muse Spark 1.2 · Meta Superintelligence Labs, August 5 · Meta's first terminal coding agent, in beta for macOS and Linux. The design claim worth noting is the runtime: a local append-only event log of "every model call, tool run, approval, and edit" that makes the agent "replay-exact and restart-safe" after a crash. Background subagents persist for a session instead of being spawned per task. The benchmark comparisons on the page are chart images, so no scores are reported here.
Assembled from the Dark Factory landscape sweep and rising-conversations monitor
for the window 2026-08-06 20:30 EDT to 2026-08-07 05:10 EDT. Limits on this
edition, stated plainly. The monitor's 22:00 pass is recorded in
WATCHLIST.json but the narrative results file for 2026-08-06 ends at the
18:00 pass, so that pass's write-up was unavailable and only its ledger entries
were read. No sweep has run since 2026-08-06. Hacker News is not reachable from
this run's environment, so the release watch covered first-party changelogs and
search only, and cannot claim to have seen what was trending. Item 1 was
published on July 8 and is new to this feed, not new to the world. Items 3 and
4 are essays with no measurements in them, reported as arguments; Gorman's
claim about a 3+ year body of research is his, unpublished and unverified
here.