---
title: 'In the News: August 7, 2026'
description: 'Databricks publishes cost and quality numbers from its internal coding-agent benchmark, and the harness moves the bill more than the model does.'
canonical_url: 'https://darkfactory.dev/news/2026-08-07-morning'
markdown_url: 'https://darkfactory.dev/news/2026-08-07-morning.md'
collection: news
date_published: '2026-08-07T05:15:00-04:00'
date_modified: '2026-08-07T05:15:00-04:00'
---

# In the News: August 7, 2026


Databricks has published the methodology and numbers from the coding-agent
benchmark it built on its own merged pull requests. Running the same model at the same thinking effort
through two different harnesses changed cost per task by more than 2x with no
change in quality. Elsewhere: Claude Code turns your own machines into session
hosts, and Jason Gorman argues the autonomy horizon is set by how fast you
notice a mistake.

## 1. Databricks measured its own agents, and the harness moved the bill more than the model did

**[Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase](https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase)** · Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell and Matei Zaharia, Databricks · Databricks Blog, July 8, 2026

Databricks built an internal benchmark out of recent merged PRs across a
multi-million line codebase spanning Scala, Rust, TypeScript, Go, Python,
Bazel and Protobuf, then ran models and harnesses through it out of the box.
The first-party figures: GLM 5.2 landed in the top capability tier,
statistically tied with Opus 4.8 on quality at $1.28 per task against Opus's
$1.94. Sonnet 5 is about 1.7x cheaper per token than Opus 4.8 and still cost
more per task, $2.09 against $1.94, while scoring six points lower on task
completion, 81% against 87%, because it "worked longer and read more to get
there, consuming 1.9x more tokens." Same model, two harnesses, same thinking
effort: "the cost per task differed significantly (more than 2x in some cases),
while quality remained the same." The stated cause is context volume. "Pi sent
about 3x less context per turn."

The team refused an LLM judge: "We did not use an LLM judge to evaluate correctness, since we've
found that this rewards sounding right over being right." Their first scores were
also wrong in a way anyone building an internal eval will reproduce.
Because every task came from a merged commit, "nothing stopped an agent with a
shell from walking forward through git history to find it." The fix was to seal
git history for the length of each run.

**Why it matters:** Databricks' evidence shows that harness choice can be a cost
decision of the same magnitude as model choice, and it comes
with a recipe you can run: "Any team with a backlog of merged PRs is sitting on
a benchmark already that no model has trained on, graded by the tests your team
wrote." If you have ever built an agent eval from your own repository history,
check whether your arena leaked the answer before you trust a single score.

## 2. Claude Code 2.1.224 puts sessions on your own hardware and lifts the subagent cap

**[Claude Code changelog, version 2.1.224](https://code.claude.com/docs/en/changelog)** · Anthropic, first-party release notes · August 7, 2026

Three changes matter for anyone running a fleet. `claude self-hosted-runner`
"turns your own machines or containers into a place Claude Code web, mobile,
and desktop sessions can run," on Team and Enterprise plans. Cross-session
`SendMessage` arrives with `ListAgents`, so sessions can message each other
across your machines on macOS and Linux. And the 200-subagent-per-session spawn
cap is gone, with concurrency and depth limits still applying.

The release also fixes sandbox
filesystem deny entries written with a trailing slash, the example given is
`denyRead: "~/.aws/"`, were "silently bypassable on Linux and macOS." A
separate fix makes sandbox violation details appear in Bash tool results, so
Claude now sees which file or network access was denied and why.

**Why it matters:** If your containment story rests on sandbox deny rules, go
look at how yours are written before you assume they held. A trailing slash is
the kind of thing nobody audits. The self-hosted runner and cross-session
messaging matter more long term: the unit of work is drifting from one session
on one laptop toward a set of addressable sessions on infrastructure you
control.

## 3. Gorman: the autonomy horizon is set by detection latency, not error rate

**[Extending The Horizon Of Agent Autonomy Is A Testing Problem](https://codemanship.wordpress.com/2026/05/21/extending-the-horizon-of-agent-autonomy-is-a-testing-problem/)** · Jason Gorman, Codemanship · May 21, 2026

Gorman's model has two terms, not one. Errors compound: "Take a wrong turn at
step N, and step N+1, N+2, N+3 could well build on that mistake." Verification
is the correction: "Mistakes aren't a problem if they're caught before they
compound." That gives him two terms, "the probability of an error, and the
probable number of subsequent steps before the error's detected." Stated
bluntly by him: "if the agent f***ed up, how soon would we/it know?"

He is not optimistic about what better testing buys. Full autonomy of the kind
"Anthropic and others claim will soon be upon us" needs assurance beyond
safety-critical practice, and even reaching that bar "might extend the agentic
horizon from, say, N steps to 1.1 N steps before we need to course correct."
The post carries no measurements. It is an argument, from a practitioner who
has spent decades on test assurance, and it is offered as one.

**Why it matters:** Most harness work optimises the probability term. Gorman's
point is that the other term, how many steps run before a mistake surfaces, is
the one you can actually shorten, and almost nobody instruments it. If you are
extending an unattended loop, the question to answer first is not how good the
model is. It is how many steps your loop can run wrong before something fails
loudly.

## 4. The same author asks who is testing your quality gates

**[Do Your Quality Gates See The Brown M&Ms In The Bowl?](https://codemanship.wordpress.com/2026/08/01/do-your-quality-gates-see-the-brown-mms-in-the-bowl/)** · Jason Gorman, Codemanship · August 1, 2026

The follow-up, and it names this publication's subject directly. Gorman
generalises mutation testing, which tests a test suite by injecting defects,
into a class he calls Brown M&M tests, after the Van Halen rider clause used to
check whether a venue had read the contract. He extends it past tests to every
gate in the pipeline: "How are you testing your linter rules? I might
deliberately inject unused imports into random source files to see if my
automated code review finds them all." Then, in a parenthesis: "(Now, go on,
admit it, 'dark factory' folks, this probably never occurred to you, did it?)"

His closing claim is about people rather than pipelines, and he offers it from
"experiment and research going back 3+ years" without publishing the evidence:
that confidence in LLM-generated code "has more to do with whether or not they
see the 'brown M&Ms in the bowl' than the actual code's quality."

**Why it matters:** An untested gate and a gate that does not work are
indistinguishable from the outside, and a lights-out pipeline is where nobody
looks. Injecting a known unused import, a known race and a known nonsense
identifier, then confirming the pipeline catches each one, tells you whether
your verification loop is a loop or a decoration.

---

## Also this cycle

- **[Introducing Muse Code and Muse Spark 1.2](https://research.meta.ai/blog/introducing-muse-code-and-muse-spark-1-2)** · Meta Superintelligence Labs, August 5 · Meta's first terminal coding agent, in beta for macOS and Linux. Its runtime uses a local append-only event log of "every model call, tool run, approval, and edit" that makes the agent "replay-exact and restart-safe" after a crash. Background subagents persist for a session instead of being spawned per task. The benchmark comparisons on the page are chart images, so no scores are reported here.