Databricks measured its own agents, and the harness moved the bill more than the model did

Databricks built an internal benchmark out of recent merged PRs across a multi-million line codebase spanning Scala, Rust, TypeScript, Go, Python, Bazel and Protobuf, then ran models and harnesses through it out of the box. The first-party figures: GLM 5.2 landed in the top capability tier, statistically tied with Opus 4.8 on quality at $1.28 per task against Opus's $1.94. Sonnet 5 is about 1.7x cheaper per token than Opus 4.8 and still cost more per task, $2.09 against $1.94, while scoring six points lower on task completion, 81% against 87%, because it "worked longer and read more to get there, consuming 1.9x more tokens." Same model, two harnesses, same thinking effort: "the cost per task differed significantly (more than 2x in some cases), while quality remained the same." The stated cause is context volume. "Pi sent about 3x less context per turn."

The team refused an LLM judge: "We did not use an LLM judge to evaluate correctness, since we've found that this rewards sounding right over being right." Their first scores were also wrong in a way anyone building an internal eval will reproduce. Because every task came from a merged commit, "nothing stopped an agent with a shell from walking forward through git history to find it." The fix was to seal git history for the length of each run.

Why it matters: Databricks' evidence shows that harness choice can be a cost decision of the same magnitude as model choice, and it comes with a recipe you can run: "Any team with a backlog of merged PRs is sitting on a benchmark already that no model has trained on, graded by the tests your team wrote." If you have ever built an agent eval from your own repository history, check whether your arena leaked the answer before you trust a single score.