← In the News

The best harness depends on the model, says a 27,280-run benchmark

Introducing FrontierHarness Eval v1.1 · Shilin Zhu, Runta, with Baseten · Runta blog, November 17, 2026

Machine-readable Download Markdown

Runta ran 22 harness configurations against five open models served by Baseten, on 62 terminal tasks with four attempts each: 27,280 runs and 83 billion tokens. Each model had a different best-value harness. Codex CLI was the cheapest per pass with GLM-5.3 and 16th of 22 by cost per task with Kimi-K3, and by cost per task the median harness moved 11 places between models. The model explains 92.8% of the variance in cost per task. With the model fixed, the dearest harness cost 1.8 to 5.0 times the cheapest per task, and the extra spend did not buy a higher pass rate.

Steps per pass tracked cost per pass (rank correlation +0.51, p = 0.015); prompt size, tool count and cache hit rate did not. At list prices, 64.5% of spend went to trials that failed or timed out. Among the five best-value harnesses, 68% to 82% of failed runs ended with a confident final answer rather than at the time limit. The post's advice: "For harness developers: cut steps, not just prompt tokens."

Runta tested only open models, on one provider's prices and cache. Each pass rate carries an interval of about plus or minus 10 points. When Runta redrew the tasks 4,000 times, Codex CLI with GLM-5.3 stayed the winner 99.6% of the time; the other four winners did so in 25% to 53% of draws. The thread had 182 points and 74 comments on Hacker News at about 13:03 EST.

Why it matters: A harness picked for one model is not a safe default for the next one, so re-test candidates on your own tasks when you switch. The confident-failure finding argues for an independent check before a run is allowed to declare itself done. Runta says such a check could recover part of the cost spent on failed runs, but it did not test that.