A harness benchmark puts a 17-fold cost gap on running the same model

Runta, a startup selling what it calls an agent execution layer, ran 360 trials across twelve harness configurations, including Codex, Claude Code, Pi, three DeepSeek Harness variants, Kimi Code, Exo Harness, OpenCode, and Hermes, on identical software engineering tasks. Every run used the same model, Kimi K3, restored from an identical cold checkpoint each time, so the comparison holds the model fixed and varies only the harness. Pass rates ranged from 50.0 percent (OpenCode and Hermes, tied for lowest) to 66.7 percent (Codex). Median cost per task ranged from $1.05 (Exo Harness) to $18.34 (Claude Code, tested at version 2.1.237, an older build than the version currently shipping), a spread of about 17.5 times that matches the post's own headline figure.

Runta's write-up notes two caveats. OpenCode's separately reported "cost per successful task" figure of $0.0615, the lowest of the twelve harnesses on that metric, excludes 15 failed attempts out of 30 total runs; counted in, the number becomes $3.24. A high cache hit rate does not guarantee a cheap run, since a long cached failure can still cost more than a short cache miss.

In the Hacker News discussion (36 points and 14 comments at publication), commenter vidarh challenged the study's framing, writing that "testing it against Kimi is potentially skewing the numbers massively" because harnesses built around Anthropic's models may not need the tool call loop workarounds Kimi requires, and called the blog's own claim of no home field advantage "a dicey assumption." A separate commenter, GodelNumbering, asked for instructions on running a custom harness through the benchmark and found none posted on the page.

Why it matters: The harness wrapped around a model is turning into its own cost lever, separate from the model underneath it, with a difference here of roughly 17 times on the same task set. That is worth checking before standardizing on one harness, but it is one benchmark, one model, and one vendor's infrastructure, so it reads as a data point rather than a ranking to copy.