Harness choice moves cost more than success rates

The researchers tested seven models across three harnesses, Claude Code, Codex CLI, and the minimal open-source harness Pi, on SWE-bench Lite and Terminal-Bench 2.0: 21 pairs, 30 sampled tasks repeated three times each. Claude Fable 5 solved 97.8 percent of attempts in Claude Code versus 96.7 percent in both Codex and Pi, a gap within noise, yet Claude Code cost about twice as much per attempt: $1.33 against $0.67 in Pi. Across shared models, Claude Code cost roughly 2.0 times as much as Pi and 1.6 times as much as Codex on SWE-bench Lite, while the average effect of harness choice on success rate stayed within about 2 percentage points on that benchmark and about 5 points on Terminal-Bench 2.0. Some of the gap shows up before the agent does anything: across all seven models, Claude Code's mean initial context ran more than ten times Pi's, in instructions and tool schemas alone. The authors put it plainly: "you may be paying such a hidden 'harness tax' when you accept a coding agent's default harness without comparing alternatives." Across six Anthropic and OpenAI models on both benchmarks, a different harness produced the best observed success rate in nine of twelve comparisons, including Sonnet 4.6 solving more tasks in Codex than in Claude Code at a similar cost. The authors say they will publicly release their profiling traces.

Why it matters: Teams standardizing a factory's coding agent should benchmark the model and harness separately. In this study, harness choice affected cost much more than task success, and vendor-matched pairings did not reliably lead the comparisons.