Harness choice moved cost much more than accuracy on self-hosted open-weight models

Using the same 64 tasks, the team compared Claude Code, Codex and Pi on Qwen3.8-27B (one H200) and GLM-5.3-Flash (four H200s). Resolve rates stayed within 44% to 53%, a spread of two to three tasks. Token use did not. On GLM-5.3-Flash, Claude Code pushed 480.2M input tokens through the model against 192.1M for Codex, and the sweep took 2.48 hours against 1.26. At the team's stated rental rate of $18.16 per hour, the authors put the cost of one sweep at about $22.8 for Codex and $45 for Claude Code and for Pi. On Qwen the cost gap was small, about $9 against $10.26.

The post also reports a regression. Claude Code version 2.1.233 showed a mean prefix-cache hit rate of 29.84% on Qwen, against 97.36% for the patched release, which the authors link to a GitHub issue. Separately, one or two Pi workers repeatedly fell into a loop and were killed after four hours; the Pi figures are corrected for that. These are single sweeps on self-hosted models, with cost computed from wall time at the authors' GPU rates, so the post says nothing about API-hosted pricing.

Why it matters: If you run agents on your own GPUs, the harness you pick decides how long each task holds a GPU, and an unpinned upgrade can break prompt caching without changing accuracy. Track cache hit rate and wall time per release, not only resolve rate.