A new benchmark holds the model fixed and swaps the harness

The repository holds Opus 5.5 and a set of 20 public multi-user tasks fixed and changes only the harness. On Opus 5.5, pass rates were QM 72.5%, Claude Code 76.3%, Hermes Agent 78.8% and OpenClaw 76.3%, and no tested pair could be told apart. Cost per solved task was $0.19 for QM, $0.22 for Claude Code, $0.24 for Hermes Agent and $0.44 for OpenClaw. The README says about 81% of OpenClaw's extra cost over Claude Code comes from cache writes.

The per-harness findings are more specific than the ranking. In Claude Code, a rule placed in the system prompt or startup file was followed in 30 of 30 turns against 18 of 30 with no rule, and a skill that was not mentioned was opened 0 times in 30 turns. Dropping its memory-and-skills text together with its tool-description advice took pass rate on harder tasks from 60.4% to 0.0%. Told to run gstack's skills, Opus 5.5 loaded the skill text every time and ran none of the steps, in 0 of 46 sessions.

Why it matters: The cost gap is large enough to matter at volume, and the placement result is cheap to act on: put the rules you need followed where the agent sees them every turn, and name the skills you want used. The README labels these dev results on public tasks, not held-out confirmation, and says most differences are descriptive. Tan runs Y Combinator, which the README says owns QM and invests in Hermes, and gstack and gbrain are his own projects. We did not open the linked per-finding reports or rerun the tasks.