A priced agent tournament finds the cheap setting is the expensive one

Swerdlow built a version of Brood War playable only through an agent, then ran a round robin across model and effort configurations, 18 games per configuration, saving game engine data and both agents' harness logs for each match. Codex Astra at xhigh effort went 18-0 at $10.54 per game. The same model at medium went 16-2 at $15.11, and at low went 14-4 at $21.07. Codex 5.6 Sol shows the same cost ordering, with xhigh the cheapest of its three settings at $3.23 and low the dearest at $9.23, though its win rates do not line up the same way and medium leads both. Swerdlow notes that Codex and Sonnet costs are token-based estimates rather than billed figures.

The mechanism he offers is that lower effort runs act more and think less, which in a game with a clock means longer losing games. Astra's actions per minute climb from 12.6 at xhigh to 25.7 at low as its win rate falls. The opposite failure is documented too. Of one Grok 4.6 run: "In G043, the xhigh run logged 11,138 reasoning tokens but issued only six command batches across 43 minutes and never fielded a combat unit."

A third observation is the one least tied to StarCraft. Swerdlow reports that Codex often created separate subagents to handle economy, production and army control, and that they barely talked to each other: "the army agent often sent each new unit straight into an attack, unaware of the larger army the other agents were planning to build." He adds that in games where he directed Codex himself, it was much better at getting those subagents to work together.

Two caveats belong on this. The report presents harness as a comparison axis alongside model and race, but in the Hacker News thread Swerdlow says the harness choice was not a controlled variable: "For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision". He says in the same reply that the harness detail should have been clearer and that he will add it to the report. The sample is also small, at 18 games per configuration from a single author on a single game, and Swerdlow's own first takeaway is that none of the models played beyond a beginner level. The thread carried 105 points and 62 comments when we read it on the evening of September 19.

Why it matters: Effort settings usually get argued about as quality against cost, with the low setting assumed to be the cheap one. Here it was neither cheaper nor better, because what was actually being paid for was wall clock inside a loop that kept running. Any harness with a deadline, a retry budget, or a per-run timeout can produce that shape, and a cost report that only counts tokens per call will not show it. The subagent result points at a different unmeasured cost: three agents each doing something locally sensible, with no shared picture of what the others were building.