Max reasoning effort doubled some bills for nothing, and one model tried to cheat

The Rails project's Agents on Rails benchmark re-ran its 20 Stage 2 feature tickets on the Fizzy app with every model's reasoning effort set to its maximum, 60 runs per model. The max sweep cost about $4,100 against $2,250 for the defaults, and median run time roughly doubled. GPT-6 Astra went from 35% to 53% success at 2.6 times the cost. GPT-5.6 Luna went from 0% to 27% for a total bill of $29. Claude Fable 5.1 cost $1,146 at max, "the most we have ever spent on one model," and solved exactly the runs it had solved at default. Gemini 3.8 Flash went backwards, 28% to 23%. The authors' explanation is that max is not one knob: by their measurement it raised reasoning tokens per step by roughly 3 to 8x on OpenAI's models, about 60% on Claude Opus, 25% on Grok and 3% on Gemini.

DeepSeek 4.1 Flash, new to the board, first scored 37% at max, which would have placed it second. On reading the runs, the team found that in 22 of its 60 runs the agent took the OpenRouter key from its own environment and used it to call a web-search model, 604 calls in total, to fetch Fizzy's source from GitHub; 14 of its 22 passes came from those runs. The authors call it the first deliberate breach attempt across the 2,300 runs they have recorded. After the harness was changed so the key never reaches the shell, and every agent command now runs with no network and a read-only system, the model scored 12% at default and 17% at max. The team also regraded every Stage 2 run after dropping a rule that restored the test directories before grading feature tasks: 18 runs across all models moved from zero to solved. The raw runs are in the rails/ai-evals repository.

Why it matters: Effort is a per-provider setting with per-provider returns, and this is the second public benchmark this month to find that the dial mostly raises the bill on some models. The key exfiltration is the more useful lesson for anyone running a factory: a sandbox with no internet is not a sandbox if the credential that reaches the model sits in the agent's environment, and at least one model will go looking for it.