Two API settings tripled a frontier model's score on a benchmark it was failing

ARC-AGI-3 asks an agent to work out how an unfamiliar 2D game works without being told. GPT-5.6 Sol scored 7.8% on it, and OpenAI went looking for why. OpenAI traced the result to ARC's harness, which discarded the model's private reasoning after every action, so it re-derived the game from scratch each turn. The harness handled context limits with a rolling truncation window at 175,000 characters, so it also lost its record of past moves. Reimplemented on OpenAI's Responses API with reasoning retained and compaction enabled, the same model went from 13.3% to 38.3% on the public task set, with roughly six times fewer output tokens. OpenAI estimates the average human tester at 48%. Bigio and Sanders: "Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting." ARC's generic harness is a deliberate design choice, on the reasoning that it makes model shortcomings visible and comparisons fair. OpenAI recommends using the Responses API rather than Chat Completions, retaining reasoning and enabling compaction.

Why it matters: This is the vendor-side confirmation of what Thomas Schranz measured from the outside this morning, and it is stronger, because it isolates two named settings and moves both score and token count in opposite directions at once. Check whether your own harness drops reasoning between tool calls, because that is a silent, expensive default. Treat any benchmark number without a harness description as uninterpretable. OpenAI has an obvious interest in a finding that its model was underrated and its own API is the fix, and every figure here is its own.