A controlled study tests planning, tools, and context management

The authors built a coding harness from scratch around a fixed ReAct execution loop, then varied three components independently: an explicit planning step, the action interface (a curated tool set versus a bash-only shell), and context management (five strategies, from no compaction through staged rule-based elision followed by summarization). They ran 176 matched configurations across four models, three sizes of Nemotron-3 plus Mistral-Medium-3.5, on SWE-Bench Verified and Terminal-Bench 2.1. Context management's accuracy gains, they report, come almost entirely from stopping agents from running out of context and quitting early, not from helping them reason better within a normal-sized budget, and a mechanism the team built to make trimmed context recoverable later went largely unused. Planning's value flips with model capability: it works as an accuracy scaffold for the weakest model tested and as a cost-cutting check for the strongest, and predefined tools help models with weaker command-line skills while a bash-only interface lets capable models batch more work per turn and spend less. The paper states its aim directly: "Are harness components generally useful across settings...?"

Why it matters: Harness choices should be re-evaluated when the underlying model changes because planning and tool-interface effects shifted across the models tested. For agents that fail on long tasks, preventing context overflow may matter more than adding a memory system.