The same capabilities, packaged differently, change what an agent does

The experiment holds what an agent can do roughly fixed and varies only how those capabilities are organized and exposed. Six tool setups, three actor models (Qwen3Coder-30B, Kimi K2.5 and Claude Sonnet 4.5), 65 issue-resolution instances sampled from SWE-bench Live, ten independent rollouts each, 11,700 trajectories in total.

Against a bash-only baseline, a set of structured low-level primitives improves consistency across repeated attempts by up to 4.7x, and is the only setup with uniformly positive consistency deltas across all three models. The gain is largest for the weakest model, which the authors attribute to fewer malformed commands and broken edits rather than better reasoning. A natural-language search interface increases access to plausibly relevant files by more than 11%. Python CodeAct-style interfaces reach similar task performance with 41.6% fewer steps and 56.3% lower token cost.

Two lightweight text aids, a scratchpad and a hypothesis tracker, gave the agent an explicit place to record intermediate reasoning and changed its behaviour very little. The authors are careful that this is a finding about the two aids they built, not a claim about every way of structuring an agent's reasoning.

Why it matters: harness authors spend most of their effort on what tools to add. This is evidence that the packaging is a live variable in its own right, and that the cheapest wins are at opposite ends: structured primitives for reliability, a code-execution interface for cost. It also suggests that a tool whose only function is to let the agent think out loud is buying less than it looks like it is.