← In the News

A controlled ablation finds AGENTS.md does not move correctness on either frontier agent

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories · Prakhar Khatri, independent researcher · arXiv:2607.27250v1 [cs.SE], 28 July 2026

Machine-readable Download Markdown

Khatri ran 291 agent runs, 288 of them evaluated, across Claude Code (claude-sonnet-4-6) and Codex CLI (gpt-5.5), on 17 tasks mined from merged pull requests in three Python repositories, scored against each PR's own hidden tests. Three conditions: no context at all, the full AGENTS.md injected into the system prompt every turn, and a retrievable wiki the agent reads on demand. Pass rates barely separate. Claude: 53.3, 55.6, 55.6 percent. Codex: 58.8, 56.9, 52.9 percent. Equivalence testing bounds every pairwise difference under 10 points for Claude and 15 for Codex. A pre-registered 36-cell probe re-ran the two convention-closest near-misses on both agents, and the result is the paper's sharpest line: "across both agents the real AGENTS.md never converts a near-miss to a pass." The failure triage explains it. Near-misses fail on pattern choice and exact wiring, not on repository facts a document could supply.

One thing did move. On opshin, the one repository whose AGENTS.md warns that the test suite takes over 20 minutes, Claude's blind full-suite pytest runs fell from 3.67 per cell with no context to 2.44 to 1.67, and wall-clock time fell about 24 percent. Khatri reports that as exploratory and underpowered at n=5.

Why it matters: This is the first study to vary injection strategy across two agent families under gold-test evaluation, and its reading is narrow but usable: a generic style-and-conventions file buys you process, not correctness. The conclusion is not to delete your CLAUDE.md. It is to stop expecting it to raise your pass rate, and to put in it the things that change what the agent does, like which test command is cheap. Khatri states his own caveats: 3 Python repositories, a minimum detectable effect above 30 points, a selective arm whose corpus is not content-matched, and two model snapshots.