Study of 6,541 agent commits finds context-file rules are followed every time or never

The authors collected 621 distinct versions of agent context files (AGENTS.md, CLAUDE.md, copilot-instructions.md) from 268 public GitHub repositories and linked each version to the commits an agent made while it was in force: 6,541 commits attributed to Copilot, Claude Code or OpenAI Codex. A local model extracted the instructions. A second model wrote a checking script for each one, and the script ran against the files each commit touched. Of 14,902 candidate instructions, 2,053 were checked against at least one real commit.

Mean compliance per instruction was 59.8%, but few instructions sit near that figure. 1,116 of them (54.4%) were followed in every applicable commit, 766 (37.3%) in none, and 171 (8.3%) somewhere in between. The authors write that context-file governance behaves "less like a dial agents turn to a partial setting, and more like a switch." The split holds when they drop instructions with fewer than three or five applicable commits.

Code-level implementation rules were followed more often than testing rules (mean 63.6% against 49.2%), the only category difference that held at every evidence threshold. Rules about development process, such as commit formatting and branching, had the lowest mean among categories with enough support, 34.0%, but that gap did not survive the stricter thresholds. Copilot's commits followed their context files at a mean rate of 66.0%, against 52.2% for Claude Code, a gap that held under the same check. The authors call it an association, because they did not control for what each agent's files ask for.

The paper is open about its limits. The median instruction was checked against three commits, and 540 were checked against one. When the authors inspected 94 of the generated checking scripts by hand, 67 (71.3%) implemented their instruction correctly. Scripts saw whole files, not diffs, so a violation that predates the agent can count against it. Many instructions could not be measured at all: "run the linter" leaves nothing in the commit to check. The paper links a replication package; the link returned a 404 page when we tried it on November 12.

Why it matters: If a rule in your AGENTS.md belongs to the never-followed group, reading a few compliant commits will not show it. The authors argue for automated per-rule checks that run continuously, in place of spot checks. Their own numbers add a condition: about three in ten of their generated checks were wrong, so each check needs a person to read it before it gates anything.