---
title: 'In the News: November 12, 2026 (Extra)'
description: 'A study of 6,541 agent commits finds context-file rules are followed every time or never, so spot checks miss the ignored ones.'
canonical_url: 'https://darkfactory.dev/news/2026-11-12-extra'
markdown_url: 'https://darkfactory.dev/news/2026-11-12-extra.md'
collection: news
date_published: '2026-11-12T10:10:00-05:00'
date_modified: '2026-11-12T10:10:00-05:00'
---

# In the News: November 12, 2026 (Extra)


A new empirical study checked whether coding agents follow the rules written in AGENTS.md and similar files. Most rules it could measure were followed in every commit or in none, and more than a third were in the second group. Separately, Matteo Vaccari of Thoughtworks describes a way to keep the tests an agent must pass in data files the agent has no reason to edit.

## 1. Study of 6,541 agent commits finds context-file rules are followed every time or never

**[Do They Do as They Are Told? Investigating Coding Agents' Adherence to Explicit Instructions in Context Files](https://arxiv.org/abs/2611.13709)** · Gianmario Voria, Gemma Catolino and Fabio Palomba, University of Salerno; Yutaro Kashiwa, Nara Institute of Science and Technology · arXiv, submitted November 10, 2026

The authors collected 621 distinct versions of agent context files (AGENTS.md, CLAUDE.md, copilot-instructions.md) from 268 public GitHub repositories and linked each version to the commits an agent made while it was in force: 6,541 commits attributed to Copilot, Claude Code or OpenAI Codex. A local model extracted the instructions. A second model wrote a checking script for each one, and the script ran against the files each commit touched. Of 14,902 candidate instructions, 2,053 were checked against at least one real commit.

Mean compliance per instruction was 59.8%, but few instructions sit near that figure. 1,116 of them (54.4%) were followed in every applicable commit, 766 (37.3%) in none, and 171 (8.3%) somewhere in between. The authors write that context-file governance behaves "less like a dial agents turn to a partial setting, and more like a switch." The split holds when they drop instructions with fewer than three or five applicable commits.

Code-level implementation rules were followed more often than testing rules (mean 63.6% against 49.2%), the only category difference that held at every evidence threshold. Rules about development process, such as commit formatting and branching, had the lowest mean among categories with enough support, 34.0%, but that gap did not survive the stricter thresholds. Copilot's commits followed their context files at a mean rate of 66.0%, against 52.2% for Claude Code, a gap that held under the same check. The authors call it an association, because they did not control for what each agent's files ask for.

The paper is open about its limits. The median instruction was checked against three commits, and 540 were checked against one. When the authors inspected 94 of the generated checking scripts by hand, 67 (71.3%) implemented their instruction correctly. Scripts saw whole files, not diffs, so a violation that predates the agent can count against it. Many instructions could not be measured at all: "run the linter" leaves nothing in the commit to check. The paper links a replication package; the link returned a 404 page when we tried it on November 12.

**Why it matters:** If a rule in your AGENTS.md belongs to the never-followed group, reading a few compliant commits will not show it. The authors argue for automated per-rule checks that run continuously, in place of spot checks. Their own numbers add a condition: about three in ten of their generated checks were wrong, so each check needs a person to read it before it gates anything.

## 2. Vaccari proposes keeping the tests an agent must pass in data files, apart from test code

**[Tests as data: ensuring AI-generated code does what we want](https://martinfowler.com/articles/tests-as-data.html)** · Matteo Vaccari, Thoughtworks Italia · martinfowler.com, November 11, 2026

Vaccari starts from interviews he ran in early 2026 with 26 Thoughtworks developers at an organization that was heavily invested in coding agents. He asked whether they trusted their tests. He reports that they said they did not trust the tests much. The reason they gave was that the agent refactors existing tests along with the code as features are added, so a test someone reviewed once may no longer check the same behaviour.

His proposal is to write tests in the customer's terms, run them against the system from the outside, and keep the cases in text files: a YAML list or a CSV table of inputs and expected outputs, or a small custom language for cases that need setup and state. An agent writes the runner that executes those files. Because the cases live apart from test code, a team can make them read-only during a task that should not change the specification, or have CI check which files a task touched. Vaccari includes a table of what each kind of task should change. A refactoring should change production code and leave both the test data and the runner alone.

He names the gaps himself. The runner is code and an agent could change it. Harness hooks give fast feedback but are "not bullet-proof". He points to Ivett Ördög's Constrained Tests pattern, in which the test language makes the assertion a required part of the syntax so an agent cannot write a test that asserts nothing. To check that the cases themselves are right he suggests mutation testing and an oracle. For a COBOL interpreter he built, every customer test had to pass the same way on a real IBM i system as on his laptop.

The article reports no measurement of the approach. The defect figures in its introduction come from three industry reports (Faros AI, DX and CloudBees), and Vaccari says he does not know for certain what is causing the decline they describe.

**Why it matters:** A change to a table of cases is a short diff, and a CI rule can reject a refactoring task that touches the table. That gives a team a mechanical check on the failure Vaccari's interviewees described, an agent rewriting the tests it has to pass. Whether teams that adopt it catch more defects is not something the article shows.