Vaccari proposes keeping the tests an agent must pass in data files, apart from test code

Vaccari starts from interviews he ran in early 2026 with 26 Thoughtworks developers at an organization that was heavily invested in coding agents. He asked whether they trusted their tests. He reports that they said they did not trust the tests much. The reason they gave was that the agent refactors existing tests along with the code as features are added, so a test someone reviewed once may no longer check the same behaviour.

His proposal is to write tests in the customer's terms, run them against the system from the outside, and keep the cases in text files: a YAML list or a CSV table of inputs and expected outputs, or a small custom language for cases that need setup and state. An agent writes the runner that executes those files. Because the cases live apart from test code, a team can make them read-only during a task that should not change the specification, or have CI check which files a task touched. Vaccari includes a table of what each kind of task should change. A refactoring should change production code and leave both the test data and the runner alone.

He names the gaps himself. The runner is code and an agent could change it. Harness hooks give fast feedback but are "not bullet-proof". He points to Ivett Ördög's Constrained Tests pattern, in which the test language makes the assertion a required part of the syntax so an agent cannot write a test that asserts nothing. To check that the cases themselves are right he suggests mutation testing and an oracle. For a COBOL interpreter he built, every customer test had to pass the same way on a real IBM i system as on his laptop.

The article reports no measurement of the approach. The defect figures in its introduction come from three industry reports (Faros AI, DX and CloudBees), and Vaccari says he does not know for certain what is causing the decline they describe.

Why it matters: A change to a table of cases is a short diff, and a CI rule can reject a refactoring task that touches the table. That gives a team a mechanical check on the failure Vaccari's interviewees described, an agent rewriting the tests it has to pass. Whether teams that adopt it catch more defects is not something the article shows.