Telling coding agents to use TDD made results worse in a 26-condition eval

Luu re-ran his earlier Zstd-in-Rust agent eval with Codex (GPT-5.6 Sol). He tested 26 different instructions, ranging from "use test-driven development" to formal tools such as TLA+, Lean 4, Kani, Verus, and SMT solvers, along with four packaged testing skills. Each condition ran 80 times at medium effort and 80 times at high effort.

"Default," the condition with no testing instruction, scored above average. Agents told to use TDD wrote more tests but were more likely to fail the eval's check for a four-stream jump-table bug in Zstd. Formal-methods conditions mostly proved trivial or irrelevant properties. Verus agents, for example, confirmed that a valid index stays in bounds instead of examining the code paths that contained bugs. One packaged testing skill built on the Hegel library increased cost by 26% at medium effort and 41% at high effort without improving correctness.

"It turns out, if you ask agents to use a particular test technique or test library, this approach doesn't change as much as you'd hope," Luu writes. The exception was narrow: in the 10 of 160 fuzzing runs where agents generated structured rather than purely random inputs, about half turned up real bugs, the strongest showing of any named technique.

Why it matters: A testing technique named in a harness prompt or skill file does not ensure that the agent applies it usefully. This eval found higher costs and no correctness gain in one packaged skill, while the TDD instruction produced more tests but worse results. Teams using instructions such as "use TDD" or "use formal verification" should measure what the agent does, not whether it repeats the technique's vocabulary.