← In the News

Telling agents how to test now seems to help, on a harder task

Test/verification technique effectiveness with Astra · Dan Luu · danluu.com, November 16, 2026

Machine-readable Download Markdown

In August, Luu found that asking GPT-5.6 Sol to use a named technique such as TDD or QuickCheck mostly made results worse than giving no testing instruction. He reran the eval with GPT-6 Astra. The original task, implementing Zstd from its RFC, is now saturated: agents spent $1 at medium effort to pass every holdout test over 99% of the time, where 5.6 Sol spent $3.50 to do it about 63% of the time. On a harder task, implementing Pandoc over roughly 14 turns, he ran each condition 25 times at a list-price cost of about $90k. There, "many different techniques have a better result than Default."

He is careful about which ones. The error bars are wide, and he writes that "we don't really have enough information to say why almost anything happened." His tentative notes: differential testing led agents to examine boundary conditions closely, TDD underperformed again, the ACL2 theorem prover did well, and property-based testing libraries such as proptest finally produced tests that do what the technique promises. He also disputes a viral claim that AI-written tests are "empirically proven to be unhelpful", which he says rests on a 9% average cost saving on a benchmark of small problems whose grader penalizes added tests.

Why it matters: A testing instruction that hurt on one model and task size may help on the next, so treat a test-technique prompt or skill as something to re-measure when you change models. Luu's own testing skill outperformed with 5.6 Sol and underperformed with Astra on Pandoc.