The same author asks who is testing your quality gates

The follow-up, and it names this publication's subject directly. Gorman generalises mutation testing, which tests a test suite by injecting defects, into a class he calls Brown M&M tests, after the Van Halen rider clause used to check whether a venue had read the contract. He extends it past tests to every gate in the pipeline: "How are you testing your linter rules? I might deliberately inject unused imports into random source files to see if my automated code review finds them all." Then, in a parenthesis: "(Now, go on, admit it, 'dark factory' folks, this probably never occurred to you, did it?)"

His closing claim is about people rather than pipelines, and he offers it from "experiment and research going back 3+ years" without publishing the evidence: that confidence in LLM-generated code "has more to do with whether or not they see the 'brown M&Ms in the bowl' than the actual code's quality."

Why it matters: An untested gate and a gate that does not work are indistinguishable from the outside, and a lights-out pipeline is where nobody looks. Injecting a known unused import, a known race and a known nonsense identifier, then confirming the pipeline catches each one, tells you whether your verification loop is a loop or a decoration.