A talk argues that production teams should stop writing prompts and start writing measures

The author builds agents "that consumers are meant to use, to perform tasks on their behalf", which he separates from chatbots with subjective outputs. His argument is that prompt text is the wrong unit of work. Adding a prompt "is to chuck it into a completely different contextual universe than the one it was tested in"; the rest of the agent's instructions will affect it, the agent will change, and the models "might just start behaving differently all on their own". His illustration is a structured-output field instructed to stay under 80 characters, which a small fraction of requests floods with repetition until it explodes. The fix that worked was renaming the field from "title" to "heading", and he expects it to break again: "since the fix is fully deranged I expect it'll be disturbed again at some point."

What he proposes instead is a loop with a measurement at every joint. Have a model read the skill and generate adversarial and benign scenarios, express them as pass^k tests, and run the suite with and without the new skill to see whether it moves anything. Feed that suite to a prompt optimizer, genetic pareto in his example, and let it rewrite the prompt against the score. Hold back unseen tests to catch the optimizer encoding the examples it was shown. Where the assertion is itself a judgment call, brand voice being his case, the judge becomes its own optimization problem against a labeled dataset of good and bad responses. Then run those judges over sampled production conversations and turn the failures into new hard cases. On the prompt that comes out: "What's in there? Who cares! We have the measurement." The organizational claim is the sharper one. "The voice team owns the voice prompts" is the wrong pattern at scale, domain experts should be building the datasets rather than the prompt text, and "Handing someone a prompt without a measure is a form of AI psychosis." The talk reports no pass rates and no before-and-after figures; the improvements are described as moving tests "from medium-good to very-good". It stood at 11 points and 4 comments on Hacker News at 17:10 UTC, about an hour after submission.

Why it matters: The transferable piece is the ownership change, not the optimizer. If the prompt is a fitted artifact, then whoever owns brand voice or legal wording or safety phrasing owns a labeled dataset and a judge, and the handoff to the agent team is a test suite rather than a paragraph of instructions. Most teams are not staffed or reviewed that way today. Hold it loosely on the evidence: nothing here is measured in public, and the case for the loop is one practitioner's account of his own production work.