HumanLayer's Dex Horthy: we tried lights off in July 2025, and it failed

Horthy told the room that HumanLayer ran its own codebase fully lights off, no human reading the code, starting in July 2025, and stopped after running into an issue its most advanced prompting could not resolve (8:18). His argument is that this is not a scale problem: no amount of additional harness or loop engineering can fix it, because the fault sits in how coding models are trained. He traces it to how benchmarks like SWE-bench score a training run, a binary pass or fail on whether tests pass, which gives a model no signal to penalize poor architecture along the way (12:43 to 13:08). He names three newer benchmarks trying to close that gap, SWE Marathon from Abundant AI, Deep Suite from Data Curve, and Frontier Code from Cognition, and says plainly that no good benchmark for a model's ability to maintain codebase quality exists yet (9:47). His prescription is to put code review back: a short upfront pass on product intent, system architecture, program design, and the order of implementation, which he says can turn hours of review into something closer to reading text you already agreed on (15:00 to 17:00). Auto-generated captions throughout; paraphrased rather than quoted per this lane's transcript rules.

Why it matters: This is the first first-party account in this lane of a real "lights off" factory that failed, from the company most associated here with pushing the idea. It gives a concrete reason to keep a human reading code beyond a general discomfort with autonomy.