Spotify removed its PR-verification judge once the models got good enough

Gustavsson describes Honk, Spotify's internal system for automating code changes across a backend monorepo of more than 20 million lines. Honk originally ran every proposed change past an LLM judge, which took its success rate from roughly 20 to 30 percent up to about 80 percent. As the underlying models improved, Spotify removed the judge entirely rather than keep paying for it. Gustavsson attributes 73 percent of the company's merged pull requests to AI authorship and a PR-frequency improvement of more than 75 percent to the tooling, across an engineering org of about 2,900 people making roughly 4,500 production deployments a day. His stated precondition for all of it: Spotify had invested heavily in test automation and CI verification well before agents could be trusted to auto-merge without a human reviewing every PR.

Why it matters: This is a dated data point on a question harness builders keep guessing at: when does a verification layer stop earning its cost? Spotify's answer is not "never use a judge," it is that the judge was load-bearing until the base model closed the gap, and removing it earlier would have been the mistake.