A code-review harness beats a single prompt, and the cheaper model wins

Sifry ran eight models through three ways of reviewing the same pull requests, a single engineered prompt and two open-source review harnesses (Compound Engineering and metareview), at three reasoning-effort levels. Every finding was checked against tests that prove a bug is real: 42 bugs came from the benchmark's own human labels, and 105 more were confirmed by writing a test that fails on the pull request and passes on the fix, for 147 verified bugs across 2,416 review runs. The harness beat the single prompt in 39 of 42 matched comparisons, a mean gain of 13.5 percentage points of recall. The best-value setup, an open-weight model (GLM-5.3) running metareview at low effort, matched Opus 5 running the same harness at about a thirteenth of the cost. Raising the reasoning effort from medium to high showed no measurable quality gain in 17 of 22 head-to-head comparisons, while costing more in 20 of them. Sifry discloses that he wrote metareview, one of the two harnesses tested, and notes: "A review sends your source code to whoever hosts the model, and hosts' terms differ widely," a reminder to check a provider's data terms before routing code through an open-weight host.

Why it matters: This is a concrete argument for spending on the review workflow before the model tier. A cheap harness on an open-weight model outreviewed a flagship model working alone, and turning the reasoning dial up mostly just raised the bill.