Review agents catch 71.9% of known bugs, and the authors still say keep a human

CRBugBench traces confirmed bugs back to the pull request that introduced them: 1,000 review tasks from 98 open-source repositories, plus 800 from two industry monorepos at one large company. The agent reviews the change in a sandboxed copy of the repository and writes a ranked list of findings. A three-model judge panel decides whether any finding matches the known bug.

With copilot-cli as the harness, claude-opus-5 finds 71.9% of the bugs within its top five findings, up from 47.7% for opus-4.6, a gain the authors place between February and July 2026. It catches 79.6% of bugs that human reviewers flagged before merge and 64.2% of those found after merge, at about $2.97 and 9.8 minutes per task. On the proprietary repositories, performance drops by up to 25.5%, which the authors partly attribute to bugs that cannot be seen in the diff alone.

In 95.6% of runs the agent opened the file holding the defect, but only 56.2% identified the bug. Agents spent most turns reading and searching, and ran code in about 8% of them. On one Selenium task every model endorsed a buggy condition because the code comment and the PR description both described behavior the code did not implement. Asking for more comprehensive findings improved recall more cheaply than multi-agent setups, which gained 2.8% on average at 3.68 times the cost, though the gain was small or absent for opus-5. The authors' conclusion: "The present evidence therefore supports 'lights-off' operation only for narrowly scoped, low-consequence PRs."

The authors say they could not release the replication package. A public repository describes itself as a rebuild of the benchmark from the paper's descriptions, with 995 open-source tasks.

Why it matters: This is a direct measurement of the step a lights-off factory depends on: a review agent standing in for the human reviewer. On these numbers, the agent works as a second reviewer: it catches bugs people miss, but it misses more than a quarter of known ones and accepts misleading PR descriptions at face value. Teams that give it repository context, size review effort to the PR, and re-test on their own code when the model changes will get more out of it than teams that trust an aggregate score.