A review agent's misses were mostly doubts it talked itself out of

Nogueira describes Hopper, a review agent his team built in house. It reviews every ready pull request in their main repository, and a person still approves every merge. When the team looked at the defects she missed, about half were things she had noticed in her own reasoning and then dismissed as minor or already handled: 13 of 24 in the sample they checked. A prompt telling her to finish what she finds changed nothing. What worked was passing those set-aside doubts, with the file and her reason for dropping each, to a second reviewer starting from a fresh context. Of the doubts that survived that second look, 27 of 30 turned out to be real.

The second reviewers run on two model families with the same prompt, so only the model differs. On 19 replayed cases he reports that the GPT-6 Astra reviewer found 16 real defects the Claude Opus 5.5 reviewer missed, and the Claude reviewer found 4 that the GPT reviewer missed. Neither raised a correctness finding that proved wrong. The GPT reviewer overstated severity on 6 of its 21 blocking findings, against 0 of 6 for the Claude reviewer. The team takes the union of both reviewers' findings and lets the lower severity stand unless Hopper verified the higher one.

Two measurement and control details are worth separate attention. First, replaying old pull requests leaks the future: runs that could see a later fix commit or an edited description found 68% of defects, and clean runs found 44%, so the team now checks every run for leaks and scores them separately. Second, Hopper never approves, and that is enforced as a tool permission on her credentials, not as a line in a prompt. The post also says a guard that refuses any review whose counts do not match its inline comments has caught 5 real slips in 181 reviews.

The numbers are self-reported from small samples, and Hopper is not available to inspect. The post's account of Anthropic's own review practice is Nogueira's summary of a separate Anthropic post, which we did not read for this item.

Why it matters: If your review agent has a "noticed it, then dismissed it" failure, the cheapest fix reported here is structural: log what the reviewer set aside and route those items to a fresh reviewer, ideally on a different model. The leak finding applies to anyone scoring a reviewer on replayed history, because a recall number measured with the fix visible overstates what the reviewer will catch on a live change.