---
title: 'In the News: November 11, 2026 (Midday)'
description: 'A Microsoft benchmark finds the best review agent catches 71.9% of known bugs, and its authors limit lights-off review to narrow, low-consequence changes.'
canonical_url: 'https://darkfactory.dev/news/2026-11-11-midday'
markdown_url: 'https://darkfactory.dev/news/2026-11-11-midday.md'
collection: news
date_published: '2026-11-11T13:10:00-05:00'
date_modified: '2026-11-11T13:10:00-05:00'
---

# In the News: November 11, 2026 (Midday)


A Microsoft team built a benchmark from bugs that humans confirmed and fixed, then asked whether review agents would have caught them in the pull request that introduced them. The best agent catches most of them and still misses enough that the authors keep a person at the gate. The second item is a talk by one of the paper's authors on why more code has not meant faster shipping.

## 1. Review agents catch 71.9% of known bugs, and the authors still say keep a human

**[(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection](https://arxiv.org/abs/2610.13668)** · Chenyang Yang (Carnegie Mellon, work done at Microsoft Research), Hao He, Ti-Chung Cheng, Jenna Butler, Emerson Murphy-Hill (Microsoft), Carmen Badea, Christian Bird (Microsoft Research) · arXiv, October 9, 2026

CRBugBench traces confirmed bugs back to the pull request that introduced them: 1,000 review tasks from 98 open-source repositories, plus 800 from two industry monorepos at one large company. The agent reviews the change in a sandboxed copy of the repository and writes a ranked list of findings. A three-model judge panel decides whether any finding matches the known bug.

With copilot-cli as the harness, claude-opus-5 finds 71.9% of the bugs within its top five findings, up from 47.7% for opus-4.6, a gain the authors place between February and July 2026. It catches 79.6% of bugs that human reviewers flagged before merge and 64.2% of those found after merge, at about $2.97 and 9.8 minutes per task. On the proprietary repositories, performance drops by up to 25.5%, which the authors partly attribute to bugs that cannot be seen in the diff alone.

In 95.6% of runs the agent opened the file holding the defect, but only 56.2% identified the bug. Agents spent most turns reading and searching, and ran code in about 8% of them. On one Selenium task every model endorsed a buggy condition because the code comment and the PR description both described behavior the code did not implement. Asking for more comprehensive findings improved recall more cheaply than multi-agent setups, which gained 2.8% on average at 3.68 times the cost, though the gain was small or absent for opus-5. The authors' conclusion: "The present evidence therefore supports 'lights-off' operation only for narrowly scoped, low-consequence PRs."

The authors say they could not release the replication package. A public repository describes itself as a rebuild of the benchmark from the paper's descriptions, with 995 open-source tasks.

**Why it matters:** This is a direct measurement of the step a lights-off factory depends on: a review agent standing in for the human reviewer. On these numbers, the agent works as a second reviewer: it catches bugs people miss, but it misses more than a quarter of known ones and accepts misleading PR descriptions at face value. Teams that give it repository context, size review effort to the PR, and re-test on their own code when the model changes will get more out of it than teams that trust an aggregate score.

## 2. Christian Bird on why more code has not meant faster shipping

**[Why aren't we shipping faster now?](https://cabird.com/talks/shipping-faster/)** · Christian Bird, Dryden Technology Group · UC Irvine Software Engineering Seminar, October 30, 2026

Bird's 21-minute talk argues that AI tools raised the volume of work started without a matching rise in what ships. We read the 19-page handout, not the recording. One slide, citing an NBER working paper by Demirer, Musolff and Yang covering more than 500,000 GitHub developers using autonomous coding agents, sets a 240% rise in commits against a 30% rise in actual releases.

Much of the talk draws on a 30-interview study of engineers, managers and PMs on long-lived products, which the handout lists as in submission. One participant says: "If you can do the code in one day, you're still talking 30 days before it actually gets deployed." A senior engineer says: "The pace at which we have to review is fast outpacing our ability to review." The talk names the result "Systemic Conversion Strain": the cost of generating code fell while the cost of shipping it barely moved, and expectations treated the two as the same. The handout closes with three recommendations: "scale verification with generation", "count outcomes, not activity" and "protect the people who ship things".

**Why it matters:** If your factory reports commits or merged PRs, it is reporting input. Bird's case is that review, integration and deployment set the pace. Review is the step the paper above tests handing to an agent. The interview study has not been published, so its quotes should be read as early findings shown in a talk.