---
title: 'In the News: September 20, 2026 (Extra)'
description: 'Across 37,623 labelled pull requests, Codex changes were reverted at half the human rate and Devin at 1.3 times it. Agents differ most from each other.'
canonical_url: 'https://darkfactory.dev/news/2026-09-20-extra'
markdown_url: 'https://darkfactory.dev/news/2026-09-20-extra.md'
collection: news
date_published: '2026-09-20T10:30:00-04:00'
date_modified: '2026-09-20T10:30:00-04:00'
---

# In the News: September 20, 2026 (Extra)


The largest published comparison of agent-written pull requests in live
repositories reports that the biggest quality gaps are between coding agents,
not between agents and people. Codex changes were reverted at about half the
human rate over 90 days and Devin changes at about 1.3 times it, measured in
the same repositories against a matched human baseline. A second item, from a
practitioner rather than a dataset, puts a name to what is throttling his own
speedup, and it is not the agent.

## 1. Revert rates for agent pull requests split by vendor, not by whether a human wrote the code

**[Not All Agents Are Equal: Code Quality and Post-Merge Maintenance Across Five Autonomous Coding Agents in the Wild](https://arxiv.org/abs/2609.17598)** · Obada Kraishan, Texas Tech University · arXiv preprint, September 12, 2026

Kraishan followed 37,623 provenance-labelled pull requests from OpenAI Codex,
Devin, GitHub Copilot, Cursor and Claude Code across 2,807 GitHub repositories
between December 2024 and July 2025, alongside 4,027 human pull requests drawn
from the 810 repositories that also received agent work. For the 26,283 merged
changes he tracked the following 90 days. Codex pull requests were reverted in
6.1% of cases against a human baseline of 11.5%, an odds ratio of 0.50 with a
95% confidence interval of 0.44 to 0.57. Devin sat the other side at 14.5%,
odds ratio 1.31. Copilot, Cursor and Claude Code came out statistically
indistinguishable from the human rate. Pooled across vendors, agent pull
requests were less likely than human ones to contain any security smell at
all, 2.9% against 4.6%, driven by fewer hardcoded credentials and fewer
eval-style constructs. Claude Code is the exception in that measure, at 9.5%,
roughly twice the human rate, and it carries the highest rate of
hardcoded-credential patterns at 3.3% against the human 2.2%. Its pull
requests are also about eight times larger than anyone else's at the median,
495 changed lines against 52 for humans, and they wait a median 12.6 hours for
a first human review where the other agents wait one to four. Copilot draws the
opposite treatment: 3.6 human reviews and 0.43 change requests per pull
request, more than any other group. Kraishan's summary of the practical point:
"Studies and procurement decisions that treat 'AI-generated code' as a single
category will average away exactly the differences a maintainer cares about."

Three limits come from the author, and they set how far the numbers travel.
Agents are not randomly assigned to work, so, in his words, "group differences
mix code quality with task mix", and nothing in the design separates the two.
The security figures come from pattern matching on diff lines rather than from
confirmed exploitability, and he says they should be read as an upper bound.
The corpus is public repositories with more than 100 stars in three languages,
where Codex supplied 58% of the agent pull requests and Claude Code only 459 of
them. The pipeline, a 63-variable codebook and the per-question statistical
reports are released, so the same measurements can be re-run against later
agent versions.

**Why it matters:** If you are routing review attention, the provenance label
on a pull request now predicts something. The size of the change and which
product produced it are better predictors of where review time is repaid than
whether a person or an agent opened it, and pooling all agent output into one
bucket hides the only differences you can act on. The paper's own read on where
this is heading is one line: "Review capacity is becoming the bottleneck." The
vendor rankings themselves are the part to hold loosely, because the task-mix
confound is real and the author says so.

## 2. A practitioner names manual testing as what caps his speedup

**[If AI coding is lowering your code quality, you're not managing quality right](https://www.i-kh.net/p/if-ai-coding-is-lowering-your-code)** · Iouri Khramtsov · Published September 14, 2026, reached Hacker News September 20

Khramtsov sets out seven defensive layers he has seen hold defect rates flat
while output rises. Six of them are ordinary quality practice with agents
attached: requirements review, unit tests at high coverage, manual testing,
end-to-end tests, code quality passes, pull request review, monitoring. The
useful part is where he says the cost sits. Manual testing is the step that has
seen the least improvement, and he calls it "the main reason that my output has
increased only 2-3x instead of something like 10x." He also finds that "coding
agents aren't great at following complex instructions in AGENTS.md or
CLAUDE.md", and works around it by running a separate pass per concern, which
he costs at 5 to 15 minutes of implementation time and no extra attention. On
review, he runs both Claude and Cursor over pull requests, says each finds
different problems, then adds a further pass "where another agent prunes the
proposed AI-generated PR comments that aren't actually meaningful."

Every throughput number in the piece is his own estimate from his own team. He
reports no instrumentation, no baseline and no measurement behind the 2-3x, and
the post gives a different multiplier in its opening than in the section that
explains it.

**Why it matters:** The pruning pass is the transferable piece here. Two review
agents that disagree produce more comments than anyone will read, so what
decides whether the second reviewer helps or just adds noise is whatever sits
downstream filtering it. His bottleneck claim points the same way as the study
above: the scarce resource is the attention needed to check the output, not the
output.