Revert rates for agent pull requests split by vendor, not by whether a human wrote the code

Kraishan followed 37,623 provenance-labelled pull requests from OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code across 2,807 GitHub repositories between December 2024 and July 2025, alongside 4,027 human pull requests drawn from the 810 repositories that also received agent work. For the 26,283 merged changes he tracked the following 90 days. Codex pull requests were reverted in 6.1% of cases against a human baseline of 11.5%, an odds ratio of 0.50 with a 95% confidence interval of 0.44 to 0.57. Devin sat the other side at 14.5%, odds ratio 1.31. Copilot, Cursor and Claude Code came out statistically indistinguishable from the human rate. Pooled across vendors, agent pull requests were less likely than human ones to contain any security smell at all, 2.9% against 4.6%, driven by fewer hardcoded credentials and fewer eval-style constructs. Claude Code is the exception in that measure, at 9.5%, roughly twice the human rate, and it carries the highest rate of hardcoded-credential patterns at 3.3% against the human 2.2%. Its pull requests are also about eight times larger than anyone else's at the median, 495 changed lines against 52 for humans, and they wait a median 12.6 hours for a first human review where the other agents wait one to four. Copilot draws the opposite treatment: 3.6 human reviews and 0.43 change requests per pull request, more than any other group. Kraishan's summary of the practical point: "Studies and procurement decisions that treat 'AI-generated code' as a single category will average away exactly the differences a maintainer cares about."

Three limits come from the author, and they set how far the numbers travel. Agents are not randomly assigned to work, so, in his words, "group differences mix code quality with task mix", and nothing in the design separates the two. The security figures come from pattern matching on diff lines rather than from confirmed exploitability, and he says they should be read as an upper bound. The corpus is public repositories with more than 100 stars in three languages, where Codex supplied 58% of the agent pull requests and Claude Code only 459 of them. The pipeline, a 63-variable codebook and the per-question statistical reports are released, so the same measurements can be re-run against later agent versions.

Why it matters: If you are routing review attention, the provenance label on a pull request now predicts something. The size of the change and which product produced it are better predictors of where review time is repaid than whether a person or an agent opened it, and pooling all agent output into one bucket hides the only differences you can act on. The paper's own read on where this is heading is one line: "Review capacity is becoming the bottleneck." The vendor rankings themselves are the part to hold loosely, because the task-mix confound is real and the author says so.