In a vendor's benchmark, the browser tool moved agents' bug-finding results by 10% to 58%

Momentic built the benchmark, ran it and graded it, and its own testing agent, Mo, comes out on top. The suite has 20 product areas in four open-source apps (Gitea, Metabase, Twenty and Saleor) with 1,396 labeled bugs. Each agent got one area in a fresh sandbox for 30 or 60 minutes, and the figures are averages of three runs per area (one for Devin). At 60 minutes Mo matched 269.3 labeled bugs on average. Codex with GPT-6.1 Sol and Playwright MCP matched 86.3, and Claude Code with Opus 5.5 and Playwright MCP matched 59.0. Precision was 84.2% for Mo and 86.0% for Codex. Cost per matched bug was $2.67 for Mo, $0.80 for Codex and $4.09 for Claude Code.

The result with the most reach for people running their own agents is the browser tool. For Codex and for Claude Code with Opus 5.5, Playwright MCP found 10% to 58% more reference bugs than agent-browser, a command-line client. Which one was cheaper per bug depended on the model. The post's other finding is how little any agent found: the best recall was 19.3%, so a single pass left about four in five labeled bugs unfound.

The post lists its own limits. The labels were drafted from bug bashes by Mo, Claude Code and Codex, three of the agents being scored. GPT-6.1 Sol judged the matches. There are no error bars. Mo also had to produce a reproduction video and a written report for each bug, which the others did not, and Codex ran with web search on. We read the post and did not run the published harness.

Why it matters: If an agent checks your web app as part of a verification loop, the tool wiring is a variable worth testing as seriously as the model. Treat the vendor ranking as a claim and the tool-integration gap as the part to reproduce.