Slack measured agent-driven end-to-end tests: reliable, slow and expensive

The team ran 20 executions of each configuration on two Slack workflows, a thread reply (about 15 to 20 steps) and a search flow (about 25 to 30 steps), in test workspaces with non-production data. The agent drove the browser through Playwright MCP or Playwright CLI, or generated a Playwright test and iterated on it. The post reports failure rates of 0% and about 12% for MCP on the two flows, about 12% and 20% for CLI, and about 8% and 48% for generated tests. Average runtimes were 5 to 8 minutes for MCP, 9 to 11 for CLI, and about 3 for generated tests. The authors put agent runs at $15 to $30 each.

Cost followed turn count and context growth. On the search flow, MCP took about 40 turns and 3.5 to 3.8 million tokens on Sonnet 4.5 and Opus 4.6, against about 85 turns and 6 million tokens for CLI on Opus 4.6. The post says most of the spend was previously seen content sent again each turn, and that most CLI failures came from authentication and navigation problems, not model reasoning. Only about 20% of runs followed the same action sequence, though runs that passed reached the same end state. In the authors' words: "Tests enforce journeys. Agents verify goals."

The limits are the authors' own. The runs used Sonnet 4.5 and Opus 4.6, the token use was not optimized, and the explanations for why MCP beat CLI are described as likely factors that were not measured. Slack did not publish the harness or the test definitions, so the numbers cannot be rerun from the post.

Why it matters: If you run agents as verifiers, this is first-party data that the execution layer and turn count drive cost and flakiness more than model choice. Slack's own conclusion is to keep deterministic tests in CI and use agents for exploration and debugging, which is a defensible default until your own per-run cost and failure rate say otherwise.