← In the News

Stripe's coding-agent eval: 78.1% on average, 29.4% when every required check must pass

Stripe evals: Measuring real-world coding agent performance · Kiran Prasad, Yiing Chau Mak and Manish Kurse, Stripe · Stripe Dot Dev Blog, November 19, 2026

Machine-readable Download Markdown

Stripe ran 69 integration scenarios (one-time payments, subscriptions and Connect) on 14 model and harness pairs across Claude Code, Codex and OpenCode, each with and without its own agent plugin. A second agent plays the developer. It derives a private set of two to six needs from the opening prompt and raises them one at a time as follow-up questions, so the coding agent never sees a full spec. A judge model grades each MUST or SHOULD criterion separately against the transcript and the workspace.

Scored by the share of criteria met, agents averaged 78.1%. Requiring every MUST criterion to pass drops that average to 29.4%. Stripe's example: in a fulfillment scenario, Opus 5.5 checked only PaymentIntent.status === 'succeeded', "which can allow orders to ship after refund or dispute." With the plugin, all 14 pairs scored roughly 3 to 8 points higher, web searches fell 96%, and references to outdated API parameters fell from 7% to 1.3%. The plugin is Stripe's own product, so these plugin figures come from the vendor that ships it.

Stripe lists one limit itself: "A well-shaped but non-functional integration can score full marks." The harness, Forge, is on GitHub under Apache-2.0, and its README says it does not run the agent's code after the session.

Why it matters: A partial-credit score can hide the one missing check that breaks the integration. If you grade agent output against a rubric, report the all-required pass rate next to the average. The roleplayer holding private needs is also a design you can copy for multi-turn evals.