---
title: 'In the News: November 20, 2026 (Extra)'
description: 'Stripe tested 14 coding agents on 69 integration tasks. Their 78.1% average score falls to 29.4% when every required criterion must pass.'
canonical_url: 'https://darkfactory.dev/news/2026-11-20-extra'
markdown_url: 'https://darkfactory.dev/news/2026-11-20-extra.md'
collection: news
date_published: '2026-11-20T10:10:00-05:00'
date_modified: '2026-11-20T10:10:00-05:00'
---

# In the News: November 20, 2026 (Extra)


Stripe has published an eval of coding agents building real payment integrations, and the number to take from it is the gap between partial credit and a complete integration. Architect Labs also described how it constrained an agent swarm designing a CPU: every change was checked against a formal specification.

## 1. Stripe's coding-agent eval: 78.1% on average, 29.4% when every required check must pass

**[Stripe evals: Measuring real-world coding agent performance](https://stripe.dev/blog/stripe-integration-evals-methodology)** · Kiran Prasad, Yiing Chau Mak and Manish Kurse, Stripe · Stripe Dot Dev Blog, November 19, 2026

Stripe ran 69 integration scenarios (one-time payments, subscriptions and Connect) on 14 model and harness pairs across Claude Code, Codex and OpenCode, each with and without its own agent plugin. A second agent plays the developer. It derives a private set of two to six needs from the opening prompt and raises them one at a time as follow-up questions, so the coding agent never sees a full spec. A judge model grades each MUST or SHOULD criterion separately against the transcript and the workspace.

Scored by the share of criteria met, agents averaged 78.1%. Requiring every MUST criterion to pass drops that average to 29.4%. Stripe's example: in a fulfillment scenario, Opus 5.5 checked only `PaymentIntent.status === 'succeeded'`, "which can allow orders to ship after refund or dispute." With the plugin, all 14 pairs scored roughly 3 to 8 points higher, web searches fell 96%, and references to outdated API parameters fell from 7% to 1.3%. The plugin is Stripe's own product, so these plugin figures come from the vendor that ships it.

Stripe lists one limit itself: "A well-shaped but non-functional integration can score full marks." The harness, Forge, is on GitHub under Apache-2.0, and its README says it does not run the agent's code after the session.

**Why it matters:** A partial-credit score can hide the one missing check that breaks the integration. If you grade agent output against a rubric, report the all-required pass rate next to the average. The roleplayer holding private needs is also a design you can copy for multi-turn evals.

## 2. Architect Labs used a formal spec as the single source of truth for an agent swarm designing a CPU

**[Introducing Frostbite: A new CPU architecture discovered, designed, and verified end-to-end by AI](https://architectlabs.com/blog/frostbite)** · Architect Labs · Architect Labs blog, November 18, 2026

Architect Labs says its agents iterated over the compiler, instruction set, microarchitecture, RTL, verification and final GDS layout of a new CPU built on interaction nets. The first specification was a 141-page PDF. After the first RTL generation the team deleted it and made contracts in its in-house specification language, ASL, the only specification of the chip. The company says this "prevents a large class of bugs, silly mistakes, and reward-hacking instances when deploying large agent swarms." By the end, the ASL had lowered to more than 800,000 lines of Lean.

Swarms proposed revisions in parallel worktrees, and each change was checked against the contracts, modeled, implemented in RTL and measured for power, then kept or dropped. They tried 58 ideas and kept 37. A separate autonomous loop recalibrated the fast model against the RTL until power estimates agreed within 5%. Architect Labs reports a 4.59x performance-per-watt geometric mean across 25 workloads: 6.02x on the 10 the agents tuned on and 3.83x on the 15 held out. These are pre-silicon figures from the company itself, against a CPU it does not identify, citing legal reasons. The Hacker News thread had 353 points and 228 comments at about five hours old.

**Why it matters:** The transferable part is the method, not the chip. If agents write the spec as well as the code, a machine-checkable contract gives each change a check it can fail. Keep the 3.83x held-out figure in view rather than the 6.02x the swarm tuned on.
