Replit's ViBench 2.0: the best agents pass every test plan in 49 to 65 percent of builds

In ViBench 2.0 the agent starts from an empty project, builds a first version of a product from a short spec, then adds up to eighteen more features on top of its own code. The benchmark has seventeen products, 241 feature requests and 544 test plans. In Replit's account of Fireworks' official runs, GPT-6.1 Sol passed every test plan in 65% of builds, GPT-6 Astra in 60% and Opus 5.5 in 49%. Sol cost about $12 per build and Astra about $90. DeepSeek V4.1 Flash passed 77% of test plans for about $2.50 per build, but no open-weight model passed a full build.

Replit reports that nearly three-quarters of the top models' failed builds fail only on intersections between features, such as undo taking back a teammate's change. In about one in 40 of those builds the product never starts on an empty database, because schema code references a table before the statement that creates it. Grading is done by an agent running written test plans in a real browser. Replit used Opus 5.5 as the grader and says it never passed a plan with one of 34 planted, visible bugs, and that GPT-6.1 Sol confirmed 99.2% of 477 of its passes. Eight of the seventeen products and the harness are public. All figures are Replit's; the full methodology write-up is listed as coming soon, and we did not rerun the benchmark.

Why it matters: If your agents add features to code they wrote earlier, test the seams between features and a clean start on an empty database, not each feature alone. The grading setup, a browser-driving agent working from written test plans and checked against planted bugs, is a pattern you can copy into your own verification loop.