← In the News

Kiro's benchmark passed while 40% of tool calls failed

Lessons learned from one million evaluations · Mac Merritt, Kiro · kiro.dev, November 17, 2026

Machine-readable Download Markdown

When Kiro added a new model, its Terminal Bench 2.1 pass rate matched expectations. Internal users still reported a worse experience. Merritt writes that "there were tools failing on 40% of their calls because the tool description that had worked for other models caused the new model to call the tools with invalid inputs." The model kept retrying until it got the call right, so the cost appeared only in tokens and latency. Kiro now scores every benchmark run in three layers: pass rate, efficiency signals such as latency, tokens and tool-failure rate, and LLM-as-judge ratings for code quality and tone. It matches the suite to aggregated internal usage across 20+ task categories and 25+ domains, and checks it against surveys and user anecdotes.

The post also describes a scenario Kiro nearly added. An internal pull request had been merged and rated 5 stars, but the next commit fixed bugs it introduced, so the benchmark would have scored buggy code as correct. The post gives no figures for the improvements it attributes to the new suite.

Why it matters: A saturated pass rate can hide a model and harness mismatch that users notice. If you swap models behind a fixed set of tool descriptions, tool-failure rate per model is a cheap number to start tracking.