A new DevOps benchmark shows agents declaring a cleanup done after an incomplete survey

CloudOpsBench v1.0 has 45 AWS tasks, including decommissioning projects, securing access, migrating services and repairing failures. The post says Claude Fable 5.1 completed 29 of the 45 and GPT-6 Astra completed 26. Across the failed tasks it classes about 80% as reasoning gaps and about 20% as wrong actions, and says no task failed because an agent misunderstood a service.

Its worked example is a task to delete every queue belonging to a project. In the Fable 5.1 run shown, the agent listed SQS queues across 12 regions, deleted two, re-checked, and reported none remained. The post says no attempt by any model listed MediaConvert queues, and the independent verifier failed the run because a project queue was still there. In a bucket-migration task, the post says agents aborted an in-progress multipart upload as leftover data, which destroyed it unrecoverably.

We read the blog post and the leaderboard page. We did not open the task set or the underlying data, and the post does not name its authors.

Why it matters: In the example, the agent's re-check searched the same places as its first pass, so it could not find what the first pass missed. A check that lists every resource type in scope, run separately from the agent's own search, would have caught this one.