Agents missed most planted traps in an unfamiliar language; an assumptions log helped more than effort

Kiss designed Loom, a small smart-contract language modelled on Move, with three behaviours that differ from what the syntax suggests. A signer can be stored and reused after the transaction ends. A transaction-supplied hint can override an existing interface implementation. And panic is a deferred failure that only aborts when its value is evaluated, so an unevaluated panic can stand in for a capability. Each agent got 45 minutes in a fresh container, a short spec, examples and a compiled interpreter without source, and had to design an asset interface and build a constant-product exchange. Models ran through Claude Code, Codex and Antigravity.

Across 75 reviewed runs (81 attempts, five models), he scored whether each agent correctly explained each hazard. With the baseline prompt, Opus 5.5 recognised 78% of hazards at medium effort and 56% at high; with an added instruction to keep an ASSUMPTIONS.md marking each assumption as documented, tested or unverified, it recognised all of them in every run (five runs). Sonnet 5.5 at high effort scored 22% baseline and 20% with the log. At least one hazard was recognised in 25 of 35 assumption-log runs against 17 of 40 baseline runs. The author reports that exploits adapted by a grader agent drained every contract whose agent had missed the hazard being exploited. Separately, eleven agents noticed a rounding bug in the specified pricing formula and proposed a fix; none of the 75 submissions shipped it.

The caveats are the author's own. An agent helped with the review and he verified it by hand. The samples are small, some of the log's gain may be better reporting, and "high" effort does not mean the same compute across providers. The per-run rows were not opened for this item.

Why it matters: A confident explanation from an agent is not evidence that it checked how the runtime behaves, and in this test the missed hazards were exploitable. If you hand an agent an environment it has not seen, requiring a log that separates documented, tested and unverified assumptions gives a reviewer something to check, which is where the reported gain came from.