---
title: 'In the News: October 25, 2026 (Extra 2)'
description: 'A hands-on test finds coding agents miss most traps in an unfamiliar language, and that an assumptions log helps more than higher effort.'
canonical_url: 'https://darkfactory.dev/news/2026-10-25-extra-2'
markdown_url: 'https://darkfactory.dev/news/2026-10-25-extra-2.md'
collection: news
date_published: '2026-10-25T16:20:00-04:00'
date_modified: '2026-10-25T16:20:00-04:00'
---

# In the News: October 25, 2026 (Extra 2)


One item this afternoon: an author-run test of coding agents on an invented language with three planted traps. Most runs missed most of them, and asking the agent to log its assumptions did more than raising the effort setting.

## 1. Agents missed most planted traps in an unfamiliar language; an assumptions log helped more than effort

**[It's a trap: testing coding agents on a language with surprising semantics](https://blog.csongor.co.uk/loom/)** · Csongor Kiss · personal blog, October 25, 2026

Kiss designed Loom, a small smart-contract language modelled on Move, with three behaviours that differ from what the syntax suggests. A signer can be stored and reused after the transaction ends. A transaction-supplied hint can override an existing interface implementation. And `panic` is a deferred failure that only aborts when its value is evaluated, so an unevaluated panic can stand in for a capability. Each agent got 45 minutes in a fresh container, a short spec, examples and a compiled interpreter without source, and had to design an asset interface and build a constant-product exchange. Models ran through Claude Code, Codex and Antigravity.

Across 75 reviewed runs (81 attempts, five models), he scored whether each agent correctly explained each hazard. With the baseline prompt, Opus 5.5 recognised 78% of hazards at medium effort and 56% at high; with an added instruction to keep an `ASSUMPTIONS.md` marking each assumption as documented, tested or unverified, it recognised all of them in every run (five runs). Sonnet 5.5 at high effort scored 22% baseline and 20% with the log. At least one hazard was recognised in 25 of 35 assumption-log runs against 17 of 40 baseline runs. The author reports that exploits adapted by a grader agent drained every contract whose agent had missed the hazard being exploited. Separately, eleven agents noticed a rounding bug in the specified pricing formula and proposed a fix; none of the 75 submissions shipped it.

The caveats are the author's own. An agent helped with the review and he verified it by hand. The samples are small, some of the log's gain may be better reporting, and "high" effort does not mean the same compute across providers. The per-run rows were not opened for this item.

**Why it matters:** A confident explanation from an agent is not evidence that it checked how the runtime behaves, and in this test the missed hazards were exploitable. If you hand an agent an environment it has not seen, requiring a log that separates documented, tested and unverified assumptions gives a reviewer something to check, which is where the reported gain came from.