A cheap, non-blocking check on tool results roughly doubled completion on failure-heavy benchmarks

The paper's framing of the problem is blunt: "A cached error page or negative price can instead arrive in the expected format and be consumed as fact." It cites a documented production incident in which 70% of a runtime's silent failures were caught only by a human reading the output. Their fix, Outcome Monitors, checks a tool's result against contracts mined from clean historical traces or derived from public API schemas. On a violation it does not block the action or attempt a repair. It appends a nonbinding receipt naming what looks wrong and which recovery tools are available, and leaves the decision to the agent.

In frozen, prespecified tests across four models in two provider families, the receipt raised full-task completion on the ToolMaze benchmark from 10.9% to 28.1%, a gain that held up on a separately run third model family (MiniMax M3, plus 18.75 points). In tau-bench's retail environment, completion rose 14.0 and 12.0 points on two model tiers. Matched control studies pin the gain to one specific piece of the message: stripping the list of recovery tools out of the receipt drops the benefit back to baseline, and restoring it restores the gain. The receipt adds $0.13 per episode over baseline cost. A review-loop alternative the authors also tested, where a second model call inspects and revises each tool call before it runs, adds $9.24, for a smaller and not statistically significant improvement.

The gains are not universal. They concentrate where a fault would otherwise block the task outright. On a set of faults modeled on a published taxonomy of real production incidents, written without reference to the paper's own contract vocabulary, detection recall fell to roughly 46%, and a held-out test on the AppWorld benchmark showed no measured net benefit. The authors are explicit that a deployment should weigh its own fault rate and false-positive rate before counting on the effect they measured.

Why it matters: This is a specific, priced design choice for anyone building a tool-calling harness: a non-blocking receipt on tool results, layered on top of whatever error handling already exists, versus a second model call reviewing every result. The paper's own numbers say the cheap option worked better in their tests, not that it catches everything.