Factory says its harness, which runs tool calls as programs, scored higher at lower cost on Terminal-Bench 4

Factory made programmatic tool calling the default in Droid 1.0. The model writes JavaScript that runs in a sandbox with no file, network or system access of its own, and one model request can trigger hundreds of tool calls. Raw tool output stays inside the program, and the model reads only what the program prints. In Factory's Terminal-Bench 4 runs (October 23 to 28, three trials per task, compared only on tasks both harnesses passed in all three trials), Droid with Claude Opus 5.5 scored 65.2% at $4.73 per passed task, against 57.6% for Claude Code. With GPT-6.1 Sol it scored 57.6% at $0.79, against 54.0% for Codex. Factory says script mode cuts Droid's cost by 15% on Opus 5.5.

The post's worked example is a Linear cleanup of 151 issues: 312 tool calls, 2 model requests and about 61,000 tokens. Factory estimates direct calls would have taken 36 requests and about 3.1M input tokens, modeled at 10 calls per request. It also reports that a typical production request reads 98% of its input from cache, across 20.4M Droid CLI requests in the seven days to October 22, after Droid began appending newly discovered tools to the end of the request instead of inserting them into the tool list.

Factory built the harness, chose the settings and ran the comparison, and we did not rerun it. Some trials were rerun after provider failures, and moderated Opus 5.5 trials were rerun on Opus 5. Devin tasks where it stopped to ask the user were counted as passed if Codex passed them. The post also says Droid takes 10% longer than Codex with GPT-6.1 Sol.

Why it matters: If you run your own agent loop, the claim to test is narrow: how much raw tool output you paste back into context, and whether your request prefix stays stable between turns. Both can be measured on your own sessions before you accept a vendor ranking.