Doc2Harness scores 19 models on building a coding-agent harness; the best passes 53%

Each model is given a development document consolidated from the user documentation of twelve frontier harnesses, plus a skeleton that declares interfaces only, and has to build a complete harness. The frozen harness is then run end to end against 328 test cases, driven by a scripted mock model, with results read from requests sent, files on disk, exit code and command output. Positive controls are there to catch tests that pass because the feature never ran.

Across 19 models, each building three times, GPT-6 Astra scored highest at 53.05% and Hy3 lowest at 27.44%. Opus 5 scored 46.65%. The top two split by scenario type: Opus 5 led on ordinary scenarios, 80.68% to 72.46%, while GPT-6 Astra led on boundary scenarios, 48.97% to 37.02%. Permissions was the category that separated models most, with a 50.8-point gap between strongest and weakest.

The architecture finding is the part a practitioner can use. Controlling for code size, harnesses split into more modules scored lower (partial correlation of -0.423), and the authors say failures cluster around how cross-cutting functionality is divided. This is an association across 57 builds. It is the authors' own benchmark, still a preprint, and each build allowed only up to three submissions; the test variants derived from one seed are correlated, which the authors flag. We read the results, architecture and limitations sections but not every part of the test-construction method.

Why it matters: This is a measurement of how well current models build the kind of harness the factory depends on, and it shows that gap is large and uneven by category, permissions most of all. Treat it as a first reference point rather than a ranking, and note that the one design lesson it offers, keep cross-cutting behavior in fewer modules, is a correlation, not a tested fix.