A validator that never talks to the implementer closes most of a benchmark's parity gap

Factory.ai split the job into two roles: an implementer that writes the code, and a validator that builds a hidden test instrument before implementation starts and shares only its findings, never the instrument itself, across a wall between the two. Asked to rebuild the GDAL geospatial toolkit, roughly 600,000 reachable lines, from scratch with no access to its source or tests, a single agent (Fable 5) reached 36 percent behavioral parity and reported itself finished. The three-role system reached 90 percent on the same task. Factory reports a similar closing of the gap across 24 of its hardest benchmark tasks. The system used 14 times the credits and 13 times the wall-clock time of the single agent on the GDAL task. The figures are Factory's own, measured on its own benchmark and model panel, with no outside replication yet.

Why it matters: Factory attributes early stopping to agents building their own undersized definition of "done." That makes verification-loop design the variable to test, and teams should weigh the reported gain against the extra compute cost before adopting the pattern.