Epoch's leaderboard puts a price on autonomous project size

MirrorCode asks a model to reimplement a whole program from a compiled binary, documentation and example inputs and outputs, with no internet, no original source, and held-out end-to-end tests. Twenty-five targets include Unix utilities, interpreters and compression software. The scale figures are first-party: Claude Opus 4.7 reimplemented gotree, a bioinformatics toolkit of roughly 16,000 lines of Go with 40 or more commands, in 14 hours for $251, against Epoch's estimate that an unassisted human engineer would need 2 to 17 weeks. One of the largest tasks "cost $2,600 for a single run and involved AI working for 19 days without human intervention." The leaderboard, which Epoch says it updates regularly, reports Claude Fable 5 at 64 percent, GPT-5.6 Sol at 20, GPT-5.4 at 16 and GPT-5.5 at 10, on 15 medium and large programs in two languages, three runs each, 10 billion tokens per attempt. Epoch's footnote says those scores are not directly comparable with the paper's, and the page carries a June date while the leaderboard now contains a model that postdates it, so the update is real but undated. Epoch states the contamination risk itself: targets are open-source programs the models likely saw in pretraining, and "we cannot rule out the possibility that memorization contributes to AI performance." The scaffold and 22 of the 25 targets are released.

Why it matters: Previous dark-factory claims in this feed have come from first-person accounts or vendor theses. Epoch's measurements put a bound on unaided project scale and attach a dollar figure to it. A 64 to 20 percent spread between the top two models on week-scale work is far wider than short-task benchmarks report, and it is the number to argue with.