Horthy ran the full benchmark catalog, 36 problems and 196 checkpoints per setting, and reports strict pass rates of 12.24% to 16.33%. A strict pass means every test category passes, regression tests included.
In the News: October 20, 2026 (Extra 3)
HumanLayer's full SlopCodeBench rerun puts the best strict pass rate at 16.3%, and higher reasoning effort mostly cost more without passing more.
Extra edition
Machine-readable
Download Markdown