Snowflake: the task description moved pass rates more than the harness did

Snowflake built a 65-task benchmark from its own merged pull requests, graded by the tests that shipped with each change and run on its own Bazel build farm. With opus-5, four harnesses passed 84.1% to 85.1% of trials, a spread the authors say 65 tasks cannot detect. Cost per trial differed: $6.34 for Cortex Code in its REM mode, $6.51 for Claude Code, $6.93 for Cursor and $8.19 for Cortex Code in code mode. Changes that cross the Java and C++ boundary passed 55% of trials, against 80% for Java alone. For one pull request, the same hidden tests and reference fix produced an 83% pass rate on one engineer's write-up of the task and 57% on another's, across 10 agents.

The authors also report that an LLM judge's predicted pass rates "barely tracked" measured ones (Spearman +0.18 over 51 tasks), so they use the judge only to filter candidate tasks, not to rank results. Snowflake builds Cortex Code, one of the harnesses compared, and REM mode is internal only for now. We found no link to the task set in the post.

Why it matters: The post measures one pull request with two descriptions, so the 26-point gap is a single data point. Even so, it is larger than any harness gap the authors could detect, which suggests a team comparing harnesses should fix the spec wording first, then compare cost per trial.