Across 40 matched setups, interventions changed speedup twice but token cost by up to 3.0 times

HPCAgent-Bench has 689 kernels for testing agents that optimize code for speed. An external judge checks correctness on hidden fuzzed inputs and times every submission, so an agent cannot report its own results. The authors say the guards were needed: with fixed inputs, one Qwen3.8-27B answer scored a 5309x speedup by caching its result, and 20x on fresh inputs.

The paper's main use is changing one part of an agent's setup at a time. The authors report that a skill packet rarely changed speed, and that on Harness20 no harness changed speedup, while OpenHands cut Qwen3.8-27B's billed tokens 2.1x and mini-SWE-agent cut OSS-120B's 3.0x. Posing a kernel as a git repository with an issue, in the SWE-bench format, changed nothing measurable. Removing the score tool lowered speedup significantly in 3 of 8 setups, and Qwen3.8-27B solved 31 kernels instead of 39 on HIP. The abstract's summary is that interventions significantly changed speedup only twice across 40 matched setups, and token cost by up to 3.0x.

Why it matters: The practical finding is about feedback. The agent that could check its candidate against a scoring tool did better than the one working blind, and the harness and skill choices mostly moved cost. If you are choosing between harnesses, measure tokens per solved task before you assume a better result. These are the authors' own results on open-weight models, in high-performance computing kernels, so the transfer to ordinary application code is untested. The benchmark code is released on GitHub; we did not run it.