A solo entrant published the harness behind a 232x kernel speedup, including the rule that unstuck it

GPU Mode ran a contest to implement batched compact-Householder QR factorization on B200 hardware. sankalp placed 12th of 183, taking the reference torch.geqrf path from roughly 419,000 microseconds to a final tracked 1,805, and made over 1,500 leaderboard submissions across 14 days doing it. He says he had known GPU kernel basics for about a year and has not worked in the domain professionally.

The harness is printed in full, including the mistake it took him days to see:

"For a long time, I did a dumb thing. I kept a single best candidate against which new candidates were tested. If the agent tries a new structural idea or a significantly big change, then it's highly likely that it would score less than our current best submission. However, after a few iterations, that change may outperform our best candidate. With this observation, I introduced some instructions to maintain a beam of 3-5 candidates."

A single-incumbent comparison kills every structural idea on its first measurement, which is exactly when a structural idea is worst. The published AGENTS.md encodes the fix as a standing rule: "Do not optimize as a single-incumbent hill climb... Do not declare a family dead after isolated singletons fail." It requires at least three live beams, one near the current best, one near-miss, one high-risk from profiling, each recording its parent candidate, hypothesis and exact changed functions.

The same file carries an evidence-class rule most harnesses do not: "Treat a timeout as inconclusive, not as a correctness/performance rejection... Only completed pass/fail/timing output is evidence." Timeouts get counted as failures by default in most loops, which silently retires whichever candidates are slowest to compile.

He states a criterion for whether a problem is loopable: an agent-usable CLI that can test, benchmark and submit, plus a checker that returns shape-wise feedback rather than pass or fail. Oversight does not stop the run. Codex's /goal runs to a quantitative objective, and one of his goals ran for over a day; /btw and /side open a side thread carrying the main context, so a running loop can be questioned without pausing it. Pressing escape, he notes, pauses the goal.

The article's own caveat matters. He attributes the escape from a plateau to the beam change and to human steering, but nothing here is a controlled comparison: this is one entrant's account of what he changed and what happened next. The externally scored parts are the placement and the timings, which the contest checker produced.

Commenters supplied the boundary the article does not. shken wrote:

"Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on."

Almondsetat reports running the same benchmark-profile-verify-research-improve loop on a video codec that ships a bitstream verifier, and getting SSE and AVX implementations that nearly doubled single-core performance in a couple of hours. Both are practitioner claims, not measured results. The thread stood at 117 points and 38 comments at three hours old, read at 10:13 EDT today.

Why it matters: the loop only pays where every step has a machine-checkable oracle, and inside that boundary the harness rules are cheap and transferable. The beam-of-candidates rule and the timeout-is-inconclusive rule can go into your own agent instructions this afternoon, and neither depends on the domain being GPU kernels.