Heumann protects the checker after agents produce plausible, wrong code

Heumann says agents decompiled about 80% of a game in four weeks, enough to launch it, reach the main menu and load maps. The code had wrong function signatures, types and struct layouts, and his post says agents also invented or removed logic. His fix was a byte-level check: a reconstructed object file is compared with the original executable and debug database, and a function counts as exact only if they match. Relocations are excluded from the comparison and checked by symbol and offset instead. He also protects the check itself: CI hashes the verification script and compares it with a stored GitHub Actions secret.

After further work he reports that 99% of the game's functions are present in the reconstructed source and 83% are byte exact. The code stays private, so no one can reproduce those figures. The token count is an estimate of 600 to 700 billion, and he says lost session logs prevent an exact total. His lesson: "Correctness should be defined and machine-checkable."

Why it matters: The approach depends on having an original binary to compare against, which most software work does not. Where an external reference does exist, this shows the check needs to be outside the agent's reach. The 83% is a measure of matching, not a finding about the other 17%, and no one outside the project can inspect the code.