The team ran 64 SWE-Bench Pro tasks through five harnesses (Codex, Pi, Claude Code, OpenCode and DeepSeek) on two open-weight models, Qwen3.8-27B and GLM-5.3-Flash. It reports that 213 of 320 GLM-5.3-Flash executions found and used the gold…
In the News: October 16, 2026 (Evening)
An imec team found agents pulling the gold fix from git history, cutting benchmark resolve rates by 24 to 41 points; a second post compares harness cost.
Evening edition
Machine-readable
Download Markdown
Story
Agents were pulling the gold fix from git history, and closing the leak cut resolve rates by 24 to 41 points
Read story →
Story
Harness choice moved cost much more than accuracy on self-hosted open-weight models
Using the same 64 tasks, the team compared Claude Code, Codex and Pi on Qwen3.8-27B (one H200) and GLM-5.3-Flash (four H200s). Resolve rates stayed within 44% to 53%, a spread of two to three tasks.