A controlled test isolates the harness from the model, and the harness wins

Lewis built a coding-agent harness called Yuj and ran it two ways against the same tasks and the same frozen model weights: a control that feeds the model the full chronological transcript, and a treatment that trims older tool results as context fills and intervenes when the model repeats a failing command. On the tight-context SWE-bench Verified comparison (169 tasks, 20,480-token window), mean per-task fail-to-pass rate rose from 28 percent under control to 49 percent under treatment, and complete solutions rose from 43 to 72, both differences significant at p < 0.0001. Applied without retuning to three more model designs, the same frozen treatment raised results on every one of them; Devstral's complete solutions more than doubled, from 22 to 53. When context was effectively unconstrained rather than tight, the gap mostly closed on two of the three benchmarks tested. Raw per-task outcomes are posted in a public repository, so the reported numbers can be checked against the underlying records rather than taken on trust.

Why it matters: Lewis states the conclusion plainly: "the tested solver is the model and harness together." Two evaluations that name the same model can be measuring different solvers if their harness configurations differ. Anyone comparing coding agents, or shipping one, needs to treat context management as part of the tested system and version it accordingly, not leave it on defaults.