← In the News

Self-improving agents pick the better skill edit about half the time

The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements · Yuanhao Zeng, Kan Ren and nine co-authors, ShanghaiTech University, ByteDance, BIGAI and SUSTech · arXiv, submitted November 17, 2026

Machine-readable Download Markdown

The authors ran seven agent-evolution methods (prompt, skill, memory and harness evolvers, including GEPA and Meta-Harness) on six models: 1,030 independent evolution runs and over 2.1 million test trajectories. On a pool of 100 candidate skills scored across 11 Terminal-Bench tasks, selection on execution scores preferred the better skill 49.7% of the time. Regressions were 30.5% of the pool but 62.1% of what got accepted. LLM judges reading the full trajectory did no better. Good skills were there to find: in a separate pool, the strongest shared skill scored 57.5% against a pool mean of 49.2%. In re-evaluated SkillOpt-Lite and GEPA runs on Terminal-Bench, the evolvers' own validation scores rose 6.9 to 9.0 percentage points while independent re-evaluation put the gain between +0.4 and -3.2. In the authors' words, "In-loop scores improve under this process; agent capability does not."

The largest gains reported elsewhere did not survive an audit. Across 27 model and benchmark pairs with a reported gain above 3 points, adding the single most responsible rule to the unevolved agent recovered a median 90% of the gain. That rule typically exploited an evaluation shortcut, such as a grader that credits only one answer format. Repairing those evaluations cut the median gain from 17.6 to 0.8 points. Across 171 results in nine papers, the median margin over the strongest competitor was 3.4 points, and none of those results reported a variance estimate. At that margin, Terminal-Bench needs about 23 runs per method to tell two methods apart at 80% power. The trajectory data is not yet public; the authors say it will be released on acceptance.

Why it matters: If an agent edits its own skills or harness and keeps whatever scored higher on one rollout, this paper's evidence says much of what you keep is noise. Score candidates on repeated rollouts, report the run-to-run spread, and check whether a gain traces to one rule that fits the grader. In a preliminary analysis, the authors found checks on intermediate behavior carried 13 to 57 times less evaluation noise than pass/fail verdicts.