← In the News

Meta's ML engineering agents kept a change only if it beat a noise probe

MLE Agents: Autonomous ML Engineering in Large-scale Recommendation Systems · Matthew Li, Albert Jiongqian Liang and Dilip Raj, Meta Platforms · arXiv, submitted November 18, 2026 (NeurIPS 2026 AutoMLR Workshop)

Machine-readable Download Markdown

The team built agents that propose a change to a search ranker as code, launch a 64-GPU training job, read the offline metrics and decide whether to keep it. Each iteration trained a nominal seven candidates plus a variance probe in parallel. A candidate was accepted only if it beat its same-iteration probe by at least 0.0005 normalized entropy. Over 31 days and roughly 360,000 GPU-hours, the second-generation system completed 1,097 candidate runs and kept 325. The resulting model raised search session volume by 0.23% (±0.15) in an A/B test of about a week. It also regressed a time-to-render latency guardrail; the authors note that "an agent optimizing a pure quality objective does not surface such trade-offs on its own."

The controller model, Claude Opus 4.8, was held fixed, so the authors attribute every reported improvement to harness and objective design. That design was a calibrated objective, parallel runs, and context rules: a "What Didn't Work" blocklist, one change per candidate, strict output formats, and feeding back only recent winners. Malformed edits fell from 33.8% to 0.8%, and the acceptance rate rose from 15.6% to 29.6%. The objective and search budget also changed between versions, so the authors say this is not a controlled ablation. Engineers still set the objective, steered the search (in the first version, after a 0-for-8 streak on optimizer tuning) and reviewed the final model.

Why it matters: This is the kind of gate the first item finds missing from common agent-evolution methods: a noise measurement trained in the same batch as every candidate, and an acceptance bar set from it. It is also a first-party report of an agent loop's changes reaching a live A/B test, with the latency cost stated next to the gain.