Datadog deleted a year of AI context files. Its evals got better.

Datadog scaled Cursor and Claude Code to its roughly 4,000-engineer org in under a year, moving from a 100-to-200-person pilot to more than a thousand daily users within a month of launch. Rather than write more rules, Boudrias's team invested early in evals, anchoring correctness to real production incidents: they replayed pull requests that had caused past outages and measured whether an AI reviewer would have caught the problem before it shipped. That habit paid off when the frontend team questioned a steering document written roughly a year earlier for Sonnet 3.5. They deleted it outright and reran their evals. Performance went up, not flat. Boudrias's explanation: the document had become what the episode calls context rot, instructions a weaker model needed that now compete with more relevant context for a frontier model's attention. Datadog's own eval data was what let the team defend the deletion; without it, Boudrias said, cutting anything from an AGENTS.md file "would be pretty controversial."

Why it matters: The instinct with agent context is almost always to add more. Datadog's account is first-party evidence that the more common failure is context nobody has gotten around to deleting, and that the fix is measurable rather than a matter of taste, provided a team has evals to check the deletion against.