A 10,000-line skill made a coding agent worse, and nobody noticed until it was measured

Nisi generated a 10,000-line skills library from WorkOS's documentation, with each skill hashed to its source section so it would only regenerate on a real doc change. Evals of that library took 68 minutes to run per pass, per the transcript, and produced worse results than giving the agent no skill at all: loading the skill got a task right 77% of the time, against 97% without it. He rewrote it down to 553 lines covering only the common failure modes his evals actually surfaced; the eval suite dropped to 6 minutes a run and accuracy recovered. He credits catching the regression entirely to measuring it, rather than assuming more documentation helps: "eval are super important when you're working with this non-deterministic code."

Why it matters: it is a measured, not asserted, case against comprehensiveness in skill files. If a skill hasn't been run against evals before and after, there's no way to know it isn't actively hurting the agent it's meant to help.