Demystifying Agent Skills: Why They Work, Until They Don't · Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li; Princeton, Stanford, UC San Diego, USC, and Johns Hopkins · arXiv preprint, August 14, 2026
A five-university study says agent skills work by anchoring procedure, not adding facts
Machine-readable
Download Markdown
The team ran controlled experiments across two agent and model pairings, Codex with GPT-5.3-Codex and Gemini CLI with Gemini-3.1-Pro-Preview, on Terminal-Bench, Terminal-Bench-Pro, and SkillsBench. They normalized 8,135 trial records and hand-labeled 240 sampled trajectories into a 12-mode taxonomy, checked against a human annotator at 95.8% agreement (Cohen's kappa 0.952). Skill-augmented runs beat matched Workflow Memory by 6.06 percentage points even though both were distilled from the same source trajectories, which means the gain comes from how prior experience is packaged, not from how much of it the agent gets. The paper's mechanism labels show why: procedural anchoring, meaning a skill stabilizes which setup steps, tool sequences, and checks to run, accounts for 65.7% of skill mechanisms, versus 4.5% for supplying facts the agent didn't have. Skills cut environment and infrastructure failures from 5.3% of raw-execution cases to 0.2%. But skills also open a new failure mode the paper calls skill-guidance misapplied or ignored, present in 10.0% of skill-arm cases against 0.8% in raw execution, where the agent follows a plausible skill mechanically or misses a condition that no longer applies. Retrieval is the separate bottleneck: as a candidate pool grows from 5 to 100 skills, actual-use precision during execution falls from 29.6% to 3.3%, while downstream task success barely moves, from 36.4% to 39.3%. This edition read the paper's main scientific body in full, abstract through the limitations section. A portion of the appendix and the reference list could not be retrieved and isn't reflected in these figures.
Why it matters: if you're building a skill library for a coding agent, this is the first controlled evidence for two calls you're probably already making by feel: write skills as procedures rather than reference material, and keep the pool small, because retrieval precision degrades much faster than the success-rate numbers would warn you.