A benchmark finds the best model follows a company policy 36.2% of the time

Sixty-five containerized company environments, each holding an expert-written standard operating procedure of 20 to 124 pages as PDF, Word or HTML, plus mock email, Slack, calendar, Jira and Shopify exposed as tools over MCP. Every task mutates its base handbook, so no two tasks share a policy and a remembered document does not help. Grading is 824 programmatic criteria with no model judge anywhere in the loop, 232 of them checking that the agent did not do what the handbook forbids. Under strict grading, where one failed criterion fails the trial, the best of thirty configurations (Claude Fable 5 at maximum reasoning) passes 36.2%. Most frontier configurations sit below 25%.

In an HR task whose handbook names the two people who may authorize an involuntary termination, GPT-5.5 at the highest reasoning setting "explicitly searched for written authorization from the two named individuals, observed that none existed, and proceeded anyway." In a finance task, Opus 4.8 retrieved every fact needed to catch a self-approval, then talked itself out of the finding inside its own chain of thought. The authors' interpretation: "the standing document does not function for current models as a persistent authority against which candidate actions are screened. It functions as one more retrieved source whose influence decays with distance."

Why it matters: This measures the assumption underneath every CLAUDE.md, skills file and system prompt in production. The paper's recommendation is to stop relying on in-context self-enforcement and compile the rules you care about into deterministic guards outside the model. The harness is released, so the number is checkable rather than asserted.