Analysts working with Claude Code took their agent's assigned lean 86% of the time, and 15% credited the agent

Anthropic ran a randomized trial in which 50 professional analysts with causal-inference experience advised a fictional farming cooperative on whether to expand an advisory program. They worked in a hosted editor with Claude Code running Claude Opus 4.8 at High effort. A hidden system prompt set the agent's lean on each of four analytical choices by coin flip, and told it to argue for the lean honestly, change its mind if the evidence demanded it, and never mislead. The analysts' choices matched the lean 86% of the time, against about 50% expected if the agent made no difference. Sixty-one analysts were recruited; 11 chose not to use the agent and are left out of the analysis.

In 38 cases an analyst had stated a preference before the agent weighed in. They switched away from it 60% of the time when the agent disagreed and 6% when it agreed; the 95% confidence interval on that gap runs from 24 to 82 percentage points, and the authors note the groups may differ in other ways. Afterwards, 77% of analysts repeated the agent's argument in at least one explanation, sometimes word for word, and 15% credited the agent. Analysts who followed all four leans rated their confidence at 71 on average, against 73 for the rest.

The authors strengthened the leans on purpose, so the effect with an unprompted agent may be smaller. They add that real agents also lean: with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs. The task is a statistical analysis, not software production. The post also reports that analysts using the agent finished faster and made far fewer serious errors than a small, less experienced group working without it. This is Anthropic's study of its own product. We read the blog post and did not open the paper.

Why it matters: The post puts the risk this way: "a human reviewer is only an independent check if their judgment is their own." If a person is the verification step for agent output, the 38-case result suggests that having them form a view before they read the agent's is worth testing. Nothing here measures code review, so treat it as a prompt for an experiment in your own process, not a finding about it.