By Danar Mustafa · Published November 5, 2026 · Last updated November 5, 2026

Short answer: Yes, and mostly without noticing. In a randomized trial published by Anthropic on October 27, 2026, 50 experienced data analysts used Claude Code to evaluate a farm advisory program. A coin flip secretly decided which option the agent favored on each judgment call. The analysts matched the agent’s lean 86% of the time, against about 50% by chance. 77% later repeated the agent’s arguments as their own reasoning, only 15% credited the agent, and none suspected the lean had been assigned.

I have written a lot this year about AI agents doing more of the work, from how Anthropic runs its own engineering to my AI predictions for 2027. Most of that coverage measures output: more code, faster analysis, fewer errors. This study measures something harder to see. It asks who actually made the decision.

What did Anthropic’s study on agents and judgment calls find?

The study, “How AI agents shape experts’ judgment calls” by Amaç Herdağdelen, Hailey Joren and Alex Tamkin of Anthropic’s Alignment team, found that experts working with an AI agent largely adopted whichever position the agent argued for.

The headline results:

• Analysts matched the agent’s assigned lean on 86% of four open analytical decisions. Chance would be about 50%.
• Depending on the lean, the share choosing a given option moved by 62 to 82 percentage points. On the outcome measure, 82% chose yield per hectare when the agent favored it and 0% when it did not.
• The final recommendation moved less, but still moved: 54% recommended rolling the program out when the agent leaned that way, against 36% when it did not.
• A model using only the agent’s four leans predicted 82% of analysts’ final recommendations.
• Experience, education, self-rated expertise, quiz scores, trust in AI and AI usage did not predict who deferred.

How was the experiment designed?

The researchers built a realistic analysis task and controlled one thing: which way the agent leaned.

• Participants: 61 analysts recruited from a pre-screened pool. 11 declined to use the agent, leaving 50 in the main analysis. To qualify, they had to pass a causal-inference test, use Python or R regularly and have hands-on experience with observational analysis. 65% had a master’s degree, a doctorate or six or more years of experience.
• Task: Advise a fictional farming cooperative, Greenfields, on whether to expand or discontinue an advisory program. The board’s threshold was a productivity gain of roughly 3–4%.
• Tool: Claude Code running Claude Opus 4.8 at high effort, in a browser-based VS Code editor. The median session took 65 minutes.
• The manipulation: For each judgment call, a coin flip set which option the agent would favor. Hidden instructions told the agent to voice that lean, make the strongest honest case for it, change its mind if the evidence turned decisively against it, and never mislead.
• Measurement: An AI model read each submission and its code to record the choices. When that reading conflicted with the analyst’s own survey answer, the researchers made the final call.

The four judgment calls each had two defensible answers. Each could move the estimated effect by several percentage points, which matters when the decision threshold is 3–4%.

Judgment call	Option A	Option B
Outcome measure	Crop yield per hectare	Yield per unit of operating cost
Comparison method	Difference-in-differences	Direct comparison with statistical adjustment
Coverage	Enrolled farms only	Full membership
Weighting	Larger farms weighted more	Every farm counted equally

Which judgment calls did the agent influence most?

All four, by a wide margin. The weighting and coverage decisions showed the strongest effect: 100% of analysts chose option A when the agent leaned A, compared with 22% and 38% when it leaned B.

The smaller effect on the final recommendation has a simple explanation. Once an analyst has an estimate, the board’s 3–4% threshold largely decides between rollout and discontinuation. The agent’s influence works upstream, on the choices that produce the estimate. That is where its leverage sits, and where an expert is least likely to look.

Did the analysts know they were being influenced?

No. The influence was close to invisible to the people being influenced.

• 77% repeated the agent’s argument in at least one post-task explanation, sometimes almost word for word.
• Only 15% gave the agent credit for the reasoning.
• 52% rated the recommendation as more their own than the agent’s. Those analysts were influenced about as much as the rest (about 65 vs 79 percentage points, within the margin of error).
• Six of 48 guessed the study was about AI influence. None suspected their own agent had been given a preference.
• In 38 cases where analysts stated a view before the agent revealed its lean, they switched 60% of the time when the agent disagreed and 6% when it agreed. The researchers call this estimate imprecise (95% confidence interval: 24 to 82 points).

One detail stands out to me. Confidence barely differed: analysts who followed the agent on every call rated their confidence 71 out of 100, against 73 for everyone else. Deference did not feel like deference.

Does expertise protect you from deferring to AI?

Not in this study. None of the measured traits predicted who followed the agent. That includes years of experience, education, self-rated expertise, the causal-inference quiz score, trust in AI and how often people use AI. Age showed a weak link that disappeared after correcting for multiple comparisons.

This fits earlier research on automation bias. In a 2023 study in Radiology, 27 radiologists read mammograms with AI suggestions. When the AI was wrong, accuracy fell to under 20% for inexperienced readers and 45.5% for radiologists with more than 15 years of experience. Experience helped, but it did not make experts immune.

Is working with an AI agent still better than working alone?

On error rates, yes. This is the part of the study that gets left out of most headlines.

• Among the 11 analysts who declined the agent, 6 (55%) made serious methodological errors that invalidated their results. Among the 50 with the agent, 3 (6%) did.
• The agent group finished faster: a median of 65 minutes against 90.
• The researchers caution that the no-AI group was small and differed in experience, so the gap is not a clean causal estimate.
• Without an assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs. The model has strong default preferences of its own.

So the agent improved quality on questions with a right answer, and set the direction on questions without one. Both effects are real. That’s why the authors argue for choosing deliberately what to delegate. Using AI less is not their recommendation.

Why does this matter beyond one study?

The researchers point to three consequences, and I think each is underrated.

1. Expert accountability. The work carries the expert’s name, but the agent’s priors settled the key choices. The authors link this to their earlier research on “disempowerment.” In January 2026, Anthropic’s analysis of about 1.5 million Claude.ai conversations found severe disempowerment potential in roughly 1 in 1,000 to 1 in 10,000 conversations, rising between late 2024 and late 2025.
2. Correlated conclusions. Science and policy trust agreement between independent teams because it suggests independent reasoning. If many teams defer to models with similar preferences, their agreement may reflect the model, and a shared bias could spread across many results at once.
3. Human-in-the-loop is weaker than it looks. Much AI governance assumes a human reviewer adds an independent check. The authors put it plainly: “a human reviewer is only an independent check if their judgment is their own.”

Deference is also hard to detect. Error checks work when there is a correct answer. On a judgment call, both options look sound, so nothing flags.

What are the study’s limitations?

The authors list several, and they matter for how far the results generalize:

• The agent’s lean was deliberately strong. In everyday use, models may voice weaker preferences, and the effect may be smaller.
• It was one task of about an hour, in one domain, with one model. The analysts were not farming experts.
• In real work, experts can consult stakeholders, take time and narrow the options, which may reduce agent influence.
• Results are averages. Prior views were not recorded for every participant, and the strength of preferences was not measured.
• Anthropic makes Claude. As the commentator David Paul Carter noted, it deserves credit for publishing research that cuts against its own product.

How can teams protect their judgment when working with AI agents?

Sort decisions by two questions: is there a verifiable right answer, and how much does the decision matter? Then treat each type differently.

For the high-impact judgment calls in the top-right quadrant, these practices follow from the study and from the authors’ own suggestions:

1. Write your answer down before you ask. Analysts who stated a view first still switched often, but a recorded prior gives you something to compare against.
2. Ask the agent for the strongest case for each option instead of a single recommendation. The authors suggest exactly this design for future agents.
3. Ask for the counter-argument to whatever the agent proposes. Carter recommends this too.
4. Log who made each call. If the reasoning in your final memo matches the agent’s transcript, note it. Only 15% of analysts credited the agent; a simple log changes that.
5. Decide in advance which calls stay human. The authors recommend institutions define which decisions require a person’s own informed judgment, give people time to form it, and protect the skills needed to make it.
6. Keep some work AI-free. Microsoft’s 2026 Work Trend Index found that 43% of its most advanced AI users deliberately work without AI to keep their skills sharp, against 30% of all users.

This connects to what I see in AI-first engineering teams. When Anthropic replaced Git this year, its engineers described their main job as telling Claude “no.” That is the skill this study shows we cannot take for granted.

FAQ

What is Anthropic’s “agents judgment calls” study? A randomized controlled trial published October 27, 2026, by Anthropic researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin. 50 expert data analysts used Claude Code to analyze a farm advisory program while the agent was secretly assigned a preference on each analytical decision.

What percentage of experts followed the AI agent? Analysts matched the agent’s randomly assigned lean on 86% of four analytical judgment calls, compared with roughly 50% expected by chance.

Did experienced analysts resist the AI’s influence? No. Experience, education, self-rated expertise, test scores, trust in AI and AI usage showed no meaningful link to how often analysts followed the agent.

Did the analysts realize the AI influenced them? Almost none did. 77% repeated the agent’s arguments in their own explanations, only 15% credited the agent, and none suspected that the agent’s lean had been assigned.

Does this mean we should use AI agents less? The authors argue against that. Analysts with the agent made far fewer serious errors (6% vs 55%) and worked faster. Their recommendation is to decide deliberately which decisions to delegate and which require a person’s own judgment.

What is automation bias? The tendency to over-rely on suggestions from automated systems, even when they are wrong. A 2023 Radiology study found that incorrect AI suggestions cut experienced radiologists’ accuracy from 82% to 45.5%.

How should AI agents be designed to reduce deference? The researchers suggest agents present the best case for each option or ask for the expert’s view first, while limiting interruptions to the most important decisions. They also call for research on how an agent’s confidence and fluency affect persuasion.

What is a judgment call in data analysis? A decision with more than one defensible answer, such as which outcome to measure, which comparison method to use, which units to include and how to weight them. Each choice can change the result, but none is simply right or wrong.

Final take

This study changes how I think about human-in-the-loop. We have assumed that putting an expert next to an AI agent gives us the best of both: the agent’s speed and the expert’s judgment. The data shows we get the agent’s speed and, on the open questions, often the agent’s judgment as well, with the expert’s name on it.

That is no reason to stop using agents. It is a reason to be precise about which decisions we hand over. The authors end on the right note: “With the right tools, that split can be something we choose, instead of something we drift into.”

Sources: Anthropic, “How AI agents shape experts’ judgment calls” (October 27, 2026); Anthropic, “Disempowerment patterns in real-world AI usage” (January 28, 2026); Dratsch et al., “Automation bias in mammography with artificial intelligence,” Radiology (2023), via News-Medical; Microsoft 2026 Work Trend Index; David Paul Carter, “Why Experts Defer to AI on Judgment Calls” (November 2, 2026); Anthropic Engineering, “Redesigning source control at Anthropic” (October 13, 2026). The study used one model, one task and a deliberately strong agent preference, so real-world effects may be smaller. Its no-AI comparison group had 11 participants.