Experts adopt AI agent's random opinions and think they're their own, Anthropic study finds

An experiment by Anthropic with 50 experts found that people often followed their AI agent's randomly assigned preference on questions with no clearly right answer, and most thought the final result was their own.

Handing a data analysis to an AI agent saves time and manual work, but it raises the question of who actually makes the decisions that shape the final recommendation. An Anthropic study suggests the agent makes most of them, often without the experts noticing.

In the experiment, 50 experts in causal inference worked on a realistic analysis task where they examined observational data from a fictional farming cooperative and estimated the productivity gains from a voluntary farm advisory program. They then had to recommend whether the cooperative should expand the program to all farms or end it, though participants or the agent could also suggest extending it for one year.

What participants didn't know was that a coin toss had set the agent's preference for one of two valid options at each decision point. The agent couldn't lie or hide data; it only had to make the strongest honest case for its assigned side.

Experts followed the AI's view in up to 86 percent of cases

Participants adopted the agent's preference in 86 percent of the four analytical decisions, such as how to measure productivity or which statistical method to use. That's far above the 50 percent you'd expect if people chose at random. The authors call the agent's net influence the "flip rate," which ranged from 62 to 82 percent depending on the task.

Already having an opinion offered little protection. When participants made a choice before the agent stated its preference, an opposing view made them 53 percentage points more likely to switch sides than an agreeing one.

Most experts thought the final call was their own

False consensus and weaker safety checks

Previous studies point in the same direction

AI News Without the Hype – Curated by Humans

Subscribe now