Even Experts Fall Under AI’s Often-Imperfect Spell

The more people rely on artificial intelligence, the more their cognitive abilities slip. That’s alarming enough. What’s even more alarming is even experts may be over-relying on AI for judgment – even when it’s wrong. And they may not even be aware of it.

That’s the conclusion of a recent study out of Anthropic, which involved studying the reactions of 50 professional analysts using Claude Code to four analytical decisions to which agents made recommendations. The results suggest “that experts may defer underdetermined decisions to an AI agent, and that this deferral can be hard to identify, even for the experts themselves,” the report’s authors state. “Checking AI-assisted work for errors or against a reference answer assumes there is a correct answer.”

This is a problem popping up across all sectors, with an almost involuntary perception of AI being authoritative and almost flawless in its knowledge. In a study of 1,372 people conducted by researchers at the University of Pennsylvania, they accepted a planted wrong answer by AI more than 80% of the time. They call this “cognitive surrender.”

As the Wharton researchers explained it, “the danger wasn’t AI. It was deference to AI. The moment when AI is not just doing a specialized task but making the decision, and the person adopts that decision as their own without recognizing the transfer has occurred."

Add AI experts themselves to this cognitive surrender. During the course of the Anthropic experiment, “participants recalled most of their decisions accurately, and more than half rated the recommendation as more their own than the agent’s,” the Anthropic researchers said.

Tellingly, 77% repeated the agent’s talking points, "sometimes nearly verbatim, and only 15% mentioned the agent in any answer.," the study shows. "Whether they were persuaded or went along, participants let the agent’s randomly assigned priors settle most of the analysis, and their own accounts often did not reflect this.”