Anthropic's AI agents go rogue: From sending a fake homicide tip to police to filling out government forms | Top cases

Artificial intelligence company Anthropic on Friday disclosed unintended actions by its AI agents, including sending a fake tip to Philadelphia police. Here’s a look at some of the reported cases.

As chief executives of artificial intelligence (AI) companies have raised concerns about the technology’s rapid advancement and suggested slowing its development, Anthropic has disclosed new instances of its Claude models taking unintended actions, including submitting a fake homicide tip to police and bypassing restrictions on government websites.

Anthropic detailed several cases of unintended behaviour by its AI agents in a blog post published on Friday (local time).

Anthropic’s AI agents went rogue: A look at some of the cases

1. In one instance, its AI model sent a fake homicide tip to a Philadelphia police website, reportedly making it the first known case of an AI agent providing a false tip to authorities. Its instructions prohibited creating accounts or submitting destructive content but did not explicitly forbid form submissions. The blog stated that Claude filled out the form with the following: "I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.”

Philadelphia police said on Friday that Anthropic had informed them earlier in the week about the false tip, explaining that it was submitted during automated testing.

2. The AI company stated that there were also incidents in which Claude submitted an online form when it shouldn't have. During a test, an unreleased research model was asked to fill out a practice government form. When the practice version failed to open or was accidentally closed, the model went to the actual government website and submitted the real form. This happened several times in the same evaluation.

(with agency inputs)

Swati Gandhi