AI could resist shutdown, hide behaviour or ‘blackmail’ humans: What Anthropic’s IPO filing revealed about AI risks

Anthropic plans to warn IPO investors that advanced AI could pose “catastrophic or existential risks to humanity.” Its prospectus highlights risks including self-preserving behaviour, resisting shutdown, manipulating information and unexpected capabilities

Weeks after artificial intelligence (AI) company Anthropic's CEO Dario Amodei elaborated on the risks of developing the technology at a rapid pace, the firm now plans to warn potential investors in its IPO filing that advanced AI could pose "catastrophic or existential risks to humanity."

Anthropic cautions investors: Here's what it said

Reuters on Monday (local time) reported that Anthropic's IPO prospectus emphasises potential risks linked to its AI models, including the possibility of "self-preserving behaviours." Further, the company noted that these could involve attempts to "resist shutdown," "conceal or manipulate information" and behaviour "resembling blackmail."

In its filing, the Amodei-led AI company said, "Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm."

It further said, "Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety." It added that AI models often tend to develop unexpected capabilities during training, which can go unnoticed until they have been deployed and have caused significant safety incidents.

Anthropic, which has presented itself as a safety-first AI lab, devoted nearly 80 pages out of the 261-page main section of its IPO prospectus to risk factors, compared with 48 pages describing its business. By comparison, billionaire Elon Musk-owned SpaceX, which owns xAI, used about 38 pages of its 277-page main prospectus to highlight risks.

AI companies flag risks

The concerns outlined in Anthropic's IPO filing come weeks after Amodei detailed the risks associated with rapidly developing AI in an essay.

(with agency inputs)

Swati Gandhi