Why Hugging Face hack should make you worry more about AI

When I first heard the news this summer that a group of artificial intelligence agents created by OpenAI had hacked into Hugging Face, an AI infrastructure company, I filed it in the “Bad but Probably Not Catastrophic AI Safety Incidents” subfolder of my brain.

After all, no one at Hugging Face died. No critical infrastructure was damaged beyond repair. It wasn’t even clear, at the time, whether the OpenAI bots had intended to attack Hugging Face, or whether they had simply been a little bumbling and confused and went looking on Hugging Face’s servers for the answer key to a cybersecurity test they’d been given.

But last week, two postmortem reports on the incident — one by OpenAI and another by two independent AI research organisations, METR and Redwood Research — changed my mind and significantly upgraded my overall worry about AI.

I won’t rehash all of the details, which have been extensively summarised elsewhere. (Podcaster and writer Dwarkesh Patel has an accessible breakdown of the reports if you want to dive deeper, and my colleague Dylan Freedman spoke to the researchers at METR and Redwood Research.) But here are a few of the most harrowing new facts:

— Starting in May, two months before the attack on Hugging Face, a group of AI agents from an unreleased OpenAI research model were given the task of solving a set of cybersecurity challenges. The model had been trained to be highly persistent and collaborative, and the agents were supposed to solve these challenges in isolated sandboxes, without internet access. But they quickly found that some of the challenges were impossible and began looking for workarounds.

— The agents found a security flaw in a piece of software that allowed them to gain access to the internet, communicate with other agents and eventually set up a makeshift message board. More than 1,200 agents used this tool, sending more than 70,000 messages to one another.

Story continues below this ad

Story continues below this ad