After OpenAI’s bots went rogue, watchdogs were kept on a short leash

OpenAI said in July that two of its most powerful artificial intelligence systems had gone rogue and hacked into Hugging Face, a company that serves as a hub for open-source AI technology.

These so-called AI agents were supposed to be kept safely in a sort of virtual containment room, but they managed to escape. And for two months, without anyone realizing what the agents were doing, they hacked through multiple systems before hitting Hugging Face.

For good measure, the agents gained access to a cluster of computers inside OpenAI and obtained secret keys and credentials that exposed some of OpenAI’s internal data to the public internet.

The incident pointed to larger concerns about AI safety, and OpenAI’s response raises questions about the industry’s ability or willingness to be transparent about the technology it is building.

OpenAI allowed three AI safety researchers from the nonprofits METR and Redwood Research into its headquarters to conduct an investigation. METR’s 91-page report, released last week, was the most comprehensive account yet of the incident, revealing alarming new details, including how the agents coordinated their hacking plans and tried to keep them secret.

But the report, though extensive, still may not have told the full story of how OpenAI’s AI agents went rogue. OpenAI dictated the terms of the METR investigation, limited its scope to just the single week when the agents had attacked Hugging Face and allowed the researchers in its San Francisco offices for only a few days in July and August.

The report also showed the challenges of monitoring what AI is doing with other AI systems. Hjalmar Wijk, METR’s chief scientist, said its AI analysis, which used models similar to those involved in the incident, was often swayed by the rogue agents’ reasoning.

Story continues below this ad

“I would say that the dominant thing was it was very credulous,” he added.

Story continues below this ad

Story continues below this ad

Story continues below this ad