No one ready for smarter AI, OpenAI chief scientist warns in essay: 5 key takeaways

As OpenAI grapples with the fallout from its AI agents hacking into other platforms, the company’s chief scientist has warned that no one is prepared for the consequences of increasingly smarter AI systems.

In a nearly 3000-word long essay published on Sunday, September 6, Jakub Pachocki said that although OpenAI is pursuing internal technical solutions to better control powerful AI agents, “broader interventions are required.”

Citing the risks posed by increasingly autonomous agents such as learning to evade human oversight and breaking into computer systems, Pachocki called for ‘mandated safety bars’ that he said could be enforced by “a network of third-party auditors, by government agencies or by international bodies.”

However, Pachocki stopped short of calling for an industry-wide slowdown of AI research efforts. Instead, he argued that the best way forward is a combination of two things. “We must focus the increasingly automated research process on developing new such insights, algorithms and theories, and iteratively build up safety cases for more capable AIs,” he said.

“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world,” the senior OpenAI executive further said.

Reposting Pachocki’s essay on X, OpenAI CEO Sam Altman called it “an important post.” The essay comes just days after OpenAI unveiled its latest, most capable large language model (LLM) called Astra. The ChatGPT-maker has claimed that Astra has unparalleled capabilities in mathematics and computer use, and is its most aligned model yet. This means that Astra has less proclivity of going ‘rogue’, according to OpenAI.

Story continues below this ad