The Pope’s AI Guy Is Worried About ‘Cartel’ Behavior Among Big Labs

Recently, warnings from researchers inside the world’s largest artificial intelligence labs about the rapid pace of development have led to panic about the possibility that the technology could extinguish humanity. Those warnings are predicated on the idea that—propelled by ferocious competition and a rivalry between the US and China—labs are barreling toward superintelligent models whose abilities will far surpass those of humans.

But according to Father Paolo Benanti, a priest who advises the Catholic Church on AI, the handwaving over superintelligence distracts from the need for open debate about how to constrain the companies building the technology. By casting the problem as so complex that only they can solve it, he says, the big labs exclude everyone else from the conversation.

“Ethics cannot be something that belongs to the board of trustees of a company. It’s something that has to be balanced, mitigated in debate in the public square.” Benanti tells WIRED. “We need democratic regulation.”

In an essay published on the heels of the safety warnings, Anthropic CEO Dario Amodei called for new AI models to undergo mandatory third-party evaluations prior to release, and some sort of crackdown on distillation, a process where rival labs train their own AI on the outputs of proprietary models. Others have argued for a total ban on recursive self-improvement, where AI uses its advanced coding capabilities to help develop future iterations of itself.

Benanti is less interested in restricting methods for developing AI, which he fears could quash innovation, and more so in establishing standards that ensure models cannot slip from human control. Whatever shape those standards take, he says, they should not be designed by the labs.

It’s difficult to know how seriously to take grand pronouncements about the capabilities of the latest and future AI models, says Benanti, owing to the overlapping economic, political, and personal incentives at play.