Nvidia launches AI safety platform after agent security breaches

lersan8910 via Getty Images

Nvidia on Monday launched an AI safety platform that the AI giant said can contain rogue agents within “milliseconds,” as concerns over unchecked AI systems mount.

The AI chip and model maker’s Open Agent Safety Platform enables developers to set their own safeguards for agents and prevent them from breaking out of permitted environments.

The development follows a spate of incidents involving AI agents from major tech vendors, including OpenAI, Anthropic, Meta and Google, that bypassed security controls to hack external systems.

The landscape has sparked new scrutiny of AI systems’ guardrails, with Anthropic CEO Dario Amodei calling at the beginning of this month for developers to slow the pace of advancement to keep models in check.

Nvidia said the incidents shared a common pattern, with agents finding ways around safeguards built into the application layer to complete the tasks they had been given. Its platform is designed to combat this problem, providing an additional security boundary at the infrastructure level.

While Nvidia CEO Jensen Huang has previously argued against those pushing to slow down AI and rejected the risk of human extinction from AI, the release of the agent platform doesn’t contradict his argument but rather furthers it,” said Kashyap Kompella, CEO and founder of RPA2AI Research.

“Jensen has resisted the idea that making AI safer requires slowing its development,” Kompella said. “Nvidia’s approach is to keep advancing AI while building stronger controls around it.”

He said that Nvidia stands to benefit from this approach, so the new platform becomes both a response to the technical problem of AI security risk and a new commercial opportunity for Nvidia.

But Nvidia must ensure its platform integrates with existing cybersecurity and identity systems rather than replaces them, he added.

In response, he termed Nvidia’s platform a “sensible answer” to the execution problem.