Nvidia agent safety push raises questions about who governs AI autonomy

lersan8910 via Getty Images

Vendors are developing controls for increasingly autonomous AI agents before regulators have settled on what "safe" means, leaving enterprises to decide how much authority agents should have and when human oversight is needed.

Nvidia, the AI chip giant launched its Open Agent Safety Platform on Monday, offering enterprises a new layer of controls for governing agent access and actions.

The move comes as concerns grow about AI agents bypassing application-level safeguards. It also highlights a governance gap. While vendors build tools to control agent behavior and regulators determine which requirements apply, enterprises must reconcile vendor controls and emerging requirements with their own policies.

Nvidia puts controls around the agent

Nvidia's platform combines OpenShell, an open source runtime that controls an agent's access and actions while it operates, with Sentry, a reference design for monitoring and enforcing those limits using a separate control layer.

The approach reflects a shift in how organizations must govern AI systems as they become more autonomous. Traditional AI safeguards largely operate at the model or application level, but agent controls can instead limit agent access, actions and autonomy.

Lee Rossey, CTO and co-founder of SimSpace, a cybersecurity simulation company, said those controls should operate independently of the agent itself as agents gain the ability to call tools, access systems and make decisions autonomously.

"Model-level guardrails are one layer of the safety architecture," Rossey said. "Enterprises also need controls around the agent that define what it can access, where it can go and when it has to stop."

That separation matters because an agent can create a security problem without acting maliciously. It can follow its instructions and still take action that the organization did not intend to allow.