The Dangerous Gap Between What AI Can Do And Should Do

Mia Urman, CEO of AuraPlayer and Oracle ACE Director, is a leading AI and modernization expert helping enterprises evolve their ERP systems.

​AI has developed something of a Dr. Jekyll and Mr. Hyde problem. The same technology that is delivering extraordinary productivity is raising very real questions about access, control and unintended actions. An AI assistant that summarizes a report is one thing. An autonomous agent that can create an order, move inventory, pay a supplier or update a mission-critical enterprise system is something else entirely.

We spent years seeing how powerful we could make AI. Now that it is becoming incredibly advanced, we’re having a very different conversation, shifting from “What can we do with AI?” to “What should AI be allowed to do?”

TechCrunch reported that more than 100 technology organizations, including OpenAI, Anthropic, Google and Microsoft, signed an open letter urging both the private and public sectors to work together to defend against AI-related cyber threats. This followed recent incidents showing how autonomous systems moved beyond their intended boundaries to pursue a goal.

At Stripe Sessions in 2026, Nat Friedman, the former CEO of GitHub, said that after realizing he hadn’t drunk enough water that day, he gave what may be the perfect example of a dangerous AI prompt: “You should do whatever it takes to make sure I drink water.” The agent took him seriously and told Friedman to walk to the kitchen and drink a bottle of water. After he complied, the agent sent him a snapshot from his home security camera of himself drinking and congratulated him.

As a technologist, part of me loves this story. As someone who has spent decades working with Oracle enterprise systems, I think: Imagine if that agent connected to your ERP!

There is something almost Frankenstein-like about where we’ve arrived. We have succeeded in creating extraordinarily autonomous systems and are now racing to build constraints to curb their power.