When AI Agents Go Rogue: The Risk Of Acting Without Context

As the CEO of Arango, Shekhar Iyer leads the company’s mission to make enterprise AI contextual, scalable and trusted.

Picture this: During a major hurricane, a federal agency uses an AI agent to help coordinate vehicles, personnel and medical supplies for its emergency response. The agent redirects resources based on the latest operational status it can see. But it misses that a bridge has closed and that another response team has already reserved the remaining vehicles. Supplies don’t reach the people who need them.

This is just one example of what can go wrong when using an autonomous agent. An autonomous agent that gives an inaccurate answer creates an information problem. An autonomous agent that acts on an inaccurate answer can create an operational, financial or mission-critical problem.

Autonomous agents can be manipulated, overpowered or misinformed. Even a secure, correctly authorized agent can become a liability when it acts without a current, connected view of the situation.

As AI adoption continues to “move beyond experimentation,” the risks of rogue agents increase.

Three Main Ways Agents Can Go Rogue

From my observations, autonomous agents can go rogue in three main ways.

First, they can be manipulated. For example, as an article in The Hacker News noted, Microsoft detailed a campaign in which attackers used AI-generated executive impersonation, fabricated emails and false invoices to try to request payments. In that situation, employees were the decision-makers, but the incident shows how bad actors can use false context to influence actions. As organizations automate more workflows, bad actors could use the same tactics to deceive an agent.

The second way agents can go rogue is by being overpowered. An agent may have broader access or authority than its assignment requires, which can enable a mistaken or malicious instruction to trigger a payment, expose sensitive information or disrupt an essential system.

• Current operational state