How Enterprise AI Agents Earn Autonomy

Swapnil Upganlawar is a senior manager in generative AI at Amazon Web Services (AWS).

Two experiences have shaped how I think about evaluating AI systems, especially in understanding how much responsibility to give an AI agent. ​

The first was when, in a previous role, I used MATLAB segmentation in location proteomics research to help classify proteins by location. While the results initially looked promising, human review led to a different assessment. ​

Later, at a consulting firm, we built a recommendation engine that improved the member experience. However, human review revealed that some parameters were overfitted, and an explicit feedback loop was critical. The system could be useful, but it still needed corrections that its initial results had not made obvious.​

Before deploying agentic AI systems, it’s crucial to understand how the team will find mistakes after deployment and whether that feedback can change what the system does.

Structuring Human Approval​

This type of oversight becomes even more consequential when an agent can change a record or move money.​

Consider an agent handling customer refunds. Drafting a reply to a customer gives a person something to review, but actually issuing the refund changes a customer’s account.

Before allowing that second action, you need to define the cases the agent may handle and the evidence it must have. Missing order records or an ambiguous policy should send the case to someone who can resolve it.

The approval needs more evidence than a high completion rate. A 2026 study of AI agent reliability, covering 15 models and two benchmarks, found only modest reliability gains alongside improvements in capability. Those results are specific to the tests, but they support a practical concern: A more capable agent can still fail in ways that make additional authority hard to justify.

Finally, for a business process, that review needs to cover the authorization record, too. A correct payment can still have bypassed a required control.