AI Agents Need Human Accountability, Not More Oversight

​Shyam Ravindranathan is Senior Director of Product Management at SAP.

As companies deploy AI agents more, questions about human oversight have been growing. Specifically, I have seen the human-in-the-loop model built into the agentic AI development process in enterprise software become almost a rubber stamp as opposed to real, effective oversight.​

Companies keep a person in the process for valid reasons, including concerns about jobs, credibility and hallucination. But the role is still not well defined. In many agentic AI systems, human oversight is treated as a checkbox or a liability-addressing measure, and reviewers are often overwhelmed and underqualified.​

With the high volumes of output that AI can generate, reviewers frequently assume the AI is mostly correct, sample only a few items and batch-approve the rest. This can let the rare irreversible action slip through. And when accountability isn’t built into the heart of the system, it’s unclear who’s responsible if something fails: Is it the AI agent, the team that developed it, the person who reviewed it, or the business owner responsible for that entire set of products?​

Right People, Right Decisions

It’s important that the intensity of oversight corresponds to the level of risk. Consider an AI automation step that approves a $400 purchase and another that approves a $4 million procurement order. If a person were handling both, you would naturally build multiple checks and balances around the bigger order. The same logic should apply to AI automation.​

It’s also crucial to document every action an agent takes in a business process and assign a named owner for the result. Even though a step is automated, final responsibility for that action, even a small one, has to belong to a human being. Most customers I’ve seen start small and expand as they become more confident in their AI approach, so manual steps will remain.​