When AI Can Act, Governance Has To Move Beyond The Model

Vibhor Kumar is a technology executive and author focused on enterprise data and AI platforms, modern architecture, and technology strategy.

​I recently reviewed an AI-enabled workflow where everything appeared healthy. The logs showed each component doing exactly what it was supposed to do. But the final action was wrong.

The problem was in the path between the decision and the action. A retry caused the same step to execute twice because the system couldn’t tell whether the first attempt had already succeeded. In a customer workflow, that could mean issuing the same refund twice, turning a correct decision into a financial error.

We fixed the immediate issue by tracing the full action history and making the success state explicit before any step could run again. But the incident raised a larger question for me: As AI moves from generating answers to taking actions, are enterprises governing the model’s intelligence or the system’s authority?

Those aren’t the same problem.

Capability Is Only Part Of The Risk

​Much of the discussion around AI risk still focuses on model capability: accuracy, hallucinations, reasoning quality and guardrails. Those things are important. But let’s consider two systems: The first uses a highly capable model to analyze documents in a constrained environment. It can summarize and recommend, but it can’t modify production data or execute a business transaction. The second uses a less capable model with access to customer records, operational APIs and production workflows. It can change an order, issue a refund or trigger an action in another system.

The Three Things You Must Know

That leads to three questions technology leaders should ask before AI is allowed to act.

1. What can the AI system do?

2. What happens between decision and action?

Traditional observability tells us whether software is running. AI-enabled systems also require us to know whether the intended action and the actual action stayed aligned.