Five Questions To Ask Before You Approve An AI System (And The Answers That Should Worry You)

Jeremy J. Samuelson, CTO/EVP, AI & Innovation, Integrated Quantum.

​Most executives already know the questions to ask about an AI system handling sensitive data. What they don’t know is that the wrong answers often sound reassuring.

“It’s encrypted at rest and in transit.”

“We removed all the PII.” ​

Statements like these point to real controls, but they can also be true about a system about to create a problem.

I’ve sat on both sides of these reviews for nearly twenty years, including in my current role, where my company builds data protection for AI systems. The main issue facing teams managing sensitive data is timing.

With AI systems, the review almost always starts after the architecture is settled. A team picks a model, wires it to the warehouse, stands up an endpoint and then brings security in to protect the flow that now exists.

By that point, raw or highly informative data are already sitting in feature stores, notebooks, logs, caches and at least one vendor environment. Instead of asking, “Should this data move?” beforehand, the conversation has to answer, “How do we defend the fact that it already did?”

Here are the questions worth asking (and the answers that should worry you) while you can still change the answer.

1. What decision should this system support?​

The answer that should worry you: “We pulled what was available and let the model decide what matters.”​

The business outcome should determine the data requirement, not the other way around.​ More data can add a useful signal, but it also adds noise, bias, legal obligation and breach impact. Those costs are rarely on the same slide as the accuracy number.

The goal here is not the smallest possible dataset—that’s a different mistake. It’s the least information that supports acceptable performance on the approved task. Ask the team to show you which inputs materially move that metric, and which ones are along for the ride.