With High-Stakes AI, The Safest Answer Is Sometimes No Answer

Murali Rajendran, Founder & CEO of PracticeVantage Consulting and co-founder/CTO of Ariv Health, building AI for regulated industries.

Software is built to be helpful, and helpful usually means always returning some kind of result, even when it’s only a recommendation or a best guess.

For most products, that instinct is correct. In high-stakes work, however, it can be dangerous.

I have spent most of my career building high-stakes systems in healthcare, financial services and government. In those settings, a confident wrong answer is often worse than no answer at all. Providing no answer at least makes a person stop and think, while a wrong answer that looks right makes them act.

To Answer Or Not To Answer​

Engineers have a name for whether a system should provide an answer: fail-open or fail-closed. A system that fails open keeps running when something breaks, hoping for the best. A system that fails closed stops and refuses to continue when something goes wrong.

Physical locks have to pick one or the other when the power goes out, as do the systems we are now putting in front of people to help them make decisions.

AI has raised the stakes on that choice. Modern models are confident even when they are wrong, and confidence reads as authority.

An AI model will hand you a clean, well-worded, completely incorrect answer with the same confidence it uses for a correct one. The person on the other end has no easy way to tell the two apart.

Over time, they stop trying, which could lead to the slow erosion of the habit of checking, which is a much bigger risk than an occasional bad output.

What Building A Fail-Closed ​AI System Looks Like

When my company built a healthcare platform, we made refusal a feature. It sits on more than 120 million verified claims, and the operative word is verified. If a number cannot be traced to a source we trust, the platform does not estimate it. It shows nothing, and it says why.