Building Healthcare AI Systems That Calibrate Trust

Boris Berat is an operator, investor, and advisor. Co-founder & CTPO of Carna Health.

Some of the most useful challenges to my thinking about clinical AI come from conversations with my sister, Nevena Nemeš, a psychologist and experience strategist whose doctoral work explores human trust in AI. She asks a question technologists rarely frame precisely enough: Do we want people to trust AI more or trust it appropriately?

In healthcare, that distinction isn’t semantic. A patient who rejects a reliable recommendation may miss an opportunity for earlier care. A clinician who defers too readily to an unreliable recommendation can make an equally serious mistake. Both are failures of trust calibration.

Yet, many AI programs still treat trust as a sentiment to maximize. Leaders ask whether users trust the tool, whether adoption is increasing or whether explanations make people more comfortable. Those measures can reward the wrong behavior. In a high-stakes system, blind acceptance isn’t success, and skepticism isn’t always failure.

Building clinical software across countries has shown me how quickly a sound output loses value inside a human workflow. The same risk flag may be interpreted differently by a specialist, a primary care clinician, a community health worker and a patient at home. Trust isn’t a property of the model alone. It emerges from the relationship among the model, the interface, the workflow and the people expected to act.

Accuracy Is Only The Beginning

Research on automation describes poor trust calibration as a “core vulnerability.” A meta-analysis of 106 experiments found that human-AI systems often improved human performance but didn’t reliably outperform the better of the human or AI working alone. Better outcomes depend not only on model performance but also on whether people know when to rely on it.

If those questions are unanswered, model accuracy hasn’t become clinical value. The software produced an output, but the care system didn’t produce an outcome.