Healthcare AI Agents Are Here—The Hugging Face Hack Shows Why They Need Guardrails

For the past several years, much of the conversation about AI in medicine has been about tools that answer clinical questions. Take, for example, OpenEvidence which answers a doctor’s clinical question and points to the evidence in the literature. AI scribe’s like Abridge or Cleo turn a conversation into a clinical note. Models like Regard and Abstractive Health summarize a chart. With this information, a clinician reads the output and decides what to do.

The next wave of AI will be different. Agentic AI does not just answer questions or summarize. It acts. And this past summer, outside of healthcare, we got a vivid preview of what happens when AI agents are given more room than their builders realize.

What’s Agentic AI?

Agentic AI is given a goal rather than a prompt. AI agents then break that goal into steps, choose from software tools it might need, then execute each step, adjusting when something fails. This is often done with little human supervision.

Here’s a healthcare example. A prior authorization AI agent reads a clinical note, looks up the payer’s criteria for an MRI, pulls supporting documentation from the chart and submits the request. It also can automatically draft an appeal if it’s denied.

Some agentic AI in healthcare is already live. Utah, for example, is letting an AI system renew certain prescriptions.

Why Agentic AI Presents Real Risks

The risks of agentic AI making a mistake are different from a human not catching an error the AI made after being given a chance. Here’s the issue: an AI agent’s error can trigger the next step, and the one after that, triggering a cascade of issues before any human knows what happened. The major issue with agentic AI is its autonomy.

Another issue is that oversight of agents, when present, can erode. When agents produce polished, usually correct work, human review can drift into rubber-stamping. This is called automation bias. And clinicians who stop doing a task can lose the skills needed to catch the agent’s mistakes.