How AI Assistants Get Paid Will Decide Whether They Help Or Harm Patients

Adam Farren is the CEO of Canvas Medical, an EMR company accelerating everyday medicine.

​Consider two hypothetical patients asking a consumer AI assistant like ChatGPT a health question. In the first one, a 58-year-old woman asks ChatGPT at 11:00 p.m. why she’s been short of breath climbing her stairs. She has her medical record connected, so ChatGPT Health can read her last three blood tests and notices her hemoglobin drifting from 13.4 to 9.6 g/dL. It recommends a doctor visit this week. The doctor’s workup finds iron deficiency; a colonoscopy finds the cause, and the tumor is stage I.

​In the second scenario, a 67-year-old man using the free version without any health record information asks the same question. ChatGPT gives a careful, well-reasoned answer based on the limited context provided but doesn’t flag any urgency or next steps. He doesn’t see a doctor, and six weeks later, he’s in an emergency department with a heart attack that could have been avoided.

​These scenarios are illustrative but point to something real. In this comparison, access to medical history looks like the clearest variable separating the two outcomes. Real-world results depend on more than that one factor, but a system’s access to a patient’s actual record is a meaningful, under-discussed piece of the picture.​

As consumer AI assistants take on more of the work of interpreting sensitive health information, are they being built with the right incentives to prioritize accuracy over engagement? That’s a business-model question as much as it is a technology question.​

Two Products, One Instructive Contrast​

OpenAI’s recent product decisions show this tension playing out, as the company made two structurally similar bets that landed in different places.​

Why The Business Model Is The Safety Question

This should be a major flag for healthcare and a no-brainer for why a paid subscription should be required, where the extra context is 100% needed with every query.​