Using AI to improve patient access to clinical trials

Paradigm believed that LLMs, with their ability to summarize unstructured text, could be a great fit for their use case and replace the approach of building one-off ML models. They explored two potential paths: integrating with a custom LLM trained on medical use cases, or integrating with GPT‑4 through OpenAI’s API. 

Paradigm believed they would need a specialized medical model to get good results. They were “shocked” to find that GPT‑4 outperformed a team of highly trained human experts on complex data evaluation tasks. 

Ultimately, they chose OpenAI for several reasons: 

• Accuracy: Paradigm ran rigorous evaluations on their expert-curated golden data sets. GPT‑4 was at least 10% more accurate than state-of-the-art ML models on a blended precision/recall metric. In some cases, they saw “unbelievable” improvements. “The accuracy of OpenAI was better than our existing deployment and optimization of industry best-practice expert-trained models, and it was sometimes better than our trained clinicians,” Hirsch said. “The more complex the information, and the more different places that information sat, the better GPT‑4 was.”  

• Ease of use: “From a product standpoint, everything was just easy. The API was easy to use, and it was easy to integrate into our stack.” The team also appreciated that OpenAI’s support included high-quality API documentation: “Compared to working with other organizations, we can be more self-sufficient with OpenAI.”

• Multimodal entry and long context windows: Both features were key for medical record data.

• Security and regulatory compliance: “What really sold us on working with OpenAI was your serious approach to supporting regulatory compliance, including supporting our need to comply with HIPAA. Since we serve healthcare providers and their patients, regulatory compliance is a non-negotiable requirement for us.”