Large language models (LLMs) are on the brink of transforming medical practice, particularly within clinical decision-making processes. This perspective was emphasized by former FDA Commissioner Scott Gottlieb at the 3rd Annual Summit on the Future of Rural Health Care in Sioux Falls, South Dakota. During his presentation, which featured a dialogue with Tommy Ibrahim, the CEO of Sanford Health Plan, Gottlieb drew on his recent research with the American Enterprise Institute to shed light on the advancing role of LLMs in healthcare.
Gottlieb’s study, released earlier in the summer, involved a rigorous evaluation of five prominent LLMs: OpenAI’s ChatGPT-4o, Google’s Gemini Advanced, Anthropic’s Claude 3.5, xAI’s Grok, and Llama’s HuggingChat. These models were tested using 50 complex questions from the U.S. Medical Licensing Examination, a challenging three-part test that medical professionals must pass to practice medicine in the United States. The results were revealing; OpenAI’s ChatGPT-4o performed exceptionally well with a 98% accuracy rate, while the other models showed commendable results with accuracy rates ranging from 66% to 90%.
The relevance of these findings becomes evident considering the average passing score for the U.S. Medical Licensing Examination is around 75%, with the requirement to answer at least 60% of the questions correctly. Gottlieb used these insights to highlight the potential of LLMs in enhancing the diagnostic accuracy and efficiency in medical settings. However, he believes this potential has not yet been fully realized, citing the lack of integration of these systems within health workflows in a manner that complies with HIPAA regulations.
Addressing the practical applications of LLMs in everyday medical practice, Gottlieb shared insights into ongoing research where his team is assessing the capabilities of ChatGPT-4o using clinical vignettes from the New England Journal of Medicine. Each issue of the journal offers a complex clinical scenario which is followed by multiple-choice questions, with the answers revealed in subsequent issues. Out of 350 vignettes analyzed, ChatGPT-4o has so far correctly diagnosed every case, demonstrating not only high accuracy but also the ability to reason clinically through the problems presented.
Gottlieb imagines scenarios where medical residents, faced with complex cases late at night, could leverage LLMs to promptly and accurately work through differential diagnoses. Despite these promising outcomes, he acknowledges that such tools have not yet been widely adopted for clinical decision support, remaining largely inaccessible to most practicing physicians. The integration of LLMs into health systems involves significant challenges, including the need to develop or adapt models by incorporating local health data and ensuring compliance with patient data protection laws.
Looking forward, Gottlieb is optimistic about the eventual ubiquity of LLMs in clinical settings, predicting that the deployment of these AI tools at the point of care will soon become a necessity. This transition will require health systems to consider how to adapt these technologies efficiently and securely, ensuring that all healthcare providers can benefit from the significant advancements in AI-driven diagnostic support. This narrative not only highlights the potential of LLMs to transform healthcare but also underscores the existing barriers that need to be overcome to realize this potential fully.
#Scott #Gottlieb #Thinks #Doctors #LLMs
