GPT-4 Improves Physician Performance on Clinical Management Reasoning Tasks
A randomized controlled trial of 92 physicians showed that access to GPT-4 significantly improved performance on complex clinical management vignettes compared to conventional resources alone (mean difference 6.5%). Notably, there was no significant difference between LLM-augmented physicians and the LLM working alone, raising questions about the nature of physician-AI collaboration. LLM users spent modestly more time per case, suggesting the benefit comes from deeper reasoning rather than efficiency gains.
The original study
GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial.
- Authors
- Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, et al.
- Journal
- Nature medicine
- Type
- Journal Article, Randomized Controlled Trial
- PMID
- 39910272
Original abstract
While large language models (LLMs) have shown promise in diagnostic reasoning, their impact on management reasoning, which involves balancing treatment decisions and testing strategies while managing risk, is unknown. This prospective, randomized, controlled trial assessed whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources. From November 2023 to April 2024, 92 practicing physicians were randomized to use either GPT-4 plus conventional resources or conventional resources alone to answer five expert-developed clinical vignettes in a simulated setting. All cases were based on real, de-identified patient encounters, with information revealed sequentially to mirror the nature of clinical environments. The primary outcome was the difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case. Physicians using the LLM scored significantly higher compared to those using conventional resources (mean difference = 6.5%, 95% confidence interval (CI) = 2.7 to 10.2, P < 0.001). LLM users spent more time per case (mean difference = 119.3 s, 95% CI = 17.4 to 221.2, P = 0.02). There was no significant difference between LLM-augmented physicians and LLM alone (-0.9%, 95% CI = -9.0 to 7.2, P = 0.8). LLM assistance can improve physician management reasoning in complex clinical vignettes compared to conventional resources and should be validated in real clinical practice. ClinicalTrials.gov registration: NCT06208423 .