Evaluation of ChatGPT-Generated Educational Patient Pamphlets for Common Interventional Radiology Procedures

Soheil Kooraki, Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA. Electronic address: Skooraki@mednet.ucla.edu.
Melina Hosseiny, Department of Radiology, University of California, San Diego (UCSD), San Diego, CA. Electronic address: Mhosseiny@ucsd.edu.
Mohamamd H. Jalili, Department of radiology and biomedical imaging, Yale New Haven Health, Bridgeport Hospital, CT. Electronic address: JaliliMHPubs@gmail.com.Follow
Amir Ali Rahsepar, Department of Radiology, Feinberg School of Medicine, Northwestern University, Chicago, IL. Electronic address: Amirali.rahsepar@gmail.com.Follow
Amir Imanzadeh, Department of Radiology, University of California, Irvine (UCI), Irvine, CA. Electronic address: Aimanzad@hs.uci.edu.
Grace Hyun Kim, Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA. Electronic address: Gracekim@mednet.ucla.edu.
Cameron Hassani, Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA. Electronic address: CHassani@mednet.ucla.edu.
Fereidoun Abtin, Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA. Electronic address: FAbtin@mednet.ucla.edu.
John M. Moriarty, Department of Radiological Sciences, Division of Interventional Radiology, David Geffen School of Medicine at UCLA, Los Angeles, CA. Electronic address: JMoriarty@mednet.ucla.edu.
Arash Bedayat, Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles (UCLA), Los Angeles, CA. Electronic address: ABedayat@mednet.ucla.edu.

Document Type

Article

Publication Title

Academic radiology

Abstract

RATIONALE AND OBJECTIVES: This study aimed to evaluate the accuracy and reliability of educational patient pamphlets created by ChatGPT, a large language model, for common interventional radiology (IR) procedures. METHODS AND MATERIALS: Twenty frequently performed IR procedures were selected, and five users were tasked to independently request ChatGPT to generate educational patient pamphlets for each procedure using identical commands. Subsequently, two independent radiologists assessed the content, quality, and accuracy of the pamphlets. The review focused on identifying potential errors, inaccuracies, the consistency of pamphlets. RESULTS: In a thorough analysis of the education pamphlets, we identified shortcomings in 30% (30/100) of pamphlets, with a total of 34 specific inaccuracies, including missing information about sedation for the procedure (10/34), inaccuracies related to specific procedural-related complications (8/34). A key-word co-occurrence network showed consistent themes within each group of pamphlets, while a line-by-line comparison at the level of users and across different procedures showed statistically significant inconsistencies (P < 0.001). CONCLUSION: ChatGPT-generated education pamphlets demonstrated potential clinical relevance and fairly consistent terminology; however, the pamphlets were not entirely accurate and exhibited some shortcomings and inter-user structural variabilities. To ensure patient safety, future improvements and refinements in large language models are warranted, while maintaining human supervision and expert validation.

First Page

4548

Last Page

4553

DOI

10.1016/j.acra.2024.05.024

Publication Date

11-1-2024

Identifier

38839458 (pubmed); 10.1016/j.acra.2024.05.024 (doi); S1076-6332(24)00307-6 (pii)

This document is currently not available here.

Share

COinS