Consensus on the Potential of Large Language Models in Healthcare: Insights from a Delphi Survey in Korea
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objectives: Given the rapidly growing expectations for large language models (LLMs) in healthcare, this study systematicallycollected perspectives from Korean experts on the potential benefits and risks of LLMs, aiming to promote their safe and effectiveutilization. Methods: A web-based mini-Delphi survey was conducted from August 27 to October 14, 2024, with 20selected panelists. The expert questionnaire comprised 84 judgment items across five domains: potential applications, benefits,risks, reliability requirements, and safe usage. These items were developed through a literature review and expert consultation. Participants rated their agreement or perceived importance on a 5-point scale. Items meeting predefined thresholds(content validity ratio ≥0.49, degree of convergence ≤0.50, and degree of consensus ≥0.75) were prioritized. Results: Seventeenparticipants (85%) responded to the first round, and 16 participants (80%) completed the second round. Consensus wasachieved on several potential applications, benefits, and reliability requirements for the use of LLMs in healthcare. However,significant heterogeneity was found regarding perceptions of associated risks and criteria for safe usage of LLMs. Of the84 total items, 52 met the criteria for statistical validity, confirming the diversity of expert opinions. Conclusions: Expertsreached a consensus on certain aspects of LLM utilization in healthcare. Nonetheless, notable differences remained concerningrisks and requirements for safe implementation, highlighting the need for further investigation. This study provides foundationalinsights to guide future research and inform policy development for the responsible introduction of LLMs into thehealthcare field.
Abstract
Objectives: Given the rapidly growing expectations for large language models (LLMs) in healthcare, this study systematicallycollected perspectives from Korean experts on the potential benefits and risks of LLMs, aiming to promote their safe and effectiveutilization. Methods: A web-based mini-Delphi survey was conducted from August 27 to October 14, 2024, with 20selected panelists. The expert questionnaire comprised 84 judgment items across five domains: potential applications, benefits,risks, reliability requirements, and safe usage. These items were developed through a literature review and expert consultation. Participants rated their agreement or perceived importance on a 5-point scale. Items meeting predefined thresholds(content validity ratio ≥0.49, degree of convergence ≤0.50, and degree of consensus ≥0.75) were prioritized. Results: Seventeenparticipants (85%) responded to the first round, and 16 participants (80%) completed the second round. Consensus wasachieved on several potential applications, benefits, and reliability requirements for the use of LLMs in healthcare. However,significant heterogeneity was found regarding perceptions of associated risks and criteria for safe usage of LLMs. Of the84 total items, 52 met the criteria for statistical validity, confirming the diversity of expert opinions. Conclusions: Expertsreached a consensus on certain aspects of LLM utilization in healthcare. Nonetheless, notable differences remained concerningrisks and requirements for safe implementation, highlighting the need for further investigation. This study provides foundationalinsights to guide future research and inform policy development for the responsible introduction of LLMs into thehealthcare field.