Large Language Models for Pre-mediation Counseling in Medical Disputes: A Comparative Evaluation against Human Experts
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objectives: Assessing medical disputes requires both medical and legal expertise, presenting challenges for patients seekingclarity regarding potential malpractice claims. This study aimed to develop and evaluate a chatbot based on a chain-ofthoughtpipeline using a large language model (LLM) for providing medical dispute counseling and compare its performancewith responses from human experts. Methods: Retrospective counseling cases (n = 279) were collected from the Korea MedicalDispute Mediation and Arbitration Agency’s website, from which 50 cases were randomly selected as a validation dataset. The Claude 3.5 Sonnet model processed each counseling request through a five-step chain-of-thought pipeline. Thirty-eightexperts evaluated the chatbot’s responses against the original human expert responses, rating them across four dimensions ona 5-point Likert scale. Statistical analyses were conducted using Wilcoxon signed-rank tests. Results: The chatbot significantlyoutperformed human experts in quality of information (p < 0.001), understanding and reasoning (p < 0.001), and overallsatisfaction (p < 0.001). It also demonstrated a stronger tendency to produce opinion-driven content (p < 0.001). Despitegenerally high scores, evaluators noted specific instances where the chatbot encountered difficulties. Conclusions: A chainof-thought–based LLM chatbot shows promise for enhancing the quality of medical dispute counseling, outperforming humanexperts across key evaluation metrics. Future research should address inaccuracies resulting from legal and contextualvariability, investigate patient acceptance, and further refine the chatbot’s performance in domain-specific applications.
Abstract
Objectives: Assessing medical disputes requires both medical and legal expertise, presenting challenges for patients seekingclarity regarding potential malpractice claims. This study aimed to develop and evaluate a chatbot based on a chain-ofthoughtpipeline using a large language model (LLM) for providing medical dispute counseling and compare its performancewith responses from human experts. Methods: Retrospective counseling cases (n = 279) were collected from the Korea MedicalDispute Mediation and Arbitration Agency’s website, from which 50 cases were randomly selected as a validation dataset. The Claude 3.5 Sonnet model processed each counseling request through a five-step chain-of-thought pipeline. Thirty-eightexperts evaluated the chatbot’s responses against the original human expert responses, rating them across four dimensions ona 5-point Likert scale. Statistical analyses were conducted using Wilcoxon signed-rank tests. Results: The chatbot significantlyoutperformed human experts in quality of information (p < 0.001), understanding and reasoning (p < 0.001), and overallsatisfaction (p < 0.001). It also demonstrated a stronger tendency to produce opinion-driven content (p < 0.001). Despitegenerally high scores, evaluators noted specific instances where the chatbot encountered difficulties. Conclusions: A chainof-thought–based LLM chatbot shows promise for enhancing the quality of medical dispute counseling, outperforming humanexperts across key evaluation metrics. Future research should address inaccuracies resulting from legal and contextualvariability, investigate patient acceptance, and further refine the chatbot’s performance in domain-specific applications.