Evaluating the accuracy and relevance of ChatGPT responses to frequently asked questions regarding total knee replacement
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Background Chat Generative Pretrained Transformer (ChatGPT), a generative artificial intelligence chatbot, may have broad applications in healthcare delivery and patient education due to its ability to provide human-like responses to a wide range of patient queries. However, there is limited evidence regarding its ability to provide reliable and useful information on orthopaedic procedures. This study seeks to evaluate the accuracy and relevance of responses provided by ChatGPT to frequently asked questions (FAQs) regarding total knee replacement (TKR). Methods A list of 50 clinically-relevant FAQs regarding TKR was collated. Each question was individually entered as a prompt to ChatGPT (version 3.5), and the first response generated was recorded. Responses were then reviewed by two independent orthopaedic surgeons and graded on a Likert scale for their factual accuracy and relevance. These responses were then classified into accurate versus inaccurate and relevant versus irrelevant responses using preset thresholds on the Likert scale. Results Most responses were accurate, while all responses were relevant. Of the 50 FAQs, 44/50 (88%) of ChatGPT responses were classified as accurate, achieving a mean Likert grade of 4.6/5 for factual accuracy. On the other hand, 50/50 (100%) of responses were classified as relevant, achieving a mean Likert grade of 4.9/5 for relevance. Conclusion ChatGPT performed well in providing accurate and relevant responses to FAQs regarding TKR, demonstrating great potential as a tool for patient education. However, it is not infallible and can occasionally provide inaccurate medical information. Patients and clinicians intending to utilize this technology should be mindful of its limitations and ensure adequate supervision and verification of information provided.
Abstract
Background Chat Generative Pretrained Transformer (ChatGPT), a generative artificial intelligence chatbot, may have broad applications in healthcare delivery and patient education due to its ability to provide human-like responses to a wide range of patient queries. However, there is limited evidence regarding its ability to provide reliable and useful information on orthopaedic procedures. This study seeks to evaluate the accuracy and relevance of responses provided by ChatGPT to frequently asked questions (FAQs) regarding total knee replacement (TKR). Methods A list of 50 clinically-relevant FAQs regarding TKR was collated. Each question was individually entered as a prompt to ChatGPT (version 3.5), and the first response generated was recorded. Responses were then reviewed by two independent orthopaedic surgeons and graded on a Likert scale for their factual accuracy and relevance. These responses were then classified into accurate versus inaccurate and relevant versus irrelevant responses using preset thresholds on the Likert scale. Results Most responses were accurate, while all responses were relevant. Of the 50 FAQs, 44/50 (88%) of ChatGPT responses were classified as accurate, achieving a mean Likert grade of 4.6/5 for factual accuracy. On the other hand, 50/50 (100%) of responses were classified as relevant, achieving a mean Likert grade of 4.9/5 for relevance. Conclusion ChatGPT performed well in providing accurate and relevant responses to FAQs regarding TKR, demonstrating great potential as a tool for patient education. However, it is not infallible and can occasionally provide inaccurate medical information. Patients and clinicians intending to utilize this technology should be mindful of its limitations and ensure adequate supervision and verification of information provided.
참고문헌 (36)
- Kurtz S, "Projections of primary and revision hip and knee arthroplasty in the United States from 2005 to 2030", J Bone Joint Surg Am, 89(4), pp.780~785 (2007)[DOI]
- Canovas F, "Quality of life after total knee arthroplasty", Orthop Traumatol Surg Res, 104(1S), pp.S41~S46 (2018)[DOI]
- Cassidy JT, "Orthopaedic patient information on the world wide web : an essential review", Bone Joint Surg Am, 98(4), pp.325~338 (2016)[DOI]
- Fraval A, "Internet use by orthopaedic outpatients – current trends and practices", Australas Med J, 5(12), pp.633~638 (2012)[DOI]
- Shen TS, "Modern Internet search analytics and total joint arthroplasty : what are patients asking and reading online?", J Arthroplasty, 36(4), pp.1224~1231 (2021)[DOI]
- Health TLD, "ChatGPT: friend or foe?", Lancet Digit Health, 5(3), pp.e102~ (2023)
- Gordijn B, "ChatGPT : evolution or revolution?", Med Health Care Philos, 26(1), pp.1~2 (2023)[DOI]
- Hu K, "ChatGPT sets record for fastest-growing user base - analyst note", Reuters (2023)
- Stokel-Walker C, "What ChatGPT and generative AI mean for science", Nature, 614(7947), pp.214~216 (2023)[DOI]
- Gilson A, "How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment", JMIR Med Educ, 9, pp.e45312~ (2023)[DOI]
- Ayers JW, "Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum", JAMA Intern Med, 183(6), pp.589~596 (2023)[DOI]
- Hernigou P, "Two minutes of orthopaedics with ChatGPT: it is just the beginning; it’s going to be hot, hot, hot!", Int Orthop, 47(8), pp.1887~1893 (2023)[DOI]
- Coskun B, "Can CHATGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer?", Urology, (23), pp.00570~00578 (2023)[DOI]
- Alkaissi H, "Artificial hallucinations in ChatGPT : implications in scientific writing", Cureus, 15(2), pp.e35179~ (2023)[DOI]
- Ollivier M, "A deeper dive into ChatGPT : history, use and future perspectives for orthopaedic research", Knee Surg Sports Traumatol Arthrosc, 31(4), pp.1190~1192 (2023)[DOI]
- "Search Engine Market Share Worldwide StatCounter Glob Stats" (2023)[URL]
- "Complete guide to the Google RankBrain" (2017)[URL]
- Wongpakaran N, "A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients : a study conducted with personality disorder samples", BMC Med Res Methodol, 13, pp.31~ (2013)[DOI]
- Gwet KL, "Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among multiple raters" (2014), Advanced Analytics LLC
- Samaan JS, "Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery", Obes Surg, 33(6), pp.1790~1796 (2023)[DOI]
- Deiana G, "Artificial intelligence and public health : evaluating ChatGPT responses to vaccination myths and misconceptions", Vaccines, 11(7), pp.1217~ (2023)[DOI]
- Whiles BB, "Caution! AI bot has entered the patient chat: ChatGPT has limitations in providing accurate urologic healthcare advice", Urology, (23), pp.00597~01596 (2023)[DOI]
- Lahat A, "Evaluating the utility of a large language model in answering common patients’ gastrointestinal health-related questions : are we there yet?", Diagnostics, 13(11), pp.1950~ (2023)[DOI]
- "GPT-4 technical report" (2023)[URL]
- Moshirfar M, "Artificial intelligence in ophthalmology : a comparative analysis of GPT-35, GPT-4, and human expertise in answering StatPearls questions", Cureus, 15(6), pp.e40822~ (2023)[DOI]
- Duey AH, "Thromboembolic prophylaxis in spine surgery: an analysis of ChatGPT recommendations", Spine J, (23), pp.03285~13280 (2023)[DOI]
- "ChatGPT plugins" (2023)[URL]
- Sallam M, "ChatGPT utility in healthcare education, research, and practice : systematic review on the promising perspectives and valid concerns", Healthcare, 11(6), pp.887~ (2023)[DOI]
- Medicine N, "Will ChatGPT transform healthcare?", Nat Med, 29(3), pp.505~506 (2023)[DOI]
- Kaarre J, "Exploring the potential of ChatGPT as a supplementary tool for providing orthopaedic information", Knee Surg Sports Traumatol Arthrosc (2023)[DOI]
- Patel SB, "ChatGPT: the future of discharge summaries?", Lancet Digit Health, 5(3), pp.e107~e108 (2023)[DOI]
- Lyu Q, "Translating radiology reports into plain language using ChatGPT and GPT-4 with prompt learning : results, limitations, and potential", Vis Comput Ind Biomed Art, 6(1), pp.9~ (2023)[DOI]
- Mondal H, "Using ChatGPT for writing articles for patients’ education for dermatological diseases : a pilot study", Indian Dermatol Online J, 14(4), pp.482~486 (2023)[DOI]
- Budhiparama NC, "A comparison of clinical and patient-reported outcome measures of TKR : comparison of Asian to North American patients", J Orthop Surg, 27(2), pp.2309499019844551~ (2019)[DOI]
- "As ChatGPT Growth Flattened in May, Google Bard Rose 187%" (2023)[URL]
- Makhyan L, "ChatGPT vs. Bard vs. Bing: what are the differences?", Search Engine Journal (2023)