KTAS 강사 매뉴얼과 ChatGPT를 활용한 중증도 분류의 일치도 분석
Inter-rater Agreement between Korean Triage and Acuity Scale between KTAS Instructor Manual and ChatGPT
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구는 KTAS 강사 매뉴얼의 추천답안과 ChatGPT가 분류한 KTAS 결과 일치도를 확인하는 것을 목표로 하였다. KTAS 강사 매뉴얼에 제공된 성인 9사례, 소아 5사례, 특수상황 10사례, 총 24사례를 대상으로 ChatGPT가 분류한 결과와 KTAS 강사매뉴얼의 추천답안 간의 일치도를 평가하였다. 분석 결과, KTAS 강사매뉴얼과 ChatGPT 간의 일치에 대한 Cohen’s kappa .518로 중증도의 일치도를 확인하였다. 성인 Cohen’s kappa .429, 소아 Cohen’s kappa .286, 특수 상황 Cohen’s kappa .625이었다. 또한, 질병 Cohen’s kappa는 .511, 질병 외 Cohen’s kappa는 .474이었다. KTAS 강사매뉴얼과 ChatGPT 간의 일치에 대한 ICC는 .798로 전반적인 일치도가 높음을 보여주었다. 본 연구는 ChatGPT가 응급의료 분야에서 KTAS 분류 도구로서 일정 수준의 신뢰성을 보일수 있음을 확인하였지만, 특히 소아나 질병 외의 사례에 대한 추가적인 자료가 필요함을 알 수 있었다.
Abstract
The goal of this study was to confirm the consistency between the answers recommended in the KTAS instructor manual and the KTAS results classified by ChatGPT. The consistency between ChatGPT's classification results and the instructor manual's recommended results was evaluated for nine adult cases, five pediatric cases, and ten special cases provided in the KTAS instructor manual. As a result of the analysis, the agreement between the KTAS instructor manual and ChatGPT was Cohen’s kappa of .518, Cohen’s kappa of .429 for adults, Cohen’s kappa of .286 for pediatric, and Cohen’s kappa of .625 for special cases. In addition, the kappa of the disease was .511 and the kappa of the exclusion disease was .474. The ICC for agreement between the KTAS instructor‘s manual and ChatGPT was .798, confirming a high overall agreement. This study confirmed that ChatGPT has a certain level of reliability as a KTAS classification tool in the field of emergency medicine, but it was found that additional data is needed, especially for cases pediatric or excluding diseases.
참고문헌 (32)
- Brandao-de-Resende, Camilo, "A Machine Learning System to Optimise Triage in an Adult Ophthalmic Emergency Department: A Model Development and Validation Study", eClinical Medicine, 66, pp.102331~ (2023)
- National Hospital Ambulatory Medical Care Survey: 2019 Emergency Department Summary Tables, "Cairns, Christopher", ~ (2022)
- Choi, Hyo Jeong, "Comparison with In-hospital Korean Triage and Acuity Scale(KTAS)and Prehospital Triage System in a Metropolitan City", Journal of the Korean Society of Emergency Medicine, 29(5), pp.391~398 (2018)
- Chowdhury, M. E., "Can AI Help in Screening Viral and COVID-19 Pneumonia?", Ieee Access, (8), pp.132665~132676 (2020)
- Classen, David C., "Bending the Patient Safety Curve: How Much Can AI Help?", NPJ Digital Medicine, 6(1), pp.2~ (2023)
- Dave, Tirth, "ChatGPT in Medicine: An Overview of Its Applications, Advantages, Limitations, Future Prospects, and Ethical Considerations", Frontiers in Artificial Intelligence, 6, pp.1169595~ (2023)
- Di Sarno, Lorenzo, "Artificial Intelligence in Pediatric Emergency Medicine:Applications, Challenges, and Future Perspectives", Biomedicines, 12(6), pp.1220~ (2024)
- Fernandes, Christopher M. B., "Five-level Triage: A Report from the ACEP", ENA Five-level Triage Task Force, 2005, pp.31(1), 39~50
- Higgins, Oliver, "Artificial Intelligence (AI) and Machine Learning (ML) Based Decision Support Systems in Mental Health: An Integrative Review", International Journal of Mental Health Nursing, 32(4), pp.966~978 (2023)
- Hwang, Soyun, "Machine Learning-Based Prediction of Critical Illness in Children Visiting the Emergency Department", Plos One, 17(2), pp.e0264184~ (2022)
- Kim, Jae Hyuk, "Reliability of ChatGPT for Performing Triage Task in the Emergency Department Using the Korean Triage and Acuity Scale", Digital Health, 10, pp.20552076241227132~ (2024)
- Koo, Terry K., "A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research", Journal of Chiropractic Medicine, 15(2), pp.155~163 (2016)
- KSEM, "KTAS (Korean Triage and Acuity Scale) Provider Mannual" (2023), Koonja Publishing Company
- Landis, J. Richard, "The Measurement of Observer Agreement for Categorical Data", Biometrics, 33, pp.159~174 (1997)
- Liu, N., "Artificial Intelligence in Emergency Medicine", Journal of Emergency and Critical Care Medicine, 2 (2018)
- Mathur, P., "Artificial Intelligence, Machine Learning, and Cardiovascular Disease", Clinical Medicine Insights : Cardiology, 14, pp.1179546820927404~ (2020)
- McKee, Courtney H., "Comparing the Accuracy of Mass Casualty Triage Systems when Used in an Adult Population", Prehospital Emergency Care, 24(4), pp.515~524 (2020)
- National Emergency Medical Center, "Emergency Medical Evaluation Standards Book" (2019)
- National Emergency Medical Center, "Emergency Medical Evaluation Standards Book" (2020)
- "Integrated Emergency Medical Information Intranet" (2024)[URL]
- National Emergency Medical Center, "National Emergency Department Information System", Standard Registration Construction and Input Form, ~ (2024)
- Oberlin, M., "Emergency Overcrowding and Hospital Organization : Causes and Solutions", La Revue de Médecine Intern, 41(10), pp.693~699 (2020)
- Park, Joonbum, "Korean Triage and Acuity Scale(KTAS)", Journal of the Korean Society of Emergency Medicine, 28(6), pp.547~551 (2017)
- Paslı, S., "Assessing the Precision of Artificial Intelligence in Emergency Department Triage Decisions : Insights from a Study with ChatGPT", The American Journal of Emergency Medicine, 78, pp.170~175 (2024)
- Pham, Kay T., "Artificial Intelligence and Chatbots in Psychiatry", Psychiatric Quarterly, 93(1), pp.249~253 (2022)
- Radford, A., "Improving Language Understanding by Generative Pre-training", Preprint, 1~12 (2018)
- Tyler, S., "Use of Artificial Intelligence in Triage in Hospital Emergency Departments : A Scoping Review", Cureus, 16(5), pp.1~ (2024)
- Vatansever, Sezen, "Artificial Intelligence and Machine Learning-aided Drug Discovery in Central Nervous System Diseases: State-of-the-arts and Future Directions", Medicinal Research Reviews, 41(3), pp.1427~1473 (2021)
- Zachariasse, J. M., "Validity of the Manchester Triage System in Emergency Care : A Prospective Observational Study", PLoS One, 12(2), pp.e0170811~ (2017)
- Zachariasse, J. M., "Performance of Triage Systems in Emergency Care : A Systematic Review and Meta-analysis", BMJ Open, 9(5), pp.e026471~ (2019)
- Zhang, Peng, "Generative AI in Medicine and Healthcare: Promises, Opportunities and Challenges", Future Internet, 15(9), pp.286~ (2023)
- Zhuang, Huangming, "A Systematic Review on Application of Deep Learning in Digestive System Image Processing", The Visual Computer, 39(6), pp.2207~2222 (2023)