Validation of an Automatic Tagging System for Identifying Respiratory and Hemodynamic Deterioration Events in the Intensive Care Unit
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objectives: Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers. Methods: This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts’ eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system. Results: The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers. Conclusions: We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.
Abstract
Objectives: Predictive models for critical events in the intensive care unit (ICU) might help providers anticipate patient deterioration. At the heart of predictive model development lies the ability to accurately label significant events, thereby facilitating the use of machine learning and similar strategies. We conducted this study to establish the validity of an automated system for tagging respiratory and hemodynamic deterioration by comparing automatic tags to tagging by expert reviewers. Methods: This retrospective cohort study included 72,650 unique patient stays collected from Electronic Medical Records of the University of Massachusetts’ eICU. An enriched subgroup of stays was manually tagged by expert reviewers. The tags generated by the reviewers were compared to those generated by an automated system. Results: The automated system was able to rapidly and efficiently tag the complete database utilizing available clinical data. The overall agreement rate between the automated system and the clinicians for respiratory and hemodynamic deterioration tags was 89.4% and 87.1%, respectively. The automatic system did not add substantial variability beyond that seen among the reviewers. Conclusions: We demonstrated that a simple rule-based tagging system could provide a rapid and accurate tool for mass tagging of a compound database. These types of tagging systems may replace human reviewers and save considerable resources when trying to create a validated, labeled database used to train artificial intelligence algorithms. The ability to harness the power of artificial intelligence depends on efficient clinical validation of targeted conditions; hence, these systems and the methodology used to validate them are crucial.
참고문헌 (21)
- Moss TJ, "Signatures of subacute potentially catastrophic illness in the ICU : model development and validation", Crit Care Med, 44(9), pp.1639~1648 (2016)
- Ramachandran SK, "Independent predictors and outcomes of unanticipated early postoperative tracheal intubation after nonemergent, noncardiac surgery", Anesthesiology, 115(1), pp.44~53 (2011)
- Skrobik Y, "Scoring systems for the critically ill : use, misuse and abuse", Can J Anaesth, 53(3), pp.432~436 (2006)
- Vincent JL, "Ten reasons why we should NOT use severity scores as entry criteria for clinical trials or in our treatment decisions", Crit Care Med, 38(1), pp.283~287 (2010)
- McGaughey J, "Outreach and Early Warning Systems(EWS)for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards", Cochrane Database Syst Rev, (3), pp.CD005529~ (2007)
- Berlot G, "Anticipating events of in-hospital cardiac arrest", Eur J Emerg Med, 11(1), pp.24~28 (2004)
- Blackwell JN, "Early detection of inpatient deterioration: one prediction model does not fit all", Crit Care Explor, 2(5), pp.e0116~ (2020)
- Pollard TJ, "Enabling machine learning in critical care", ICU Manag Pract, 17(3), pp.198~199 (2017)
- Johnson AE, "Real-time mortality prediction in the Intensive Care Unit", AMIA Annu Symp Proc, 2017, pp.994~1003 (2018)
- Bailly S, "What's new in ICU in 2050 : big data and machine learning", Intensive Care Med, 44(9), pp.1524~1527 (2018)
- Zimlichman E, "Early recognition of acutely deteriorating patients in non-intensive care units : assessment of an innovative monitoring technology", J Hosp Med, 7(8), pp.628~633 (2012)
- Landrigan CP, "Temporal trends in rates of patient harm resulting from medical care", N Engl J Med, 363(22), pp.2124~2134 (2010)
- Chong JL, "Can we understand population healthcare needs using electronic medical records", Singapore Med J, 60(9), pp.446~453 (2019)
- Hanskamp-Sebregts M, "Measurement of patient safety : a systematic review of the reliability and validity of adverse event detection with record review", BMJ Open, 6(8), pp.e011078~ (2016)
- Cao H, "Heuristics to determine ventilation times of ICU patients from the MIMIC-II database", Annu Int Conf IEEE Eng Med Biol Soc, 2010, pp.5213~5216 (2010)
- Jeffery AD, "Advancing in-hospital clinical deterioration prediction Models", Am J Crit Care, 27(5), pp.381~391 (2018)
- Ueno R, "Value of laboratory results in addition to vital signs in a machine learning algorithm to predict in-hospital cardiac arrest : a single-center retrospective cohort study", PLoS One, 15(7), pp.e0235835~ (2020)
- Meyer A, "Machine learning for real-time prediction of complications in critical care : a retrospective study", Lancet Respir Med, 6(12), pp.905~914 (2018)
- Thorsen-Meyer HC, "Dynamic and explainable machine learning prediction of mortality in patients in the intensive care unit : a retrospective study of high-frequency data in electronic patient records", Lancet Digit Health, 2(4), pp.e179~e191 (2020)
- Bulgarelli L, "Prediction on critically ill patients : the role of"big data"", J Crit Care, 60, pp.64~68 (2020)
- Fleuren LM, "Machine learning for the prediction of sepsis : a systematic review and metaanalysis of diagnostic test accuracy", Intensive Care Med, 46(3), pp.383~400 (2020)