Protected Health Information Recognition by Fine-Tuning a Pre-training Transformer Model
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Objectives: De-identifying protected health information (PHI) in medical documents is important, and a prerequisite to deidentificationis the identification of PHI entity names in clinical documents. This study aimed to compare the performanceof three pre-training models that have recently attracted significant attention and to determine which model is more suitablefor PHI recognition. Methods: We compared the PHI recognition performance of deep learning models using the i2b2 2014dataset. We used the three pre-training models—namely, bidirectional encoder representations from transformers (BERT),robustly optimized BERT pre-training approach (RoBERTa), and XLNet (model built based on Transformer-XL)—to detectPHI. After the dataset was tokenized, it was processed using an inside-outside-beginning tagging scheme and WordPiecetokenizedto place it into these models. Further, the PHI recognition performance was investigated using BERT, RoBERTa,and XLNet. Results: Comparing the PHI recognition performance of the three models, it was confirmed that XLNet had asuperior F1-score of 96.29%. In addition, when checking PHI entity performance evaluation, RoBERTa and XLNet showeda 30% improvement in performance compared to BERT. Conclusions: Among the pre-training models used in this study,XLNet exhibited superior performance because word embedding was well constructed using the two-stream self-attentionmethod. In addition, compared to BERT, RoBERTa and XLNet showed superior performance, indicating that they were moreeffective in grasping the context.
Abstract
Objectives: De-identifying protected health information (PHI) in medical documents is important, and a prerequisite to deidentificationis the identification of PHI entity names in clinical documents. This study aimed to compare the performanceof three pre-training models that have recently attracted significant attention and to determine which model is more suitablefor PHI recognition. Methods: We compared the PHI recognition performance of deep learning models using the i2b2 2014dataset. We used the three pre-training models—namely, bidirectional encoder representations from transformers (BERT),robustly optimized BERT pre-training approach (RoBERTa), and XLNet (model built based on Transformer-XL)—to detectPHI. After the dataset was tokenized, it was processed using an inside-outside-beginning tagging scheme and WordPiecetokenizedto place it into these models. Further, the PHI recognition performance was investigated using BERT, RoBERTa,and XLNet. Results: Comparing the PHI recognition performance of the three models, it was confirmed that XLNet had asuperior F1-score of 96.29%. In addition, when checking PHI entity performance evaluation, RoBERTa and XLNet showeda 30% improvement in performance compared to BERT. Conclusions: Among the pre-training models used in this study,XLNet exhibited superior performance because word embedding was well constructed using the two-stream self-attentionmethod. In addition, compared to BERT, RoBERTa and XLNet showed superior performance, indicating that they were moreeffective in grasping the context.
참고문헌 (30)
- 박영택, "Clinical Decision Support Functions and Digitalization of Clinical Documents of Electronic Medical Record Systems", Healthcare Informatics Research, 25(2), pp.115~123 (2019), 대한의료정보학회[DOI]
- Choi YI, "Development of machine learning model to predict the 5-year risk of starting biologic agents in patients with inflammatory bowel disease(IBD) : K-CDM network study", J Clin Med, 9(11), pp.3427~ (2020)
- Seong D, "Research trends in clinical natural language processing", Commun Korean Inst Inf Sci Eng, 35(5), pp.20~26 (2017)
- 신수용, "Privacy Protection and Data Utilization", Healthcare Informatics Research, 27(1), pp.1~2 (2021), 대한의료정보학회
- "Health Information Privacy Beyond HIPAA: A 2018Environmental Scan of Major Trends and Challenges" (2017)[URL]
- Soo-Yong Shin, "A De-identification Method for Bilingual Clinical Texts of Various Note Types", Journal of Korean Medical Science, 30(1), pp.7~15 (2015), 대한의학회
- Lafferty J, "Conditional random fields : probabilistic models for segmenting and labeling sequence data", Proceedings of the 18th International Conference on Machine Learning(ICML), pp.282~289 (2001)
- Wang Y, "Annotating and recognising named entities in clinical notes", Proceedings of the ACL-IJCNLP 2009Student Research Workshop, pp.18~26 (2009)
- Dreyfus SE, "Artificial neural networks, back propagation, and the Kelley-Bryson gradient procedure", J Guid Control Dyn, 13(5), pp.926~928 (1990)
- "Recurrent neural network (RNN) tutorial, Part 1" (2015)[URL]
- Hochreiter S, "Long short-term memory", Neural Comput, 9(8), pp.1735~1780 (1997)
- Liu Z, "Entity recognition from clinical texts via recurrent neural network", BMC Med Inform Decis Mak, 17(Suppl 2), pp.67~ (2017)
- "Efficient estimation of word representations in vector space" (2013)[URL]
- "fastText" (2020)[URL]
- "Deep contextualized word representations" (2018)[URL]
- 김정미, "Word2vec을 활용한 RNN기반의 문서 분류에 관한 연구", 한국지능시스템학회 논문지, 27(6), pp.560~565 (2017), 한국지능시스템학회
- Stubbs A, "Automated systems for the de-identification of longitudinal clinical narratives:overview of 2014 i2b2", UTHealth shared task Track 1, 2014, pp.58(Suppl(Suppl)), S11~S19
- "n2c2 NLP Research Data Sets" (2019)[URL]
- "BERT: pretraining of deep bidirectional transformers for language understanding" (2018)[URL]
- Lan Z, "ALBERT: a lite BERT for self-supervised learning of language representations", Proceedings of the 8th International Conference on Learning Representations (ICLR) (2020)
- "ROBERTa: a robustly optimized BERT pretraining approach" (2019)[URL]
- "deepset/roberta-base-squad2" (2020)[URL]
- "Adam: a method for stochastic optimization" (2014)[URL]
- Yang Z, "Xlnet : generalized autoregressive pretraining for language understanding", Adv Neural Inf Process Syst, 32, pp.5754~5764 (2019)
- "xlnet-base-cased" (2019)[URL]
- Warby SC, "Sleep-spindle detection : crowdsourcing and evaluating performance of experts, non-experts and automated methods", Nat Methods, 11(4), pp.385~392 (2014)
- 김용운, "Trends in Research on the Security of Medical Information in Korea: Focused on Information Privacy Security in Hospitals", Healthcare Informatics Research, 24(1), pp.61~68 (2018), 대한의료정보학회[DOI]
- "Auto-encoding variational bayes" (2013)[URL]
- Goodfellow I, "Generative adversarial nets", Adv Neural Inf Process Syst, 27, pp.2672~2680 (2014)
- "Publicly available clinical BERT embeddings" (2019)[URL]