ANNO: A General Annotation Tool for Bilingual Clinical Note Information Extraction

Abstract

Objectives: This study was conducted to develop a generalizable annotation tool for bilingual complex clinical text annotation,which led to the design and development of a clinical text annotation tool, ANNO. Methods: We designed ANNO toenable human annotators to support the annotation of information in clinical documents efficiently and accurately. First,annotations for different classes (word or phrase types) can be tagged according to the type of word using the dictionaryfunction. In addition, it is possible to evaluate and reconcile differences by comparing annotation results between humanannotators. Moreover, if the regular expression set for each class is updated during annotation, it is automatically reflectedin the new document. The regular expression set created by human annotators is designed such that a word tagged once isautomatically labeled in new documents. Results: Because ANNO is a Docker-based web application, users can use it freelywithout being subjected to dependency issues. Human annotators can share their annotation markups as regular expressionsets with a dictionary structure, and they can cross-check their annotated corpora with each other. The dictionary-basedregular expression sharing function, cross-check function for each annotator, and standardized input (Microsoft Excel) andoutput (extensible markup language [XML]) formats are the main features of ANNO. Conclusions: With the growing needfor massively annotated clinical data to support the development of machine learning models, we expect ANNO to be helpfulto many researchers.