딥러닝 언어 모델을 이용한 연구보고서의 참고문헌 자동추출 연구
Automatic Extraction of References for Research Reports using Deep Learning Language Model
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구는 단행본, 학술지, 보고서 등 다양한 종류의 발간물로 구성된 연구보고서의 참고문헌 데이터베이스를 효율적으로 구축하기 위한 것으로 딥러닝 언어 모델을 이용하여 참고문헌의 자동추출 성능을 비교 분석하고자 한다. 연구보고서는 학술지와는 다르게 기관마다 양식이 상이하여 참고문헌 자동추출에 어려움이 있다. 본 연구에서는 참고문헌 자동추출에 널리 사용되는 연구인 메타데이터 추출과 더불어 참고문헌과 참고문헌이 아닌 문구가 섞여 있는 환경에서 참고문헌만을 분리해내는 원문 분리 연구를 통해 이 문제를 해결하였다. 자동 추출 모델을 구축하기 위해 특정 연구기관의 연구보고서 내 참고문헌셋, 학술지 유형의 참고문헌셋, 학술지 참고문헌과 비참고문헌 문구를 병합한 데이터셋을 구성했고, 딥러닝 언어 모델인 RoBERTa+CRF와 ChatGPT를 학습시켜 메타데이터 추출과 자료유형 구분 및 원문 분리 성능을 측정하였다. 그 결과 F1-score 기준 메타데이터 추출 최대 95.41%, 자료유형 구분 및 원문 분리 최대 98.91% 성능을 달성하는 등 유의미한 결과를 얻었다. 이를 통해 비참고문헌 문구가 포함된 연구보고서의 참고문헌 추출에 대한 딥러닝 언어 모델과 데이터셋 유형별 참고문헌 구축 방향을 제안하였다.
Abstract
The purpose of this study is to assess the effectiveness of using deep learning language models to extract references automatically and create a reference database for research reports in an efficient manner. Unlike academic journals, research reports present difficulties in automatically extracting references due to variations in formatting across institutions. In this study, we addressed this issue by introducing the task of separating references from non-reference phrases, in addition to the commonly used metadata extraction task for reference extraction. The study employed datasets that included various types of references, such as those from research reports of a particular institution, academic journals, and a combination of academic journal references and non-reference texts. Two deep learning language models, namely RoBERTa+CRF and ChatGPT, were compared to evaluate their performance in automatic extraction. They were used to extract metadata, categorize data types, and separate original text. The research findings showed that the deep learning language models were highly effective, achieving maximum F1-scores of 95.41% for metadata extraction and 98.91% for categorization of data types and separation of the original text. These results provide valuable insights into the use of deep learning language models and different types of datasets for constructing reference databases for research reports including both reference and non-reference texts.
참고문헌 (35)
- 이강산다정, "국가R&D보고서 참고문헌 기술항목 구축 개선방안 연구", 한국융합학회논문지, 13(1), pp.31~42 (2022), 한국융합학회[DOI]
- 지선영, "사전학습 된 언어 모델 기반의 양방향 게이트 순환 유닛 모델과 조건부 랜덤 필드 모델을 이용한 참고문헌 메타데이터 인식 연구", 정보관리학회지, 38(1), pp.221~242 (2021), 한국정보관리학회[DOI]
- Besagni, D., "A segmentation method for bibliographic references by contextual tagging of fields", Seventh International Conference on Document Analysis and Recognition (2003)
- Chen, C., "CiteSpace II : Detecting and visualizing emerging trends and transient patterns in scientific literature", Journal of the American Society for Information Science and Technology, 57(3), pp.359~377 (2006)[DOI]
- Choi, W., "Building an annotated corpus for automatic metadata extraction from multilingual journal article references", PloS one, 18(1), pp.e0280637~ (2023)[DOI]
- Councill, I., "ParsCit : an Open-source CRF Reference String Parsing Package", LREC, 8, pp.661~667 (2008)
- Dai, Z., "Named entity recognition using BERT BiLSTM CRF for Chinese electronic health records", 2019 12th international congress on image and signal processing, biomedical engineering and informatics, pp.1~5 (2019)
- "Bert: Pre-training of deep bidirectional transformers for language understanding" (2018)[URL]
- Fritzler, A., "Few-shot classification in named entity recognition task", Proceedings of the 34th ACM, 2019, 993~1000
- "Yes but.. Can ChatGPT Identify Entities in Historical Documents?" (2023)[URL]
- Hetzner, E., "A simple method for citation metadata extraction using hidden markov models", Proceedings of the 8th ACM, 2008, 280~284
- Hollingsworth, B., "Retrieving hierarchical text structure from typeset scientific articles: a prerequisite for e-science text mining", Proc. of the 4th UK E-Science All Hands Meeting, pp.67~273 (2005)
- "Zero-shot Clinical Entity Recognition using ChatGPT" (2023)[URL]
- Huang, I. ., "Extracting citation metadata from online publication lists using BLAST", Advances in Knowledge Discovery and Data Mining: 8th Pacific-Asia Conference, pp.539~548 (2004)
- Kim, J., "Analysis of Zero-Shot Crosslingual Learning between English and Korean for Named Entity Recognition", Proceedings of the 1st Workshop on Multilingual Representation Learning, pp.224~237 (2021)
- "DeepData-REFMETA Version 1.0" (2022)[URL]
- "From zero to hero: on the limitations of zero-shot cross-lingual transfer with multilingual transformers" (2020)[URL]
- Liu, X., "Overview of named entity recognition", Journal of Contemporary Educational Research, 6(5), pp.65~68 (2022)[DOI]
- "Roberta: A robustly optimized bert pretraining approach" (2019)[URL]
- Lopez, P., "GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications", Research and Advanced Technology for Digital Libraries: 13th European Conference, pp.473~474 (2009)
- "Introducing ChatGPT" (2022)[URL]
- "Klue: Korean Language Understanding Evaluation" (2021)[URL]
- Radford, A., "Improving language understanding by generative pre-training", ~ (2018)
- Rodrigues A. D., "Deep reference mining from scholarly literature in the arts and humanities", Frontiers in Research Metrics and Analytics, 21 (2018)[DOI]
- Segura-Bedmar, I., "Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013)", Association for Computational Linguistics, pp.341~350 (2013)
- "Portuguese named entity recognition using BERTCRF"[URL]
- Tkaczyk, D., "CERMINE : automatic extraction of structured metadata from scientific literature", International Journal on Document Analysis and Recognition, 18, pp.317~335 (2015)[DOI]
- Van Eck, N., "Software survey : VOSviewer, a computer program for bibliometric mapping", Scientometrics, 84(2), pp.523~538 (2010)[DOI]
- "Improving reference mining in patents with BERT" (2021)[URL]
- "GPT-NER:Named Entity Recognition via Large Language Models" (2023)[URL]
- "Zero-shot information extraction via chatting with chatgpt" (2023)[URL]
- "A prompt pattern catalog to enhance prompt engineering with chatgpt" (2023)[URL]
- Wu, Y., "Research on named entity recognition of electronic medical records based on roberta and radical-level feature", Wireless Communications and Mobile Computing, 2021, pp.1~10 (2021)[DOI]
- "Simple and effective few-shot named entity recognition with structured nearest neighbor learning" (2020)[URL]
- Zhang, X., "A structural SVM approach for reference parsing", BMC bioinformatics, 12, pp.1~7 (2011)[DOI]