Mapping Drug Terms via Integration of a Retrieval-Augmented Generation Algorithm with a Large Language Model
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objectives: This study evaluated the efficacy of integrating a retrieval-augmented generation (RAG) model and a largelanguage model (LLM) to improve the accuracy of drug name mapping across international vocabularies. Methods: Drugingredient names were translated into English using the Japanese Accepted Names for Pharmaceuticals. Drug concepts wereextracted from the standard vocabulary of OHDSI, and the accuracy of mappings between translated terms and RxNorm wasassessed by vector similarity, using the BioBERT-generated embedded vectors as the baseline. Subsequently, we developedLLMs with RAG that distinguished the final candidates from the baseline. We assessed the efficacy of the LLM with RAG incandidate selection by comparing it with conventional methods based on vector similarity. Results: The evaluation metricsdemonstrated the superior performance of the combined LLM + RAG over traditional vector similarity methods. Notably,the hit rates of the Mixtral 8x7b and GPT-3.5 models exceeded 90%, significantly outperforming the baseline rate of 64%across stratified groups of PO drugs, injections, and all interventions. Furthermore, the r-precision metric, which measuresthe alignment between model judgment and human evaluation, revealed a notable improvement in LLM performance, rangingfrom 41% to 50% compared to the baseline of 23%. Conclusions: Integrating an RAG and an LLM outperformed conventionalstring comparison and embedding vector similarity techniques, offering a more refined approach to global druginformation mapping.
Abstract
Objectives: This study evaluated the efficacy of integrating a retrieval-augmented generation (RAG) model and a largelanguage model (LLM) to improve the accuracy of drug name mapping across international vocabularies. Methods: Drugingredient names were translated into English using the Japanese Accepted Names for Pharmaceuticals. Drug concepts wereextracted from the standard vocabulary of OHDSI, and the accuracy of mappings between translated terms and RxNorm wasassessed by vector similarity, using the BioBERT-generated embedded vectors as the baseline. Subsequently, we developedLLMs with RAG that distinguished the final candidates from the baseline. We assessed the efficacy of the LLM with RAG incandidate selection by comparing it with conventional methods based on vector similarity. Results: The evaluation metricsdemonstrated the superior performance of the combined LLM + RAG over traditional vector similarity methods. Notably,the hit rates of the Mixtral 8x7b and GPT-3.5 models exceeded 90%, significantly outperforming the baseline rate of 64%across stratified groups of PO drugs, injections, and all interventions. Furthermore, the r-precision metric, which measuresthe alignment between model judgment and human evaluation, revealed a notable improvement in LLM performance, rangingfrom 41% to 50% compared to the baseline of 23%. Conclusions: Integrating an RAG and an LLM outperformed conventionalstring comparison and embedding vector similarity techniques, offering a more refined approach to global druginformation mapping.
참고문헌 (32)
- Hripcsak G, "Observational Health Data Sciences and Informatics(OHDSI) : opportunities for observational researchers", Stud Health Technol Inform, 216, pp.574~578 (2015)
- Stang PE, "Advancing the science for active surveillance : rationale and design for the observational medical outcomes partnership", Ann Intern Med, 153(9), pp.600~606 (2010)[DOI]
- Reich C, "OHDSI standardized vocabulariesa large-scale centralized reference ontology for international data harmonization", J Am Med Inform Assoc, 31(3), pp.583~590 (2024)[DOI]
- Wang L, "Toward a normalized clinical drug knowledge base in China-applying the RxNorm model to Chinese clinical drugs", J Am Med Inform Assoc, 25(7), pp.809~818 (2018)[DOI]
- Henke E, "Assessing the use of German claims data vocabularies for research in the observational medical outcomes partnership common data model : development and evaluation study", JMIR Med Inform, 11, pp.e47959~ (2023)[DOI]
- Maier C, "Towards implementation of OMOP in a German University Hospital Consortium", Appl Clin Inform, 9(1), pp.54~61 (2018)[DOI]
- Nelson SJ, "Normalized names for clinical drugs : RxNorm at 6 years", J Am Med Inform Assoc, 18(4), pp.441~448 (2011)[DOI]
- de Groot R, "Determining and assessing characteristics of data element names impacting the performance of annotation using Usagi", Int J Med Inform, 178, pp.105200~ (2023)[DOI]
- Zhang Y, "Understanding bag-of-words model : a statistical framework", Int J Mach Learn Cybern, 1(1), pp.43~52 (2010)[DOI]
- "Similarity mapping of national drug code formulary systems between nations" (2022)[URL]
- Zhang Y, "Extraction of English drug names based on Bert-CNN mode", J Inf Hiding Multimed Signal Process, 11(2), pp.70~78 (2020)
- Brown T, "Language models are few-shot learners", Adv Neural Inf Process Syst, 33, pp.1877~1901 (2020)[DOI]
- "Retrievalaugmented generation for large language models: a survey" (2023)[URL]
- "Few-shot learning with retrieval augmented language models" (2022)[URL]
- "National Health Insurance (NHI) drug price listing code 2024" (2024)[URL]
- "Japanese accepted names for pharmaceuticals" (2024)[URL]
- "ATHENA: OHDSI Vocabularies Repository" (2024)[URL]
- Lee J, "BioBERT : a pre-trained biomedical language representation model for biomedical text mining", Bioinformatics, 36(4), pp.1234~1240 (2020)[DOI]
- "Google’s neural machine translation system:bridging the gap between human and machine translation" (2016)[URL]
- "The Faiss library" (2024)[URL]
- "Faiss indexes" (2023)[URL]
- "Mistral 7B" (2023)[URL]
- "Welcome Mixtral: a SOTA mixture of experts on Hugging Face" (2023)[URL]
- Ji Z, "BERT-based ranking for biomedical entity normalization", AMIA Jt Summits Transl Sci Proc, 2020, pp.269~277 (2020)
- "Outrageously large neural networks: the sparsely-gated mixture-of-experts layer" (2017)[URL]
- "GPT-4 technical report" (2023)[URL]
- Wu T, "A brief overview of ChatGPT : the history, status quo and potential future development", IEEE, 2023, pp.10(5), 1122~1136
- "LangChain" (2022)[URL]
- Bassani E, "European Conference on Information Retrieval", pp.259~264 (2022), Springer
- Jarvelin K, "Cumulated gain-based evaluation of IR techniques", ACM Trans Inf Syst, 20(4), pp.422~446 (2002)[DOI]
- Moffat A, "Rank-biased precision for measurement of retrieval effectiveness", ACM Trans Inf Syst, 27(1), pp.1~27 (2008)[DOI]
- Chen Y, "Automatic SNOMED CT coding of Chinese clinical terms via attention-based semantic matching", Int J Med Inform, 159, pp.104676~ (2022)[DOI]