An Evaluation of Multiple Query Representations for the Relevance Judgments used to Build a Biomedical Test Collection
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Objectives: The purpose of this study is to validate a method that uses multiple queries to create a set of relevance judgments used to indicate which documents are pertinent to each query when forming a biomedical test collection. Methods: The aspect query is the major concept of this research; it can represent every aspect of the original query with the same informational need. Manually generated aspect queries created by 15 recruited participants where run using the BM25 retrieval model in order to create aspect query based relevance sets (QRELS). In order to demonstrate the feasibility of these QRELSs, The results from a 2004 genomics track run supported by the National Institute of Standards and Technology (NIST) were used to compute the mean average precision (MAP) based on Text Retrieval Conference (TREC) QRELSs and aspect-QRELSs. The rank correlation was calculated using both Kendall’s and Spearman’s rank correlation methods. Results: We experimentally verified the utility of the aspect query method by combining the top ranked documents retrieved by a number of multiple queries which ranked the order of the information. The retrieval system correlated highly with rankings based on human relevance judgments. Conclusions: Substantial results were shown with high correlations of up to 0.863 (p < 0.01) between the judgment-free gold standard based on the aspect queries and the human-judged gold standard supported by NIST. The results also demonstrate that the aspect query method can contribute in building test collections used for medical literature retrieval.
Abstract
Objectives: The purpose of this study is to validate a method that uses multiple queries to create a set of relevance judgments used to indicate which documents are pertinent to each query when forming a biomedical test collection. Methods: The aspect query is the major concept of this research; it can represent every aspect of the original query with the same informational need. Manually generated aspect queries created by 15 recruited participants where run using the BM25 retrieval model in order to create aspect query based relevance sets (QRELS). In order to demonstrate the feasibility of these QRELSs, The results from a 2004 genomics track run supported by the National Institute of Standards and Technology (NIST) were used to compute the mean average precision (MAP) based on Text Retrieval Conference (TREC) QRELSs and aspect-QRELSs. The rank correlation was calculated using both Kendall’s and Spearman’s rank correlation methods. Results: We experimentally verified the utility of the aspect query method by combining the top ranked documents retrieved by a number of multiple queries which ranked the order of the information. The retrieval system correlated highly with rankings based on human relevance judgments. Conclusions: Substantial results were shown with high correlations of up to 0.863 (p < 0.01) between the judgment-free gold standard based on the aspect queries and the human-judged gold standard supported by NIST. The results also demonstrate that the aspect query method can contribute in building test collections used for medical literature retrieval.
참고문헌 (20)
- Best practices for test collection creation and information retrieval system evaluation, "Sanderson M.", TrebleCLEF, ~, pp.Technical report no.: D4.2 (2009)
- Voorhees EM., "Building a question answering test collection", Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.200~207 (2000)
- Oard DW., "Building an information retrieval test collection for spontaneous conversational speech", Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.41~48 (2004)
- Hersh W., "OHSUMED: an interactive retrieval evaluation and new large test collection for research", Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.192~201 (1994)
- Heppin KF, "MedEval: a Swedish medical test collection with doctors and patients user groups", Proceedings of the NAACL HLT 2010 Second Louhi Workshop on Text and Data Mining of Health Document, pp.1~7 (2010)
- Efron M., "Using multiple query aspects to build test collections without human relevance judgments", Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieva, pp.276~287 (2009)
- Sanderson M., "Forming test collections with no system pooling", Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.33~40 (2004)
- Soboroff, I, "Ranking retrieval systems without relevance judgments", Proceedings ofthe 24th Annual International ACM SIGIR Conferenceon Research and Development in Information Retrieval, pp.66~73 (2001)
- Wu S., "Methods for ranking information retrieval systems without relevance judgments", Proceedings of the 2003 ACM Symposium on Applied Computing, pp.811~816 (2003)
- Grady C., "Crowdsourcing document relevance assessment with mechanical turk", Proceedings of the NAACL HLT Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pp.172~179 (2010)
- Cao YG., "Evaluation of the clinical question answering presentation", Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, pp.171~178 (2009)
- Luo G., "Design and evaluation of the iMed intelligent medical search engine", Proceedings of the IEEE International Conference on Data Engineering, pp.1379~1390 (2009)
- ". Gaithersburg (MD): NIST" (2011)[URL]
- Si L., "Combining multiple resources.,evidence and criteria for genomic information retrieval", In: Proceedings of the Fifteenth Text Retrieval Conference (TREC) (2006)
- Yin X., "Promoting ranking diversity for biomedical information retrieval using wikipedia", Proceedings of the 32nd European Conference on Advances in Information Retrieval, pp.495~507 (2010)
- Yin X., "A survival modeling approach to biomedical search result diversification using wikipedia", Proceedings of the 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.901~902 (2010)
- "TREC genomics track" (2008)[URL]
- Hersh WR., "Report on the TREC 2004 genomics track", ACM SIGIR Forum, 39, pp.21~24 (2005)
- "Korean Medical Library Engine" (2011)[URL]
- ". Bethesda (MD): National Library Medicine" (2011)[URL]