웹 말뭉치 생성을 위한 집중 웹 크롤러 구축 연구
A Study on Constructing a Focused Web Crawler for Producing Web Corpus
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
정보공유를 위한 인터넷 서비스로서 웹이 소개된 후 최근까지 방대한 양의 자료들이 웹을 통해 공개되었다. 이에 웹 공간으로부터 대규모 말뭉치를 구축하기 위한 목적으로 집중 웹 크롤러의 제작이 다양하게 시도되었다. 집중 웹 크롤러는 획득한 웹 문서를 분석하여 요청한 정보를 추출하고 사용자가 원하는 문서와 관련성이 높은 URL들을 추출, 방문함으로써 효과적인 정보 검색을 가능하게 한다. 이는 자연어 연구자가 웹에서 특정 단어 혹은 문구가 사용된 문장을 검색, 수집 및 관리에 활용할 수 있으므로, 집중 웹 크롤러는 특정 조건을 만족하는 대규모의 웹 말뭉치 구축에 적합하다. 본 연구에서 우리는 웹 말뭉치 생성 집중 웹 크롤러를 구성하는데 있어서 URL들의 크롤링 방식과 크롤링할 URL의 우선순위를 결정하는 방식이 성능에 어떤 영향을 미치는 지를 살펴보았다. 그리고 이를 고려하여 성능의 향상을 추구한 웹 말뭉치 생성 집중 웹 크롤러의 구축 방법을 제시한다. 제안된 시스템의 성능을 증명하기 위해 몇몇 어휘에 대한 말뭉치 구축을 수행했다. 실험의 결과는 본 논문에서 제안하는 말뭉치 구축 알고리즘이 기존의 방식보다 향상되었음을 보였다.
Abstract
Since the introduction of the web as an Internet service for information sharing, a vast amount of data has been made public through the web until recently. Accordingly, various attempts have been made to produce focused web crawlers for the purpose of building large-scale corpora from web space. The focused web crawler analyzes the obtained web pages to extract the requested information and enables effective information retrieval by extracting and visiting URLs that are highly relevant to the web pages the user wants. This allows natural language researchers to search, collect, and manage sentences using specific words or phrases on the web, making focused web crawlers suitable for building large-scale web corpora that meet specific conditions. In this study, we examined how the method of crawling URLs and the procedure of determining the priority of URLs to be crawled affect performance in constructing a focused web crawler for building a web corpus. Taking this into consideration, we present a method of building a web crawler focused on web corpus generation that seeks to improve performance. To prove the performance of the proposed system, corpus construction for several terms was performed. The results of the experiment showed that the corpus construction algorithm proposed in this paper is an improvement over the existing methods.
참고문헌 (19)
- "How many websites are there in the world?" (2023)[URL]
- A. Kilgarriff, "Introduction to the special issue on the web as corpus", Computational Linguistics, 29(3), pp.333~348 (2003)
- O. S. Kvashnina, "Using google to search language patterns in web-corpus: efl writing pedagogy", International Journal of Emerging Technologies in Learning (iJET), 13(3), pp.173~179 (2018)
- K. L. Nam, "Changes in the use of neologisms based on a large scale web crawled corpus-an investigation of the current state and related issues", Kolex, 29, pp.72~106 (2017)
- V. Suchomel, "Efficient web crawling for large text corpora", Proceedings of the seventh Web as Corpus Workshop, pp.39~43 (2012)
- S. W. Kim, "Focused crawler for efficient web gathering", pp.2007
- S. Chakrabarti, "Focused crawling: a new approach to topic-specific web resource discovery", proceedings of 8th International World Wide Web Conference, pp.545~562 (1999)
- H. Lu, "An improved focused crawler : using web page classification and link priority evaluation", Mathematical Problems in Engineering, 2016, pp.1~10 (2016)
- S. Shah, "Focused and deep web crawling-A review", International Journal of Computer Science and Information Technologies, 5(6), pp.7488~7492 (2014)
- D. Bhatt, "Focused web crawler", Advanced in Computer Science and Information Technology, 2(11), pp.1~6 (2015)
- N. W. Min, "Ranking hyperlinks approach for focused web crawler", International Conference on Advances in Engineering and Technology, pp.233~235 (2014)
- N. O. Kang, "A design and implementation of paragraph-based focused web crawler using semantic priority of link", Journal of Knowledge Information Technology and System, 15(6), pp.1075~1083 (2020)
- S. Remus, "Domain-specific corpus expansion with focused webcrawling", Proceedings of the Tenth International Conference on Language Resources and Evaluation, pp.3607~3611 (2016)
- R. Schafer, "Focused web corpus crawling", Proceedings of the nineth Web as Corpus Workshop, pp.9~15 (2014)
- R. Kumar, "Survery of web crawling algorithms", Advances in Vision Comupting: An International Journal (AVC), 3(3) (2016)
- "googlesearch-python 1.2.3" (2023)[URL]
- "WebCrop: The Web as Corpus" (2023)[URL]
- "spaCy" (2023)[URL]
- S. J. Park, "Paragraph-based kmeans clustering by using meaning-based paragraph division", Journal of Knowledge Information Technology and System, 12(1), pp.157~164 (2017)