소셜 미디어 텍스트의 미분석어 처리를 위한 전처리기 및 사전확장 연구
A Methodology for Building Linguistic Resources that Recognize Unanalyzed Sequences in Social Media Texts
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The Linguistic Association of Korea Journal, 25(4), 193-226. This study aims to analyze linguistic problems with unanalyzed tokens of Social Media (SM) texts and to propose methodologies for dealing with them effectively. Recently, with SM users on the rise, the need for analyzing such texts has significantly increased. However, the unanalyzed tokens severally hamper the overall performance of processing SM textual data. This study proposes two methodologies: 1) a normalizing process with a preprocessing module named Preprocessing Grammar Table (PGT) to correct frequent unanalyzed sequences such as orthographic errors and space errors; 2) a lexicon-based method utilizing DECO dictionary and Local Grammar Graph (LGG). By applying PGT and an enhanced DECO dictionary to SM texts, preprocessing performance considerably improves with 87% of the unanalyzed tokens removed, which reveals the significance of the research.
Abstract
The Linguistic Association of Korea Journal, 25(4), 193-226. This study aims to analyze linguistic problems with unanalyzed tokens of Social Media (SM) texts and to propose methodologies for dealing with them effectively. Recently, with SM users on the rise, the need for analyzing such texts has significantly increased. However, the unanalyzed tokens severally hamper the overall performance of processing SM textual data. This study proposes two methodologies: 1) a normalizing process with a preprocessing module named Preprocessing Grammar Table (PGT) to correct frequent unanalyzed sequences such as orthographic errors and space errors; 2) a lexicon-based method utilizing DECO dictionary and Local Grammar Graph (LGG). By applying PGT and an enhanced DECO dictionary to SM texts, preprocessing performance considerably improves with 87% of the unanalyzed tokens removed, which reveals the significance of the research.
참고문헌 (23)
- 김선호, "한국어 문서 처리를 위한 동적 생성 로컬 사전 기반 미등록어 분석", 정보과학회논문지 : 소프트웨어 및 응용, 29(6), pp.407~416 (2002), 한국정보과학회
- 남길임, "상품평 텍스트에 나타난 감성표현 연구: 감성분석과 국어학 연구의 접점", 언어과학연구, (78), pp.101~123 (2016), 언어과학회[DOI]
- 남지순, "Korean Electronic Dictionary DECO" (2010), 한국외국어대학교 디코라연구센터
- 남지순, "모리스 그로스의 언어처리 모델과 전산학적 적용의 이해", 인문언어, 15(1), pp.125~151 (2013), 국제언어인문학회
- 박봉래, "용례 분석에 기반한 미등록어의 인식", 정보과학회논문지, 25(2), pp.397~407 (1998)
- 박소영, "웹문서에서의 출현빈도를 이용한 한국어 미등록어 사전 자동 구축", 한국컴퓨터정보학회논문지, 13(3), pp.27~34 (2008), 한국컴퓨터정보학회
- 박영준, "현대국어의 국어사적 연구" (1994), 국학자료원
- 배주채, "교체의 개념과 조건", 國語學, (81), pp.295~324 (2017), 국어학회[DOI]
- 양장모, "이용한 한국어 미등록어 추정", 한국정보과학회봄 학술발표논문집, 23(1), pp.957~960 (1996)
- 이도길, "명사 출현 특성을 이용한 효율적인 한국어 명사 추출 방법", 정보과학회논문지 : 소프트웨어 및 응용, 30(2), pp.173~183 (2003), 한국정보과학회
- 이세희, "음절 통계를 이용한 경량화된 철자 오류 교정 모델", 한국정보과학회학술발표논문집, 36(1), pp.84~85 (2009)
- 차정원, "형태소 패턴 사전을 이용한 일반화된 미등록어 처리", 정보과학회 인공지능연구회 춘계학술대회 논문집, pp.37~42 (1997)
- Gross, M., "Finite-State Language Processing", pp.329~354 (1997), The MIT Press
- Gross, M., "A bootstraph method for constructing local grammars", Proceedings of the Symposium Comtemporary Mathematics, University of Belgrad, pp.229~250 (1999)
- Lee, S., "A Korean part-of-speech tagging system with handling unknown words", Proceedings of International Conference on Computer Processing Pacific Rim Symposium, pp.89~94 (1995)
- Liu, B., "Sentiment analysis and opinion mining" (2012), Morgan and Claypool Publishers
- Mikheev, A., "Unsupervised learning of word-category guessing rules", Proceedings of 34th ACL, pp.327~334 (1996)
- Nagata, M., "Automatic extraction of new words from Japanese texts using generalized forward-backward search", Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp.48~59 (1996)
- Nam, J., "Korean Electronic Dictionary DECO TR-2015-02, DICORA", Hankuk University of Foreign Studies, ~ (2015)
- Park, B., "A Korean corpus refining system based on automatic analysis of corpus", Proceedings of Natural Language Processing Pacific Rim Symposium, pp.89~94 (1995)
- Paumier, S., "De la reconnaissance de formes linguistiques a l’analyse syntaxique. Unpublished doctoral Dissertation, Univ. of PEMLV", doctor, pp.2003
- Weichedel, R., "Coping with ambiguity and unknown words through probabilistic models", Computational Linguistics, 19(2), pp.359~382 (1993)
- Yoo, G., "DecoTex users’ mannual DICORA-TR-2017-12. Version V01-2017MAY", Hankuk University of Foreign Studies, ~ (2017)