형태 분석을 위한 메신저 텍스트 처리 방안
Messenger Data Processing for POS Tagging
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
This study presents a methodology that facilitates UGC morphological analysis by normalizing non-standard texts, and shows that the performance of existing NLP tools can be improved through it. The KakaoTalk messenger texts were morphologically analyzed using Utagger and their error rates were checked over a total of three steps. In order to increase the accuracy of morphological analysis, it is necessary to grasp the characteristics of the messenger data. In the case of messenger data, the spacing norms are not observed, and punctuation marks are often omitted. Variants that are written as they sound or whose letters are partially modified are used, and typos are also frequent. In addition, various abbreviations are used, new words and dialects appear, and new sentence final endings are frequently used. These linguistic features can cause an error of POS and should be considered in the process of normalization, The error rate was changed from 35.1% to 17.7% after due to wrong interword space detection, and the error rate was reduced by 12.2% due to spelling checking. The remaining errors were the case of misanalyzing the sentence final endings as a connecting endings due to the absence of punctuation marks, and errors related to new proper nouns, new words, and interjections.
Abstract
This study presents a methodology that facilitates UGC morphological analysis by normalizing non-standard texts, and shows that the performance of existing NLP tools can be improved through it. The KakaoTalk messenger texts were morphologically analyzed using Utagger and their error rates were checked over a total of three steps. In order to increase the accuracy of morphological analysis, it is necessary to grasp the characteristics of the messenger data. In the case of messenger data, the spacing norms are not observed, and punctuation marks are often omitted. Variants that are written as they sound or whose letters are partially modified are used, and typos are also frequent. In addition, various abbreviations are used, new words and dialects appear, and new sentence final endings are frequently used. These linguistic features can cause an error of POS and should be considered in the process of normalization, The error rate was changed from 35.1% to 17.7% after due to wrong interword space detection, and the error rate was reduced by 12.2% due to spelling checking. The remaining errors were the case of misanalyzing the sentence final endings as a connecting endings due to the absence of punctuation marks, and errors related to new proper nouns, new words, and interjections.
참고문헌 (22)
- 김한샘, "한국어 학습자 말뭉치 주석의 요건과 실제 - 형태 분석 말뭉치 구축을 중심으로 -", 배달말, (61), pp.149~173 (2017), 배달말학회
- 민경모, "현대국어 형태 정보 주석 문어 말뭉치의 주석 단위 표기 방안에대하여", 언어사실과 관점, 12(13), pp.69~111 (2003)
- 박선우, "SNS 모바일 텍스트의 언어학적 양상: 성별과 연령의 차이를 중심으로", 현대문법연구, (82), pp.95~120 (2015), 현대문법학회[DOI]
- 안의정, "형태 분석 말뭉치 구축을 위한 한국어 구어 분석", 언어사실과 관점, 47, pp.5~24 (2019), 언어정보연구원[DOI]
- 이정복, "스마트폰 시대의 통신 언어 특징과 연구 과제", 사회언어학, 20(1), pp.177~211 (2012), 한국사회언어학회[DOI]
- 이종화, "소셜네트워크서비스에 활용할 비표준어 한글 처리 방법 연구", 한국산업정보학회논문지, 21(3), pp.35~46 (2016), 한국산업정보학회
- 최명원, "SNS 메신저 ‘카카오톡’ 언어현상 연구", 텍스트언어학, 33, pp.469~495 (2012), 한국텍스트언어학회[DOI]
- Aw, A., "A phrase based statistical model for SMS text normalization", proceedings of COLING, ACL (2006)
- Beißwenger, M., "Tagset und Richtlinie für das Part-of-Speech-Tagging von Sprachdaten aus Genres internetbasierter Kommunikation", Guideline document, ~ (2015)
- Duran, M. S., "Some issues on the normalization of a corpus of products reviews in Portuguese", proceedings of the EACL 2014 9th Web as Corpus Workshop (WAC-9), pp.22~27 (2014)
- Duran, M. S., "A Normalizer for UGC in Brazilian Portuguese", Proceedings of the ACL 2015 Workshop on Noisy User-generated Text, pp.38~47 (2015)
- Eisenstein, J., "What to do about bad language on the Internet", Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), pp.359~369 (2013)
- Han, B., "Lexical normalization for social media text", ACM Transactions on Intelligent Systems and Technology, 4(1), pp.5:1~5:27 (2013)
- Kobus, C., "Normalizing SMS: are two metaphors better than one?", Proceedings of the 22nd International Conference on Computational Linguistics, pp.441~448 (2008)
- Leech G., "Word Frequencies in Written and Spoken English" (2001), Longman
- Markus, M., "Placing Middle English in Context", pp.181~198 (2000), Mouton de Gruyter
- Panckhurst, R, "A large SMS corpus in French : from design and collation to anonymisation, transcoding and analysis", Colloque CILC2013, pp.14~16 (2013), Colloque CILC2013
- Schulz, S., "Multimodular Text Normalization of Dutch User-Generated Content", ACM Trans. Intell. Syst. Technol, 7(4), pp.61:1~61:22 (2016)
- Sproat, R., "Normalization of non-standard words. Computer", Speech and Language, 15(3), pp.287~333 (2001)
- van Deursen, A. J. A. M., "Using the internet:skill related problems in users’ online behavior", Interacting with computers, 21(5-6), pp.393~402 (2009)
- Vandekerckhove, R., "Code eclecticism: Linguistic variation and code alternation in the chat language of Flemish teenagers", Journal of Sociolinguistics, 14(5), pp.657~677 (2010)
- Xue, Z., "Normalizing microtext", Proceedings of the AAAI-11 Workshop on Analyzing Microtex, pp.74~79 (2011)