Grammatical Error Correction Models for Korean Language via Pre-trained Denoising
0
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
Since the era of burgeoning big data, text data have not only become increasingly accessible but also been widely applied to diverse domains. In these circumstances, adequate language processing is urgently required to handle the enormous amount of unorganized data (e.g., wrong, missing, incomplete). To deal with text data errors, varied efforts have been applied to develop grammatical error correction (GEC) models, especially for the English language. However, correction models for Korean have remained relatively unexplored. In this paper, we propose a novel GEC model specialized in Korean. Owing to the lack of training sample-label pairs (parallel corpus) in the pre-training phase, prior to training in the main stage, this model accommodates a pre-defined noise function that produces artificial errors to reinforce the previous language-correction models. For numerical study, we generate approximately 37 million training sentences and choose the case study of Korean learners’ parallel corpus, a benchmark dataset for text correction. We conclude from the study that the proposed model outperforms humans in the context of bilingual evaluation understudy scores.
Abstract
Since the era of burgeoning big data, text data have not only become increasingly accessible but also been widely applied to diverse domains. In these circumstances, adequate language processing is urgently required to handle the enormous amount of unorganized data (e.g., wrong, missing, incomplete). To deal with text data errors, varied efforts have been applied to develop grammatical error correction (GEC) models, especially for the English language. However, correction models for Korean have remained relatively unexplored. In this paper, we propose a novel GEC model specialized in Korean. Owing to the lack of training sample-label pairs (parallel corpus) in the pre-training phase, prior to training in the main stage, this model accommodates a pre-defined noise function that produces artificial errors to reinforce the previous language-correction models. For numerical study, we generate approximately 37 million training sentences and choose the case study of Korean learners’ parallel corpus, a benchmark dataset for text correction. We conclude from the study that the proposed model outperforms humans in the context of bilingual evaluation understudy scores.
참고문헌 (22)
- Sidorov G, "Rule-based system for automatic grammar correction using syntactic n-grams for english language learning (l2)", Conf. on Computational Natural Language Learning: Shared Task, pp.96~101 (2013)
- Lample G, "Phrasebased & neural unsupervised machine translation", ArXiv Preprint ArXiv:1804.07755 2018, ~ (2018)
- Peters ME, "Deep contextualized word representations", ArXiv Preprint ArXiv:1802.05365, ~ (2018)
- Devlin J, "Bert: Pre-training of deep bidirectional transformers for language understanding", ArXiv Preprint ArXiv:1810.04805, ~ (2018)
- Yang Z, "Xlnet: Generalized autoregressive pretraining for language understanding", Adv Neur In, pp.5754~5764 (2019)
- Xie Z, "Noising and denoising natural language: Diverse backtranslation for grammar correction", Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.619~628 (2018)
- Vincent P, "Extracting and composing robust features with denoising autoencoders", Int’l Conf. on Machine Learning, pp.1096~1103 (2008)
- Zhao W, "Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data", ArXiv Preprint ArXiv:1903.00138, ~ (2019)
- Lample G, "Phrasebased & neural unsupervised machine translation", ArXiv Preprint ArXiv:1804.07755, ~ (2018)
- Szegedy C, "Going deeper with convolutions", IEEE Conf. on Computer Vision and Pattern Recognition, pp.1~9 (2015)
- He K, "Deep residual learning for image recognition", IEEE Conf. on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- "Improving language understanding by generative pre-training" (2018)[URL]
- Vaswani A, "Attention is all you need", Adv Neur In, pp.5998~6008 (2017)
- Mikolov T, "Distributed representations of words and phrases and their compositionality", Adv Neur In, 2013, pp.3111~3119 (2013)
- Pennington J, "Glove: Global vectors for word representation", Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp.1532~1543 (2014)
- Ott M, "fairseq: A fast, extensible toolkit for sequence modeling", ArXiv Preprint ArXiv:1904.01038, ~ (2019)
- Papineni K, "BLEU: a method for automatic evaluation of machine translation", Annual Meeting on Association for Computational Linguistics, pp.311~318 (2002)
- Sennrich R, "Neural machine translation of rare words with subword units", ArXiv Preprint ArXiv:1508. 07909, ~ (2015)
- Kudo T, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing", ArXiv Preprint ArXiv:1808.06226, ~ (2018)
- 서윤범, "Predicting Bitcoin Market Trend with Deep Learning Models", Quantitative Bio-Science, 37(1), pp.65~71 (2018), 자연과학연구소[DOI]
- Kingma DP, "Adam: A method for stochastic optimization", ArXiv Preprint ArXiv:1412.6980, ~ (2014)
- Zhou W, "Improving Grammatical Error Correction with Machine Translation Pairs", ArXiv Preprint ArXiv:1911.02825, ~ (2019)