TOEFL11 코퍼스에서 주제 프롬프터의 자동 분류
Automatic classification of prompts in the TOEFL11 corpus
0
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
The aim of the paper is to develop classification models of the prompts of the TOEFL essays in the TOEFL11 corpus. The corpus is a collection of TOEFL essays written in response to one of 8 prompts of various topics and by test-takers of different proficiency levels who are from 11 different countries. The number of essays is 11,000 for each language (that is, 121,000 in total). The paper aims at developing prompt classification models using an automatic method of Support Vector Machine (SVM), to which a number of different features are fed: The input features to the model include high frequency words which are observed in the raw essay texts, and high frequency nouns which are extracted from a POS-tagged essay texts. High frequency nouns among three different proficiency levels are also used as input features. The results indicated that even though high frequency words taken from raw textual materials performed quite well with an accuracy of 90.4%, the words tagged as nouns did even better with an accuracy of 97.3%. The inspecting of high frequency nouns revealed that the words were independently distributed among prompts with nearly no overlapping across different prompts. The classification test of essay samples of different proficiency levels confirmed that the accuracy rate of automatically classifying prompts by observing the frequency occurrence of nouns in texts increased in general as the proficiency levels of the essay samples increase. The paper serves as a foundation for further details studies on topic modeling used by learners of English.
Abstract
The aim of the paper is to develop classification models of the prompts of the TOEFL essays in the TOEFL11 corpus. The corpus is a collection of TOEFL essays written in response to one of 8 prompts of various topics and by test-takers of different proficiency levels who are from 11 different countries. The number of essays is 11,000 for each language (that is, 121,000 in total). The paper aims at developing prompt classification models using an automatic method of Support Vector Machine (SVM), to which a number of different features are fed: The input features to the model include high frequency words which are observed in the raw essay texts, and high frequency nouns which are extracted from a POS-tagged essay texts. High frequency nouns among three different proficiency levels are also used as input features. The results indicated that even though high frequency words taken from raw textual materials performed quite well with an accuracy of 90.4%, the words tagged as nouns did even better with an accuracy of 97.3%. The inspecting of high frequency nouns revealed that the words were independently distributed among prompts with nearly no overlapping across different prompts. The classification test of essay samples of different proficiency levels confirmed that the accuracy rate of automatically classifying prompts by observing the frequency occurrence of nouns in texts increased in general as the proficiency levels of the essay samples increase. The paper serves as a foundation for further details studies on topic modeling used by learners of English.
참고문헌 (13)
- 남춘호, "일기자료 연구에서 토픽모델링 기법의 활용가능성 검토", 비교문화연구, 22(1), pp.89~135 (2016), 비교문화연구소
- Blei, D., "Latent dirichlet allocation", The Journal of Machine Learning Research, 3, pp.993~1022 (2003)
- Blei, D. M., "The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies", Journal of the ACM (JACM), 57(2) (2010)
- Blei, D. M., "Probabilistic topic models", Communications of the ACM, 55(4), pp.77~84 (2012)
- TOEFL11: A corpus of non-native English, "Blanchard, D.", Educational Testing Service, ~, pp.ETS RR-13-24 (2013)
- Chang, C.-C., "LIBSVM: A library for support vector machines", Machine Learning, 2(3), pp.1~27 (2011)
- "Package ‘wordcloud’" (2017)[URL]
- Jockers, M. L., "Text Analysis with R for Students of Literature" (2014), Springer
- Kristina T., "Enriching the knowledge sources used in a maximum entropy part-of-speech tagger", Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP, 2000, 63~70
- Karatzoglou, A., "kernlab – an S4 package for kernel methods in R", Journal of statistical software, 11, pp.1~20 (2004)
- Newman, D., "Probabilistic Topic Decomposition of An Eighteenth-century American Newspaper", Journal off the American Society for Information Science and Technology, 57(6), pp.753~767 (2006)
- Quirk, R., "A comprehensive grammar of the English language" (1985), Harlow
- "R: A language and environment for statistical computing" (2015)[URL]