본문 바로가기
KERD
논문KCI 등재

Automated Scoring of L2 Spoken English with Random Forests

Yuichiro Kobayashi|Mariko Abe
범태평양 응용언어학회 (Pan-Pacific association of Applied Linguistics)|2016.07|Journal of Pan-Pacific Association of Applied Linguistics|제20권 제1호|pp. 55-73

0

조회

0

다운로드

3

인용

0

참고문헌

키워드

learner corpusautomated scoringrandom forestsL2 spoken English
주제분류언어학

초록

The purpose of the present study is to assess second language (L2) spoken English using automated scoring techniques. Automated scoring aims to classify a large set of learners’ oral performance data into a small number of discrete oral proficiency levels. In automated scoring, objectively measurable features such as the frequencies of lexical and grammatical items are generally used as “exploratory variables” to predict oral proficiency levels, any of which can be used as a “criterion variable” in this study. We have chosen the NICT JLE Corpus, a corpus of 1,281 Japanese EFL learners’ speech productions coded into nine oral proficiency levels (Izumi, Uchimoto, & Isahara, 2004). The nine oral proficiency levels were used as the criterion variables and linguistic features analyzed in Biber (1988) as explanatory variables. We employed random forests (Breiman, 2001), a powerful method for text classification and feature extraction, to predict oral proficiency. As a result of random forests with the out-of-bag error estimate, 60.11% of the productions were correctly classified. Compared to the baseline accuracy of the simplest possible algorithm of always choosing the most frequent level (37.63%), our random forests model improved prediction by 22.48 points. The Pearson product-moment correlation coefficient with human scoring was 0.85. Predictors that showed a clear discrimination of oral proficiency levels were tokens, types, and the frequency of nouns in the order of strength.

Abstract

The purpose of the present study is to assess second language (L2) spoken English using automated scoring techniques. Automated scoring aims to classify a large set of learners’ oral performance data into a small number of discrete oral proficiency levels. In automated scoring, objectively measurable features such as the frequencies of lexical and grammatical items are generally used as “exploratory variables” to predict oral proficiency levels, any of which can be used as a “criterion variable” in this study. We have chosen the NICT JLE Corpus, a corpus of 1,281 Japanese EFL learners’ speech productions coded into nine oral proficiency levels (Izumi, Uchimoto, & Isahara, 2004). The nine oral proficiency levels were used as the criterion variables and linguistic features analyzed in Biber (1988) as explanatory variables. We employed random forests (Breiman, 2001), a powerful method for text classification and feature extraction, to predict oral proficiency. As a result of random forests with the out-of-bag error estimate, 60.11% of the productions were correctly classified. Compared to the baseline accuracy of the simplest possible algorithm of always choosing the most frequent level (37.63%), our random forests model improved prediction by 22.48 points. The Pearson product-moment correlation coefficient with human scoring was 0.85. Predictors that showed a clear discrimination of oral proficiency levels were tokens, types, and the frequency of nouns in the order of strength.

참고문헌 (55)

  1. Abe, M., "Frequency change patterns across proficiency levels in Japanese EFL learner speech", Journal of Applied Language Studies, 8(3), pp.85~96 (2014)
  2. Abe, M., "Using multivariate statistical techniques to analyze the writing of East Asian learners of English", Learner Corpus Studies in Asia and the World, 1, pp.55~65 (2013)
  3. ACTFL-ALC Press, "Standard speaking test manual" (1996), ALC Press
  4. Attali, Y., "Handbook of automated essay evaluation: Current applications and new directions", pp.181~198 (2013), Routledge
  5. Bejar, I. I., "Automated scoring of complex tasks in computer-based testing", pp.49~81 (2006), Lawrence Erlbaum Associates
  6. Bhat, S., "Automatic assessment of syntactic complexity for spontaneous speech scoring", Speech Communication, 67, pp.42~57 (2015)
  7. Biber, D., "Variation across speech and writing" (1988), Cambridge University Press
  8. Biber, D., "Longman grammar of spoken and written English" (1999), Longman
  9. Breiman, L., "Bagging predictors", Machine Learning, 24(2), pp.123~140 (1994)
  10. Breiman, L., "Random forests", Machine Learning, 45(1), pp.5~23 (2001)
  11. "Random forests"[URL]
  12. Breiman, L., "Classification and regression trees" (1984), Chapman and Hall
  13. Burstein, J., "Automated scoring using hybrid feature identification technique", Proceedings of the 17th International Conference of Computational Linguistics, 1, pp.206~210 (1998)
  14. Chen, M., "Computing and evaluating syntactic complexity features for automated scoring of spontaneous non-native speech", Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pp.722~731 (2011)
  15. Conrad, S., "Variation in English: Multidimensional studies" (2001), Longman
  16. Downey, R., "Evaluation of the usefulness of the Versant for English Test : A response", Language Assessment Quarterly, 5(2), pp.160~167 (2008)
  17. Elliot, S., "Automated essay scoring: A cross-disciplinary perspective" (2003), Lawrence Erlbaum Associates
  18. Ericsson, P. F., "Machine scoring of student essays: Truth and consequences", pp.28~37 (2006), Utah State University Press
  19. Franco, H., "EduSpeak®: A speech recognition and pronunciation scoring toolkit for computer-aided language learning applications", Language Testing, 27(3), pp.401~418 (2010)
  20. Frignal, E., "Twenty-five years of Biber’s multi-dimensional analysis : Introduction to the special issue and an interview with Douglas Biber", Corpora, 8(2), pp.137~152 (2013)
  21. Gamon, M., "Handbook of automated essay evaluation: Current applications and new directions", pp.251~266 (2013), Routledge
  22. Hawkins, J. A., "Criterial features in L2 English : Specifying the reference levels of the Common European Framework" (2012), Cambridge University Press
  23. Higgins, D., "The Cambridge handbook of learner corpus research", pp.587~604 (2015), Cambridge University Press
  24. Hyland, K., "The author in the text: Hedging scientific writing", Hong Kong Papers in Linguistics and Language Teaching, 18, pp.33~42 (1995)
  25. Ishikawa, S., "Studies in language and text analysis", pp.11~20 (2008), University of Strathclyde Publishing
  26. Izumi, E., "A speaking corpus of 1200 Japanese learners of English" (2004), ALC Press
  27. Jin, M., "Authorship identification using random forests", Proceedings of Institute of Statistical Mathematics, 55(2), pp.255~268 (2007)
  28. Keith, T. Z., "Automated essay scoring: A cross-disciplinary perspective", pp.147~167 (2003), Lawrence Erlbaum Associates, Inc
  29. Kimura, M., "Tracing Japanese EFL learners’ development in productive vocabulary: The NICT JLE Corpus-based analysis for seeking lexical characteristics", The Proceedings of the NICT JLE Corpus Symposium, pp.54~71 (2005)
  30. Kobayashi, Y., "Twenty years of learner corpus research: Looking back, moving ahead", pp.277~287 (2013), Presses universitaires de Louvain
  31. Kobayashi, Y., "Computer-aided error analysis of L2 spoken English : A data mining approach", Proceedings of the Conference on Language and Technology, 2014, pp.127~134 (2014)
  32. Kobayashi, Y., "A machine learning approach to the effects of writing task prompts", Learner Corpus Studies in Asia and the World, 2, pp.163~175 (2014)
  33. Landauer, T. K., "The Intelligent Essay Assessor", System, 15(5), pp.27~31 (2000)
  34. Larkey, L. S., "Automated essay scoring: A cross-disciplinary perspective", pp.55~70 (2003), Lawrence Erlbaum Associates
  35. Lewis, M., "Oral proficiency self-study course for beginners" (1999), ALC Press
  36. McGee, T., "Machine scoring of student essays: Truth and consequences", pp.79~92 (2006), Utah State University Press
  37. Meunier, F., "Tracking learners’ progress: Adopting a dual ‘corpus cum experimental data’ approach", The Modern Language Journal, 97(S1), pp.61~76 (2013)
  38. Murakami, A., "A corpus-based study of English textbooks in Japan and Asian countries: Multidimensional approach", master’s, pp.2009
  39. Page, E. B., "New computer grading of student prose, using modern concepts and software", Journal of Experimental Education, 62(2), pp.127~142 (1994)
  40. Page, E. B., "Automated essay scoring: A cross-disciplinary perspective", pp.43~54 (2003), Lawrence Erlbaum Associates
  41. Pelton, T., "Mining pronunciation data for consonant cluster problems", Proceedings of International Symposium on Automatic Detection of Errors in Pronunciation Training, pp.31~36 (2012), KTH, Computer Science and Communication Department of Speech, Music and Hearing
  42. Rudner, L. M., "Automated essay scoring using Bayes’ theorem", The Journal of Technology, Learning and Assessment, 1(2), pp.3~21 (2002)
  43. Sardinha, T. B., "Multi-dimensional analysis, 25 years on: A tribute to Douglas Biber" (2014), John Benjamins
  44. Schmid, H., "Probabilistic part-of-speech tagging using decision trees", Proceedings of International Conference on New Methods in Language Processing, 12(4), pp.44~49 (1994)
  45. Shermis, M. D., "Automated essay scoring: A cross-disciplinary perspective" (2003), Routledge
  46. Shermis, M. D., "Handbook of automated essay evaluation: Current applications and new directions", pp.1~15 (2013), Routledge
  47. Shin, D., "Validation research for developing and applying the automated scoring program for the speaking section of the NEAT", Paper presented at the 35th Annual Language Testing Research Colloquium (2013)
  48. Stanley, T., "Statistical machine translation framework for modeling phonological errors in computer assisted pronunciation training system", Proceedings of International Speech Communication Association Special Interest Group Workshop on Speech and Language Technology in Education, pp.125~128 (2011)
  49. Tabata, T., "Mining textual patterns", pp.45~64 (2012), Institute of Statistical Mathematics
  50. Thewissen, J., "Capturing L2 accuracy developmental patterns: Insights from an error-tagged EFL learner corpus", The Modern Language Journal, 97(S1), pp.77~101 (2013)
  51. Williamson, D. M., "Handbook of automated essay evaluation: Current applications and new directions", pp.153~180 (2013), Routledge
  52. Xi, X., "Automated scoring and feedback systems: Where are we and where are we heading?", Language Testing, 27(3), pp.291~300 (2010)
  53. Xie, S., "Exploring content features for automated speech scoring", Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.103~111 (2012)
  54. Yoon, S., "Vocabulary profile as a measure of vocabulary sophistication", Proceedings of the 7th Workshop on Innovative Use of NLP for Building Educational Applications, pp.180~189 (2012)
  55. Zechner, K., "Automatic scoring of non-native spontaneous speech in tests of spoken English", Speech Communication, 51(10), pp.883~895 (2009)

관련 논문AI추천

Salient Linguistic Features of EFL Learner Spoken Corpus Elicited by a Computerized Speaking Test문영선, 최인철한국멀티미디어언어교육학회 · 멀티미디어 언어교육2019 · 제22권 제3호Measuring Lexical Sophistication in Written and Spoken Production by EFL Learners유현빈언어과학회 · 언어과학연구2022 · 제100호A Cross-cultural Analysis of Monologues Using Random Forest Classifier김원빈한국영어학학회 · 영어학연구2025 · 제31권 제1호Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools정연주, Danielle McNamara, Scott Crossley아시아테플 · Journal of Asia TEFL2019 · 제16권 제1호L2 발화 유창성 지표와 학습자 변인 간의 관계에 대한 머신러닝 연계 분석임우열한국언어학회 · 언어2025 · 제50권 제3호영어 말하기 자동채점의 현재와 미래박용효, 박태준, 신동광, 임수연한국멀티미디어언어교육학회 · 멀티미디어 언어교육2015 · 제18권 제1호

이 간행물의 다른 논문 (8건)

Peer/Teacher-Assessment Using Criteria in the EFL Classroom for Developing Students’ L2 Writing정미영pp. 1-20A Survey on EFL Teachers’ Assessment Methods in Entry-Level Writing Courses in Technological UniversitiesCheryl Wei-yu Chenpp. 21-36Skilled and Unskilled Japanese EFL Student Writers’ Narrative Story ProductionTaeko Kamimurapp. 37-54Exploration into the Effects of the Schema-Based Instruction: A Bottom-up ApproachKazuma Fujiipp. 75-94Discourse Connector Usage in Argumentative Essays by American and Thai University StudentsKamolphan Jangarun, Sudaporn Luksaneeyanawinpp. 95-112
원문 보기