본문 바로가기
KERD
논문KCI 등재

Not Yet as Native as Native Speakers: Comparing Deep Learning Predictions and Human Judgments

송상헌|유석훈|박권식
한국영어학학회 (The English Linguistics Society Of Korea)|2020.04|영어학연구 (English Language and Linguistics)|제26권 제1호|pp. 199-228|DOI 10.17960/ell.2020.26.1.010

0

조회

0

다운로드

6

인용

0

참고문헌

키워드

deep learningplausibilitynativeness judgmentlanguage experimentwell-formedness
주제분류영어와문학

초록

The purpose of this paper is to examine feasibility of replacing humans with deep learning in nativeness judgments and figure out in which way to develop the model in order to reach the level of humans by comparing nativeness judgments by deep learning and humans on English data. The controlled items, composed of 210 sentences, are categorized into two types: well-formedness test (i.e., no syntactic violation) and plausibility (i.e., no awkwardness) test items, most of which are excerpted from precedent linguistics literature. The deep learning model and five English native speakers are asked to classify the nativeness of the same stimulus sentences and the results reveal differences and similarities between them; although the overall performance of humans overwhelms that of deep learning, they are quite similar in judging plausibility items and learner data. The length of response time―hanging back from decision of nativeness―does not guarantee the accuracy, which means judging nativeness depends on something like intuition rather than deliberation.

Abstract

The purpose of this paper is to examine feasibility of replacing humans with deep learning in nativeness judgments and figure out in which way to develop the model in order to reach the level of humans by comparing nativeness judgments by deep learning and humans on English data. The controlled items, composed of 210 sentences, are categorized into two types: well-formedness test (i.e., no syntactic violation) and plausibility (i.e., no awkwardness) test items, most of which are excerpted from precedent linguistics literature. The deep learning model and five English native speakers are asked to classify the nativeness of the same stimulus sentences and the results reveal differences and similarities between them; although the overall performance of humans overwhelms that of deep learning, they are quite similar in judging plausibility items and learner data. The length of response time―hanging back from decision of nativeness―does not guarantee the accuracy, which means judging nativeness depends on something like intuition rather than deliberation.

참고문헌 (48)

  1. Ackema, P., "The Middle Construction and the Syntax-semantics Interface", Lingua, 93(1), pp.59~90 (1994)
  2. 안성호, "A Note on the Topic-Comment Stage in Korean EFL Syntactic Development", 생성문법연구, 13(2), pp.369~382 (2003), 한국생성문법학회
  3. Bernardo, A. S., "Philippine English in the ESL Classroom : A Much Closer Look", Philippines ESL Journal, 19, pp.117~144 (2017)
  5. Bublitz, W, "Semantic Prosody and Cohesive Company: Somewhat Predictable", Leuvense Bijdragen: Tijdschrift voor Germaanse Filologie, 85(1-2), pp.1~32 (1996)
  6. 최재웅, "피험자의 배경 변인이 수용성 판단에 영향을 끼치는가?", 언어와언어학, (74), pp.181~206 (2017), 언어연구소[DOI]
  7. 최인철, "Why IS ‘be’ appear there?: Topic marker vs. underdeveloped functional category", 언어연구, 30(3), pp.567~581 (2013), 언어정보연구소[DOI]
  8. Chomsky, N., "Syntactic Structures" (1957), Mouton
  9. Culbertson, J., "Are Linguists Better Subjects", The British Journal for the Philosophy of Science, 60(4), pp.721~736 (2009)
  10. Davies, M., "The TV Corpus: 325 Million Words", 1950~2018 (2019)
  11. Davies, M., "A Frequency Dictionary of Contemporary American English : Word Sketches, Collocates and Thematic Lists" (2013), Routledge
  12. Devlin, J., "Bert : Pre-training of Deep Bidirectional Transformers for Language Understanding", arXivpreprint arXiv:1810.04805, ~ (2018)
  13. Dita, S. N, "The Semantics and Grammar of Disjuncts in Philippine English", Studies in Philippine English : Exploring the ICE-Philippines, pp.33~50 (2011)
  14. Divjak, D., "The Role of Lexical Frequency in the Acceptability of Syntactic Variants : Evidence From that‐Clauses in Polish", Cognitive Science, 41(2), pp.354~382 (2017)
  15. Fraser, I. S., "Twenty-One Kicks at the Grammar Horse : Close-Up : Grammar and Composition", English Journal, 67(9), pp.49~54 (1978)
  16. Gao, L., "Latin Squares in Experimental Design", pp.2005
  17. 고광윤, "교과서 영어의 영어다움에 대한 코퍼스 기반 분석: 기능어의 쓰임을 중심으로", 영어학연구, (26), pp.1~19 (2008), 한국영어학학회[DOI]
  18. Hirsh, D., "What Vocabulary Size Is Needed to Read Unsimplified Texts for Pleasure", Reading in a Foreign Language, 8, pp.689~689 (1992)
  19. Ishikawa, S., "The ICNALE and Sophisticated Contrastive Interlanguage Analysis of Asian Learners of English", Learner Corpus Studies in Asia and the World, 1, pp.91~118 (2013)
  20. Ishikawa, S., "Learner Corpus Studies in Asia and the World 2", pp.63~76 (2014)
  21. Ishikawa, S., "The ICNALE Edited Essays : A Dataset for Analysis of L2 English Learner Essays Based on a New Integrative Viewpoint", English Corpus Studies, 25, pp.117~130 (2018)
  22. Shin’ichiro Ishikawa, "The ICNALE Spoken Dialogue: A New Dataset for the Study of Asian Learners’ Performance in L2 English Interviews", 영어교육, 74(4), pp.153~177 (2019), 한국영어교육학회
  23. Keyser, S. J., "On the Middle and Ergative Constructions in English", Linguistic Inquiry, 15(3), pp.381~416 (1984)
  24. Kondo, T., "Overpassivization in Second Language Acquisition", International Review of Applied Linguistics in Language Teaching, 43(2), pp.129~161 (2005)
  25. Kory Lauzon, "How Korean EFL Learners Comprehend Negative Frequency Adverbs", 언어사실과 관점, 43, pp.119~149 (2018), 언어정보연구원[DOI]
  26. Linzen, T., "The Reliability of Acceptability Judgments across Languages", Glossa : A Journal of General Linguistics, 3(1), pp.1~25 (2018)
  27. Louw, B., "Words in Context: A Tribute to John Sinclair on His Retirement" (2000), University of Birmingham.
  28. Mathôt, S., "OpenSesame : An Open-source, Graphical Experiment Builder for the Social Sciences", Behavior Research Methods, 44(2), pp.314~324 (2012)
  29. Mikolov, T., "Recurrent Neural Network Based Language Model", Paper presented at Eleventh Annual Conference of the International Speech Communication Association. (2010)
  30. Morales, M., "Examining the Usage of"Actually"and"in Fact"in Philippine English through a Corpus-based Analysis", Philippine ESL Journal, 10, pp.88~124 (2013)
  31. Newmeyer, F. J., "Grammar is Grammar and Usage is Usage", Language, 79(4), pp.682~707 (2003)
  32. Oh, S-Y., "Actually and in fact in American English : A Data-based Analysis", English Language and Linguistics, 4(2), pp.243~268 (2000)
  33. Park, K., "Explaining English Middle Sentences", Journal of Pan-Pacific Association of Applied Linguistics, 13(2), pp.125~140 (2009)
  34. 박권식, "한국인 영어학습자의 영어 문장은 얼마나 원어민스러운가: 딥러닝 기반 분석", 언어사실과 관점, 48, pp.195~227 (2019), 언어정보연구원[DOI]
  35. Partington, A., ""Utterly Content in Each Other’s Company"Semantic Prosody and Semantic Preference", International Journal of Corpus Linguistics, 9(1), pp.131~156 (2004)
  36. Pawley, A., "Language and Communication", pp.203~239 (2014), Routledge
  37. Sampson, G. R., "Grammar without Grammaticality", Corpus Linguistics and Linguistic Theory, 3(1), pp.1~32 (2007)
  38. Schütze, C. T., "Research Methods in Linguistics" (2014), Cambridge University Press
  39. 송상헌, "What do you mean by contrast in syntax?", 언어연구, 34(3), pp.387~426 (2017), 언어정보연구소[DOI]
  40. Stubbs, M., "Collocations and Semantic Profiles : On the Cause of the Trouble with Quantitative Studies", Functions of Language, 2(1), pp.23~55 (1995)
  41. Stubbs, M, "Words and Phrases: Corpus Studies of Lexical Semantics" (2001), Blackwell Publishers
  42. Sundermeyer, M., "LSTM Neural Networks for Language Modeling", Paper presented at Thirteenth Annual Conference of the International Speech Communication Association (2012)
  43. Thornton, R., "Plausibility and Grammatical Agreement", Journal of Memory and Language, 48(4), pp.740~759 (2003)
  44. Warstadt, A., "Neural Network Acceptability Judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
  45. Xiao, R., "Collocation, Semantic Prosody, and Near Synonymy : A Cross-linguistic Perspective", Applied Linguistics, 27(1), pp.103~129 (2006)
  46. Yang, H-K, "Categorical Properties of Korean EFL Learners’ Be", Korean Journal of English Language and Linguistics, 1(1), pp.59~79 (2001)
  47. Yang, Z., "Xlnet : Generalized Autoregressive Pretraining for Language Understanding", Advances in Neural Information Processing Systems, pp.5754~5764 (2019)
  48. Zethsen, K. K., "Semantic Prosody : Creating Awareness about a Versatile Tool", Tidsskrift for Sprogforskning, 4(1), pp.275~294 (2006)

관련 논문AI추천

한국인 영어학습자의 영어 문장은 얼마나 원어민스러운가: 딥러닝 기반 분석박권식, 송상헌, 유석훈언어정보연구원 · 언어사실과 관점2019 · 제48권 딥러닝 언어모델의한국어 학습자 말뭉치 원어민성 판단 결과 분석 연구이진, 김한샘, 정진경한국언어문화교육학회 · 언어와 문화2021 · 제17권 제1호Deep learning can contrast the minimal pairs of syntactic data박명관, 송상헌, Kwonsik Park한국현대언어학회 · 언어연구2021 · 제38권 제2호The Ability of L2 LSTM Language Models to Learn the Filler-Gap Dependency김유희한국컴퓨터정보학회 · 한국컴퓨터정보학회논문지2020 · 제25권 제11호How the L2 LSTM language model predicts island effects김유희, 최선주동서사상연구소 · 철학∙사상∙문화2021 · 제36호Already as Native as Native Speakers : Using Fine-Tuned Korean Language Models Based on Korean Language Learner Corpora송상헌, 정윤희한국언어학회 · 언어2025 · 제50권 제1호

이 간행물의 다른 논문 (10건)

Old English Short Vowels before Nasals: An Optimality-Theoretic Analysis문안나pp. 1-19존슨(Ben Jonson) 영문법서(1640)의 영문법사적 의의에 관한 연구황인태pp. 21-39Get-수동구문의 빈도 및 의미 제약에 관한 코퍼스 기반 연구이필환, 임진지pp. 41-59Significance Test of N-grams Using Bi-grams and χ2-test이용훈, 김지혜pp. 61-84한국인 학습자 코퍼스와 원어민 코퍼스에 나타나는 if-조건문의 사용양상 비교김지혜, 김지혜pp. 85-108
원문 보기