Analyzing ChatGPT’s Judgments on Nativelikeness of Sentences Written by English Native Speakers and Korean EFL Learners
2
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
In this study, ChatGPT was asked to determine the likeliness of a given sentence being written by a native speaker of English. A total of 2,000 prompt sentences were provided: 1,000 from an English native-speaker corpus (LOCNESS) and 1,000 from a Korean EFL learner corpus (GLC). It was found that ChatGPT determined the sentences from the native corpus to be more nativelike than those from the nonnative corpus. The classification accuracy (proportion of native sentences classified as native and nonnative sentences classified as nonnative) was 62.75%. Most of the responses were simple phrases, such as “very likely” or “not very likely,” but in some cases, ChatGPT also provided reasons for a judgment or suggestions for correction. ChatGPT’s answers show that it considers grammaticality and plausibility as the criteria for determining the nativelikeness of a sentence, conforming to the literature. Following the initial testing, ChatGPT was fine-tuned with 1,600 sentences from the prompt data and tested with the remaining 400 sentences. After fine-tuning, the classification accuracy increased to 96.75%. This result showed that the fine-tuning of ChatGPT can enhance accuracy. Finally, the study briefly discussed the characteristics of the answers provided by GPT-4 in comparison with those provided by GPT-3.5.
Abstract
In this study, ChatGPT was asked to determine the likeliness of a given sentence being written by a native speaker of English. A total of 2,000 prompt sentences were provided: 1,000 from an English native-speaker corpus (LOCNESS) and 1,000 from a Korean EFL learner corpus (GLC). It was found that ChatGPT determined the sentences from the native corpus to be more nativelike than those from the nonnative corpus. The classification accuracy (proportion of native sentences classified as native and nonnative sentences classified as nonnative) was 62.75%. Most of the responses were simple phrases, such as “very likely” or “not very likely,” but in some cases, ChatGPT also provided reasons for a judgment or suggestions for correction. ChatGPT’s answers show that it considers grammaticality and plausibility as the criteria for determining the nativelikeness of a sentence, conforming to the literature. Following the initial testing, ChatGPT was fine-tuned with 1,600 sentences from the prompt data and tested with the remaining 400 sentences. After fine-tuning, the classification accuracy increased to 96.75%. This result showed that the fine-tuning of ChatGPT can enhance accuracy. Finally, the study briefly discussed the characteristics of the answers provided by GPT-4 in comparison with those provided by GPT-3.5.
참고문헌 (30)
- 조혜선, "Assessing Nativelikeness of Korean College Students’ English Writing Using fastText", 영어학, 22, pp.19~39 (2022), 한국영어학회[DOI]
- Cogo, A., "Efficiency in ELF communication: From pragmatic motives to lexicogrammatical innovation", Nordic Journal of English Studies, 5, pp.59~94 (2006)
- Davies, A., "The native speaker: myth and reality" (2002), Multilingual Matters
- Goldin, G., "Native language identification with user generated content", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.3591~3601 (2018)
- Goldwin-Jones, R., "Emerging spaces for language learning: AI bots, ambient intelligence, and the metaverse", Language Learning & Technology, 27(2), pp.6~27 (2023)
- Harust, O., "Native-like expression identification by contrasting native and second language speakers", Proceedings of the 28th International Conference on Computational Linguistics, pp.5843~5854 (2020)
- Jenkins, J., "The phonology of English as an international language" (2000), Oxford University Press
- Jenkins, J., "A sociolinguistically based, empirically researched pronunciation syllabus for English as an international language", Applied Linguistics, 23(1), pp.83~103 (2002)
- Joulin, A., "Bag of tricks for efficient text classification", Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2, pp.427~431 (2017)
- Lee, J. J., "The native speaker: An achievable model?", Asian EFL Journal, 7(2), pp.152~163 (2005)
- 이진, "딥러닝 언어모델의한국어 학습자 말뭉치 원어민성 판단 결과 분석 연구", 언어와 문화, 17(1), pp.155~177 (2021), 한국언어문화교육학회[DOI]
- Lee, S., "A corpus-based analysis of Korean EFL learners’ use of amplifier collocations", English Teaching, 61(1), pp.3~17 (2006)
- Linzen, T., "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)[DOI]
- Linzen, T., "Syntactic structure from deep learning", Annual Review of Linguistics, 7, pp.195~212 (2021)[DOI]
- Mahowald, K., "A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction", ~ (2023)
- Morgan-Short, K., "Second language processing shows increased native-like neural responses after months of no exposure", PLoS ONE, 7(3), pp.e32974~ (2012)[DOI]
- 남대현, "Native-speakerness: A Case Study of Two Korean-English Bilinguals", 외국어교육연구, 25(2), pp.171~193 (2011), 외국어교육연구소[DOI]
- Nation, I. S. P., "Teaching Vocabulary: Strategies and Techniques" (2008), Heinle ELT
- Owens B., "How Nature readers are using ChatGPT", Nature, 615(7950), pp.20~ (2023)[DOI]
- 박권식, "한국인 영어학습자의 영어 문장은 얼마나 원어민스러운가: 딥러닝 기반 분석", 언어사실과 관점, 48, pp.195~227 (2019), 언어정보연구원[DOI]
- 박권식, "Not Yet as Native as Native Speakers: Comparing Deep Learning Predictions and Human Judgments", 영어학연구, 26(1), pp.199~228 (2020), 한국영어학학회[DOI]
- Pawley, A., "Language and Communication", pp.191~226 (1983), Longman
- Petersen, E., "Lexical semantics with Large Language Models: A case study of English “break.”", Findings of the Association for Computational Linguistics: EACL 2023, pp.490~511 (2023)
- Robert Phillipson, "English as a lingua franca, myths and realities", 통번역학연구, 18(3), pp.1~15 (2014), 통번역연구소[DOI]
- Salvagno, M., "Can artificial intelligence help for scientific writing?", Critical Care, 27(1), pp.1~5 (2023)[DOI]
- Seidlhofer, B., "Understanding English as a lingua franca" (2011), Oxford University Press
- 신효정, "한국 대학생들의 ELF(English as a Lingua Franca) 사용 실태와 인식에 관한 연구", 교양교육연구, 15(5), pp.185~197 (2021), 한국교양교육학회
- Taecharungroj, V., "“What can ChatGPT do?” analyzing early reactions to the innovative AI chatbot on Twitter", Big Data and Cognitive Computing, 7(1), pp.35~ (2023)[DOI]
- Warstadt, A., "Algebraic Structures in Natural Language", pp.17~60 (2022)
- van Dis, E. A. M., "ChatGPT: five priorities for research", Nature, 614, pp.224~226 (2023)[DOI]