A Comparative Study Between Human Raters and ChatGPT in L2 Speaking Performance Assessment
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study aimed to compare L2 learners’ oral performance scores by human raters with ChatGPT in multiple scoring categories. It also examined score consistency within each group. A total of 39 university students participated in this study, completing various speaking tasks including a self- introduction and two picture descriptions. Statistical analyses using ANOVA and the Multi-Facets Rasch Model (MFRM) were used to analyze the scoring data from both human raters and ChatGPT. The ANOVA results revealed significant mean differences between the two rater groups across all three speaking tasks and all scoring categories (content/organization, vocabulary, collocation use, grammar), except for the category of collocation use in self-introduction. While participants’ vocabulary and collocation use showed gradual improvement in speaking tasks, substantial gains in content/organization and grammar were not observed due to time limitations. Furthermore, the MFRM results indicated that ChatGPT demonstrated higher sensitivity in discerning score patterns across different task types and scoring categories compared to the human raters. Overall, human raters consistently assigned higher scores across all task types. Although both groups scored differently, they maintained consistency in scoring, indicating agreement in their evaluation standards. Pedagogical implications pertaining to the potential advantages of integrating AI-based technology into performance-based assessments were further discussed. (201 words)
Abstract
This study aimed to compare L2 learners’ oral performance scores by human raters with ChatGPT in multiple scoring categories. It also examined score consistency within each group. A total of 39 university students participated in this study, completing various speaking tasks including a self- introduction and two picture descriptions. Statistical analyses using ANOVA and the Multi-Facets Rasch Model (MFRM) were used to analyze the scoring data from both human raters and ChatGPT. The ANOVA results revealed significant mean differences between the two rater groups across all three speaking tasks and all scoring categories (content/organization, vocabulary, collocation use, grammar), except for the category of collocation use in self-introduction. While participants’ vocabulary and collocation use showed gradual improvement in speaking tasks, substantial gains in content/organization and grammar were not observed due to time limitations. Furthermore, the MFRM results indicated that ChatGPT demonstrated higher sensitivity in discerning score patterns across different task types and scoring categories compared to the human raters. Overall, human raters consistently assigned higher scores across all task types. Although both groups scored differently, they maintained consistency in scoring, indicating agreement in their evaluation standards. Pedagogical implications pertaining to the potential advantages of integrating AI-based technology into performance-based assessments were further discussed. (201 words)
참고문헌 (34)
- Argyle, M., "The experimental analysis of social performance", Advances in Experimental Social Psychology, 3, pp.55~98 (1967)
- Baldwin, D., "Guidelines for constructed-response and other performance assessments", Educational Testing Service, ~ (2005)
- Bernstein, J., "Validating automated speaking tests", Language Testing, 27(3), pp.355~377 (2010)
- Carter, M., "Assessing technical writing in institutional contexts : Using outcomes-based assessment for programmatic thinking", Technical Communication Quarterly, 12(1), pp.101~114 (2003)
- "Common European Framework of Reference for Languages (CEFR)" (2023)[URL]
- Davis, L., "Complementary strengths? Evaluation of a hybrid human-machine scoring approach for a test of oral academic English", Assessment in Education: Principles, Policy & Practice, 28(4), pp.437~455 (2021)
- Derakhshan, A., "Developing EFL learner’s speaking ability, accuracy and fluency", English Language and Literature Studies, 6(2), pp.177~186 (2016)
- Derakhshan, A., "The difficulties of teaching English language : The relationship between research and teaching", International Journal of Linguistics, 7(1), pp.102~110 (2015)
- Dikli, S., "An overview of automated scoring of essays", The Journal of Technology, Learning and Assessment, 5(1) (2006)
- Elliot, S., "Automated essay scoring: A cross-disciplinary perspective", pp.71~86 (2013), Lawrence Erlbaum Associates
- Freedman, S. W., "Student characteristics and essay test writing performance", Research in the Teaching of English, 17(4), pp.313~325 (1983)
- Fulcher, G., "The ‘communicative’ legacy in language testing", System, 28(4), pp.483~497 (2000)
- Harder, R. J., "Teaching strategies: A guide to effective instruction" (2010), Wadsworth Publishing
- He, A. W., "Language proficiency interviews: A discourse approach", Talking and Testing: Discourse Approaches to the Assessment of Oral Proficiency, 14, pp.1~24 (1998)
- Huot, B., "Toward a new discourse of assessment for the college writing classroom", College English, 65(2), pp.163~180 (2002)
- Indramawan, A., "Improving the students’ English speaking competence through storytelling (study in Pangeran Diponegoro Islamic College (STAI) of Nganjuk, East Java, Indonesia)", International Journal of Language and Literature, 1(2), pp.18~24 (2013)
- Johnson, D. O., "Improved automatic English proficiency rating of unconstrained speech with multiple corpora", International Journal of Speech Technology, 19, pp.755~768 (2016)
- Johnson, M., "Re-analyzing the OPI : How much does it look like natural conversation", Talking and Testing : Discourse Approaches to the Assessment of Oral Proficiency, 14(2), pp.27~51 (1998)
- Kang, O., "Reverse linguistic stereotyping : Measuring the effect of listener expectations on speech evaluation", Journal of Language and Social Psychology, 28(4), pp.441~456 (2009)
- Levelt, W. J., "Accessing words in speech production: Stages, processes and representations", Cognition, 42(1-3), pp.1~22 (1992)
- McNamara, T., "Fairness, Justice and Language Assessment" (2019), Oxford University Press
- Mizumoto, A., "Exploring the potential of using an AI language model for automated essay scoring", Research Methods in Applied Linguistics, 2(2), pp.100050~ (2023)[DOI]
- Ross, S., "The discourse of accommodation in oral proficiency interviews", Studies in Second Language Acquisition, 14(2), pp.159~176 (1992)
- Rudner, L. M., "An evaluation of IntelliMetric™ essay scoring system", The Journal of Technology, Learning and Assessment, 4(4) (2006)
- Skehan, P., "Task-based instruction", Language Teaching, 36(1), pp.1~14 (2003)
- Srikaew, D., "Development of an English speaking skill assessment model for grade 6 students by using portfolio", Procedia-Social and Behavioral Sciences, 191, pp.764~768 (2015)
- Monitoring of scoring using the e-rater® automated scoring system and human raters on a writing test, "Wang, Z.", 1~21, pp.ETS Research Report Series, 2014(1) (2014)
- Williamson, D. M., "‘Mental model’ comparison of automated and human scoring", Journal of Educational Measurement, 36(2), pp.158~184 (1999)
- Williamson, D. M., "Automated scoring of complex tasks in computer-based testing" (2006), Psychology Press
- Wright, B. D., "Rating scale analysis" (1982), MESA press
- Xi, X., "The Routledge handbook of language testing", pp.452~466 (2013), Routledge
- Xu, J., "Assessing L2 English speaking using automated scoring technology: Examining automarker reliability", Assessment in Education: Principles, Policy & Practice, 28(4), pp.411~436 (2021)
- Yan, D., "Impact of ChatGPT on learners in an L2 writing practicum: An exploratory investigation", Education and Information Technologies, pp.1~25 (2023)
- Zhang, M., "Contrasting automated and human scoring of essays", R & D Connections, 21(2), pp.1~11 (2013)