초급 수준 대학생의 영어 능력 평가를 위한 챗봇 기반 말하기 평가 실행
Implementing a Chatbot-Based Speaking Assessment of Beginner College Students’ English Language Proficiency
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
There is a lack of research on the use of the AI voice chatbot in speaking tests. This study compared ESWT (English Speaking & Writing Test) and CEST (Chatbot-based English Speaking Test) scores, analyzed the impact of interactional competence on the wholistic scores of CEST, and examined test-takers’ perceptions of item difficulties. Twenty-seven Korean university students, primarily at novice English language proficiency levels, participated in the study. They took both ESWT and CEST followed by surveys asking their self-perceived item difficulty for each test. The results of paired ttest showed that ESWT yielded higher mean scores across all analytical constructs and holistic scores with significant differences in pronunciation and coherence. A Ridge regression revealed that the interactivity and fluency constructs of CEST strongly influenced its wholistic score, highlighting the potential of AI chatbots to enhance EFL learners' speaking abilities. Participants, however, perceived CEST as more challenging than ESWT, pointing out the need for more preparation and response time due to their low proficiency levels. Regarding item difficulty, read-aloud and selfintroduction tasks were considered the easiest, while opinion-giving and long consecutive questions were the most challenging in ESWT. Participants also found all items challenging overall, reflecting their limited language competence and underscoring the necessity of targeted speaking skill development. Based on these findings, the study presents discussions and practical implications for integrating AI chatbots into speaking assessment.
Abstract
There is a lack of research on the use of the AI voice chatbot in speaking tests. This study compared ESWT (English Speaking & Writing Test) and CEST (Chatbot-based English Speaking Test) scores, analyzed the impact of interactional competence on the wholistic scores of CEST, and examined test-takers’ perceptions of item difficulties. Twenty-seven Korean university students, primarily at novice English language proficiency levels, participated in the study. They took both ESWT and CEST followed by surveys asking their self-perceived item difficulty for each test. The results of paired ttest showed that ESWT yielded higher mean scores across all analytical constructs and holistic scores with significant differences in pronunciation and coherence. A Ridge regression revealed that the interactivity and fluency constructs of CEST strongly influenced its wholistic score, highlighting the potential of AI chatbots to enhance EFL learners' speaking abilities. Participants, however, perceived CEST as more challenging than ESWT, pointing out the need for more preparation and response time due to their low proficiency levels. Regarding item difficulty, read-aloud and selfintroduction tasks were considered the easiest, while opinion-giving and long consecutive questions were the most challenging in ESWT. Participants also found all items challenging overall, reflecting their limited language competence and underscoring the necessity of targeted speaking skill development. Based on these findings, the study presents discussions and practical implications for integrating AI chatbots into speaking assessment.
참고문헌 (38)
- ACTFL, "ACTFL proficiency guidelines 2024", ACTFL Language Connects, ~ (2024)
- Bachman, L. F., "Language testing in practice: Designing and developing useful language tests" (1996), Oxford University Press
- Cha, S., "An analysis of research trends in AI chatbot studies related to English education", English Language & Literature Teaching, 20(1), pp.203~225 (2021)[DOI]
- Chalhoub-Deville, M., "Second language interaction : Current perspectives and future trends", Language Testing, 20(4), pp.369~383 (2003)[DOI]
- Choi, W., "Process-based speaking assessment using AI chatbots in elementary English education : Possibilities and limitations", Primary English Education, 26(1), pp.131~152 (2020)[DOI]
- Council of Europe, "Common European Framework of Reference for Languages: Learning, teaching, assessment-companion volume" (2020), Council of Europe Publishing
- Ducasse, A. M., "Interaction in paired oral proficiency assessment in Spanish", doctor, pp.2008
- Ducasse, A. M., "Assessing paired orals : Raters’ orientation to interaction", Language Testing, 26(3), pp.423~443 (2009)[DOI]
- Elder, C., "Planning for test performance : Does it make a difference", Language Testing, 22(3), pp.239~263 (2005)[DOI]
- Fors, K. L., "Production and perception of pauses in speech", Doctor, pp.2015
- Fryer, L., "Bots as language learning tools", Language Learning & Technology, 10(3), pp.8~14 (2006)[DOI]
- Galaczi, E. D., "Peer-peer interaction in a speaking test : The case of the first certificate in English examination", Language Assessment Quarterly, 5(2), pp.89~119 (2008)[DOI]
- Gass, S. M., "Second language acquisition: An introductory course" (2020), Routledge
- Hall, J. K., "Recreating our worlds with words : A sociohistorical perspective of face-to-face interaction", Applied Linguistics, 16, pp.206~232 (1995)[DOI]
- He, A. W., "Talking and testing: Discourse approaches to the assessment of oral proficiency", pp.1~24 (1998), John Benjamins
- "IELTS Speaking: Band descriptors (public version)" (2022)[URL]
- Jhajaria, S., "Study and comparative analysis of chatGPT, GPT and DAll-E2", Proceedings of 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT), pp.1~5 (2023), IEEE
- Kim, H., "A study of AI chatbot as an assistant tool for school English curriculum", Journal of Learner-Centered Curriculum and Instruction, 19(1), pp.89~110 (2019)[DOI]
- Kim, H. -S., "Effects of AI chatbots on EFL students’ communication skills", Korean Journal of English Language and Linguistics, 21, pp.712~734 (2021)[DOI]
- Kim, N. -Y., "Effects of voice chat on EFL learners’ speaking ability according to proficiency levels", Multimedia-Assisted Language Learning, 19(4), pp.63~88 (2016)[DOI]
- Kim, J., "Development of task types and scoring constructs of computer-based English speaking test", Multimedia-Assisted Language Learning, 22(3), pp.150~178 (2019)[DOI]
- Kramsch, C., "From language proficiency to interactional competence", The Modern Language Journal, 28, pp.171~183 (1986)
- Krashen, S. D., "The input hypothesis: Issues and implications" (1985), Longman
- Lee, J., "A study on improvement plans through discourse analysis of conversational chatbots for foreign language learning", Multimedia-Assisted Language Learning, 22(1), pp.132~153 (2019)[DOI]
- May, L., "Interactional competence in a paired speaking test : Features salient to raters", Language Assessment Quarterly, 8(2), pp.127~145 (2011)[DOI]
- Nguyen, Q. N., "Understanding user interactions with a chatbot: A selfdetermination theory approach", the Americas Conference on Information Systems (2018)
- O’Sullivan, B., "Language testing: Theories and practices", pp.164~185 (2011), Palgrave Macmillan
- Price, J. M, "The perceived effect of pause length and location on speaker likability and communicative effectiveness", master, pp.2021
- Seo, S., "A meta-analysis on the effects of using AI chatbots in Korean English education", Foreign Languages Education, 30(3), pp.25~42 (2023)[DOI]
- Song, B., "Interactional competence in a paired speaking test : Toward designing an evidencebased rating scale", The Journal of English Education, 28(2), pp.133~152 (2016)[DOI]
- Sun, H., "English learning through real-time text-based conversations with an AI chatbot : A case study of Korean English learners", Modern English Education, 25, pp.115~127 (2024)
- Swain, M., "Input in second language acquisition" (1985), Newbury House
- Tavakoli, P., "Strategic planning, task structure, and performance testing", Applied Linguistics, 26(3), pp.375~400 (2005)[DOI]
- Wigglesworth, G., "An investigation of planning time and proficiency level on oral test discourse", Language Testing, 14(1), pp.85~106 (1997)[DOI]
- Yang, H., "Implementation of an AI chatbot as an English conversation partner in EFL speaking classes", ReCALL, 34(3), pp.327~343 (2022)[DOI]
- Yeon, Y., "A study on interaction patterns in AI chatbot-assisted classes: Focusing on elementary English speaking lessons", master, pp.2023
- Yoon, T., "The effectiveness of using EBS AI PengTalk for elementary learners' English learning", Journal of English Teaching through Movies and Media, 23(3), pp.27~38 (2022)[DOI]
- Young, R. F., "Interactional competence: Challenges for validity", the Annual meeting of the American Association for Applied Linguistics (2000)