본문 바로가기
KERD
논문KCI 등재

Validity Argument for Post-Entry Oral Performance Assessment for International Students in Korean Higher Education

Lee Kilryoung|Han Jiyoung
현대영어교육학회 (The Modern English Education Society)|2023.12|현대영어교육 (Modern English Education)|제24권 |pp. 91-109|DOI 10.18095/meeso.2023.24.1.91

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

test validationInternational students’ English speaking abilitydiagnostic oral performance test국제학생의 영어말하기능력영어 말하기 진단평가시험 타당도 검증

초록

With an increasing number of international students in Korean universities, communication breakdowns among international students, domestic students, and faculty members have been an important issue. However, their language ability and difficulties have received little attention. Since English is a lingua franca, this study evaluated their English oral communication ability by designing a post-entry oral performance assessment (PEOPA), conducting a pilot test, and investigating the validity of PEOPA based on Toulmin’s (2003) argument structure. Firstly, a survey was conducted with 15 international students studying at a Korean university to ask about their English use and identify the target language use (TLU) domain. Among them, 11 took pilot PEOPA. A many-Facet Rasch analysis indicated that PEOPA tasks stably assessed the oral communication ability of participants and reliably separated them into different ability levels. Raters consistently assessed performances according to a scoring rubric reflecting intended construction, with each scoring category measuring distinct aspects of oral communication ability. Along with Rasch analysis, qualitative analysis of participants’ transcribed performances provided further backing for test validity. Ensured validity of the assessment allowed us to propose the implementation of PEOPA in universities in similar contexts. Finally, the implications of findings for the diagnosis phase of PEOPA are discussed.

Abstract

With an increasing number of international students in Korean universities, communication breakdowns among international students, domestic students, and faculty members have been an important issue. However, their language ability and difficulties have received little attention. Since English is a lingua franca, this study evaluated their English oral communication ability by designing a post-entry oral performance assessment (PEOPA), conducting a pilot test, and investigating the validity of PEOPA based on Toulmin’s (2003) argument structure. Firstly, a survey was conducted with 15 international students studying at a Korean university to ask about their English use and identify the target language use (TLU) domain. Among them, 11 took pilot PEOPA. A many-Facet Rasch analysis indicated that PEOPA tasks stably assessed the oral communication ability of participants and reliably separated them into different ability levels. Raters consistently assessed performances according to a scoring rubric reflecting intended construction, with each scoring category measuring distinct aspects of oral communication ability. Along with Rasch analysis, qualitative analysis of participants’ transcribed performances provided further backing for test validity. Ensured validity of the assessment allowed us to propose the implementation of PEOPA in universities in similar contexts. Finally, the implications of findings for the diagnosis phase of PEOPA are discussed.

참고문헌 (49)

  1. Altbach, P. G., "The internationalization of higher education: Motivations and realities", Journal of Studies in International Education, 11(3-4), pp.290~305 (2007)
  2. Bachman, L. F, "Statistical analyses for language assessment" (2004), Cambridge University Press
  3. Bachman, L. F., "Building and supporting a case for test use", Language Assessment Quarterly, 2, pp.1~34 (2005)
  4. Bond, T. G., "Applying the Rasch model: Fundamental measurement in the human sciences" (2015), Routledge
  5. Bonk, W. J., "A many-facet Rasch analysis of the second language group oral discussion task", Language Testing, 20, pp.89~110 (2003)
  6. Brown, L., "The incidence of study-related stress in international students in the initial stage of the international sojourn", Journal of Studies in International Education, 12(1), pp.5~28 (2008)
  7. Campbell, J., "Asian students' voices : An empirical study of Asian students' learning experiences at a New Zealand university", Journal of Studies in International Education, 12(4), pp.375~396 (2008)
  8. Chang, S., "English medium instruction, English-enhanced instruction, or English without instruction : The affordances and constraints of linguistically responsive practices in the higher education classroom", TESOL Quarterly, 55(4), pp.1114~1135 (2021)
  9. 최현실, "외국인 유학생의 한국 대학생활 적응에 대한 연구", 인문사회과학연구, (61), pp.71~94 (2018), 인문사회과학연구소[DOI]
  10. Crowther, D., "Linguistic dimensions of second language accent and comprehensibility : Nonnative listeners’ perspectives", Journal of Second Language Pronunciation, 2(2), pp.160~182 (2016)
  11. Csizmazia, R., "Challenges and internationalization of higher education in South Korea", International Journal of Humanities and Social Science, 6(6), pp.11~18 (2019)
  12. Doe, C., "Diagnostic English language needs assessment(DELNA)", Language Testing, 31(4), pp.537~543 (2014)
  13. D'Silva, F., "Developing disciplinary discourse in a first-year engineering course : The DELNA initiative", Discourse and Writing, 2021, pp.31, 126~135
  14. Eckes, T., "Introduction to many-facet Rasch measurement: Analyzing and evaluating rater- mediated assessments" (2015), Peter Lang
  15. Jon, Jae-Eun., "Power dynamics with international students : From the perspective of domestic students in Korean higher education", Higher Education, 64(4), pp.441~454 (2012)
  16. Jon, Jae-Eun., "Realizing internationalization at home in Korean higher education", Journal of Studies in International Education, 17(4), pp.455~470 (2013)
  17. Jon, Jae-Eun., "The emergence of a regional hub : Comparing international student choices and experiences in South Korea", Higher Education, 67(5), pp.691~710 (2014)
  18. Kane, M., "Validating score interpretations and uses", Language Testing, 29(1), pp.3~17 (2012)
  19. Kane, M., "Validating the interpretations and uses of test scores", Journal of Educational Measurement, 50, pp.1~73 (2013)
  20. Kane, M., "The Routledge handbook of language testing", pp.32~47 (2021), Routledge
  21. Kasper, G., "Research methods in interlanguage pragmatics", Studies in Second Language Acquisition, 13(2), pp.215~247 (1991)
  22. Kim, Hyekyeng, "The effects of pragmatic instruction on Korean university students with different language proficiency levels", doctor, pp.2015
  23. Kim, Hyun Jung., "A qualitative analysis of rater behavior on an L2 speaking assessment", Language Assessment Quarterly, 12, pp.239~261 (2015)
  24. "Hankuk University of Foreign Studies" (2022)[URL]
  25. 이지우, "한국인 영어 학습자의 발음에 대한 영어 원어민과 한국인 채점자의 평가 행위에 관한 비교 연구", 현대영어교육, 15(1), pp.195~216 (2014), 현대영어교육학회
  26. Lee, Yong-won., "Future of diagnostic language assessment", Language Testing, 32(3), pp.295~298 (2015)
  27. Lim, G. S., "The development and maintenance of rating quality in performance writing assessment : A longitudinal study of new and experienced raters", Language Testing, 28, pp.543~560 (2011)
  28. Lunz, M. E., "Measuring the impact of judge severity on examination scores", Applied Measurement in Education, 3(4), pp.331~345 (1990)
  29. May, L., "Interactional competence in a paired speaking test : Features salient to raters", Language Assessment Quarterly, 8(2), pp.127~145 (2011)
  30. May, L., "Developing tools for learning oriented assessment of interactional competence : Bridging theory and practice", Language Testing, 37(2), pp.165~188 (2020)
  31. "Statistics of foreign students in domestic higher education institutions in 2021" (2022)[URL]
  32. Muller, A., "Addressing the English language needs of international nursing students", Journal of Academic Language and Learning, 5(2), pp.A14~A22 (2011)
  33. Monitoring sources of variability within the Test of Spoken English assessment system, "Myford, C. M.", Educational Testing Service, ~, pp.TOEFL Research Report No. 65 (2000)
  34. Ockey, G. J., "Human versus computer partner in the paired oral discussion test", Applied Linguistics, 42(5), pp.924~944 (2021)
  35. Paul, T. H., "Doing conversation analysis: A practical guide" (2007), Sage
  36. 박순영, "외국인 유학생의 대학생활 적응 - 사회적 관계를 중심으로 -", 지역사회연구, 24(2), pp.75~102 (2016), 한국지역사회학회[DOI]
  37. Poyrazli, S., "Marital status, ethnicity, academic achievement, and adjustment strains", College Student Journal, 40(4), pp.767~780 (2006)
  38. Read, J., "Identifying language needs through diagnostic assessment", Journal of English for Academic Purposes, 7(3), pp.180~190 (2008)
  39. Read, J., "Issues in post-entry language assessment in English-medium universities", Language Teaching, 48(2), pp.217~234 (2015)
  40. Roberts, P. A., "Towards a reframing of student support : A case study approach", Higher Education, 75(1), pp.19~33 (2018)
  41. Roever, C., "What scores from monologic speaking tests can(not)tell us about interactional competence", Language Testing, 39(1), pp.7~29 (2022)
  42. Roever, C., "Speaking in turns and sequences : Interactional competence as a target construct in testing speaking", Language Testing, 35(3), pp.331~355 (2018)
  43. Saito, K., "Multi-or single-word units? The role of collocation use in comprehensible and contextually appropriate second language speech", Language Learning, 70(2), pp.548~588 (2020)
  44. Taguchi, N, "Critical reflections on data in second language acquisition", pp.7~32 (2018), John Benjamin
  45. Thibault, N, "Student-faculty interaction experiences of international students in Korea: A phenomenology study", doctor, pp.2022
  46. Toulmin, S. E., "The uses of argument" (2003), Cambridge University Press
  47. Youn, Soo Jung., "Validity argument for assessing L2 pragmatics in interaction using mixed methods", Language Testing, 32(2), pp.199~225 (2015)
  48. Yu, B., "Asian international students at an Australian university : Mapping the paths between integrative motivation, competence in L2 communication, cross-cultural adaptation and persistence with structural equation modeling", Journal of Multilingual and Multicultural Development, 34, pp.727~742 (2013)
  49. Yu, B., "Language and interpersonal resource predictors of psychological and sociocultural adaptation : International students in Hong Kong", Journal of Studies in International Education, 23(5), pp.572~588 (2019)

관련 논문AI추천

한국어 말하기 평가 타당화를 위한 논거 기반 접근법의 이해와 적용이성준한국화법학회 · 화법연구2018 · 제41호Effects of Examinees’ Perceptions of Interlocutor Proficiency on Paired Oral Assessment Results이희경, Hyun-kyu Choi아시아테플 · Journal of Asia TEFL2021 · 제18권 제1호Improving the Validity of L2 Performance Assessments: Use of Many-Facet Rasch Measurement김현정외국어교육연구소 · 외국어교육연구2017 · 제31권 제3호A Case Study of the Use of an In-house Oral Exam in a University English Program유경아언어과학회 · 언어과학연구2013 · 제65호The Validation Process in Developing a Web-based English Speaking and Writing Test김정태한국멀티미디어언어교육학회 · 멀티미디어 언어교육2011 · 제14권 제2호A Study of the Korean English Teachers’ Rating Tendency to the Oral English Proficiency Tests강석한한국영어영문학회 · 영어영문학2016 · 제21권 제4호

이 간행물의 다른 논문 (21건)

Analysis of Passives in English-to-Korean Machine Translation Systems’ Output김채은pp. 1-9A Washback Effect Model for Investigating the Impact of High-Stakes Language Testing on Teaching and LearningHan Gil Kimpp. 10-25AI펭톡 활용 수업이 초등학생 영어 발음 능력 향상과 정의적 영역에 미치는 효과우미혜, 김지연pp. 26-44초등예비교사의 교실 담화에 나타난 상호작용 유형에 관한 연구김현진pp. 45-61한국 EFL 고등학생의 문학동아리 역할 수행 연구:영어 능력과 태도 변화를 중심으로최윤하pp. 62-76
원문 보기