The Effect of Task Complexity on Test-takers’ Performance in a Performance-based L2 Oral Communication Test for International Teaching Assistants
1
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
The main purpose of this study is to understand the effects of task complexity on test-takers’ linguistic output and raters’ evaluation of test-takers’ performance. One hundred and fifty-six audio clips by 52 non-native-English speaking international teaching assistants (ITAs) at a U.S. university were graded by two raters with four different rating scales a holistic scale, a pronunciation scale, a vocabulary and grammar scale, and a pace and fluency scale. The clips were then transcribed and coded for linguistic complexity, accuracy, and fluency measures per AS-unit. Task complexity was evaluated based on the resource-directing task complexity criterion from Robinson’s (2001b) Triadic Componential Framework. The grades and linguistic measures were statistically analyzed using multiple Friedman tests and follow-up pairwise comparisons with Wilcoxon signed-rank tests. It was found that only one of the fluency measures, phonation-time ratio, and fluency scores, not holistic scores, with high and low task complexity, were statistically different with mid-effect sizes. In the follow-up interview, raters reported that they tended to adjust their holistic rating severity according to task complexity. Implications of the findings for rater training and evaluation rubric design are discussed.
Abstract
The main purpose of this study is to understand the effects of task complexity on test-takers’ linguistic output and raters’ evaluation of test-takers’ performance. One hundred and fifty-six audio clips by 52 non-native-English speaking international teaching assistants (ITAs) at a U.S. university were graded by two raters with four different rating scales a holistic scale, a pronunciation scale, a vocabulary and grammar scale, and a pace and fluency scale. The clips were then transcribed and coded for linguistic complexity, accuracy, and fluency measures per AS-unit. Task complexity was evaluated based on the resource-directing task complexity criterion from Robinson’s (2001b) Triadic Componential Framework. The grades and linguistic measures were statistically analyzed using multiple Friedman tests and follow-up pairwise comparisons with Wilcoxon signed-rank tests. It was found that only one of the fluency measures, phonation-time ratio, and fluency scores, not holistic scores, with high and low task complexity, were statistically different with mid-effect sizes. In the follow-up interview, raters reported that they tended to adjust their holistic rating severity according to task complexity. Implications of the findings for rater training and evaluation rubric design are discussed.
참고문헌 (59)
- Awwad, A., ""I think that’s what he’s doing" : Effects of intentional reasoning on second language(L2)speech performance", System, 67, pp.158~169 (2017)
- Bachman, L. F., "Language testing in practice : Designing and developing useful language tests" (1996), Oxford University Press
- Beaman, K., "Spoken and written language: Exploring orality and literacy", pp.45~80 (1984), Ablex
- Becker, A., "The acquisition of spatial relations in a second language (Vol. 11)" (1997), John Benjamins Publishing
- Box, G. E., "Statistics for experimenters: Design, innovation, and discovery (Vol. 2)" (2005), Wiley-Interscience
- Brown, G., "Teaching talk: Strategies for production and assessment" (1984), Cambridge University Press
- Carr, N. T., "Designing and analyzing language tests" (2011), Oxford University Press
- 장지연, "절대평가 도입 전후 수능 영어 독해 지문의 통사적 복잡성 비교", 영어교과교육, 18(2), pp.161~188 (2019), 한국영어교과교육학회[DOI]
- Cohen, J., "A coefficient of agreement for nominal scales", Educational and Psychological Measurement, 20(1), pp.37~46 (1960)
- Cohen, J., "A power primer", Psychological bulletin, 112(1), pp.155~159 (1992)
- Daller, H., "Lexical richness in the spontaneous speech of bilinguals", Applied Linguistics, 24(2), pp.197~222 (2003)
- Darus, S., "Common errors in written English essays of form one Chinese students : A case study", European Journal of Social Sciences, 10(2), pp.242~253 (2009)
- De Jong, N. H., "Praat script to detect syllable nuclei and measure speech rate automatically", Behavior Research Methods, 41(2), pp.385~390 (2009)
- Derwing, T. M., "English pronunciation and fluency development in Mandarin and Slavic speakers", System, 34(2), pp.183~193 (2006)
- Field, A., "Discovering statistics using SPSS" (2009), Sage publications
- Foster, P., "Measuring spoken language : A unit for all reasons", Applied Linguistics, 21(3), pp.354~375 (2000)
- Friedman, M., "The use of ranks to avoid the assumption of normality implicit in the analysis of variance", Journal of the American Statistical Association, 32(200), pp.675~701 (1937)
- Fulcher, G., "Task difficulty in speaking tests", Language Testing, 20(3), pp.321~344 (2003)
- Gilabert, R., "Effects of manipulating task complexity on self-repairs during L2 oral production", IRAL, 45(3), pp.215~240 (2007)
- ": Effects on L2 oral production"
- Hout, R. v., "Modelling and assessing vocabulary knowledge", pp.93~115 (2007), Cambridge University Press
- Huberty, C. J., "Multivariate analysis versus multiple univariate analyses", Psychological Bulletin, 105(2), pp.301~307 (1989)
- IBM Corp, "IBM SPSS statistics for Macintosh (Version 22)", IBM Corp, ~ (2013)
- Ishikawa, T., "The effect of task demands of intentional reasoning on L2 speech performance", The Journal of Asia TEFL, 5(1), pp.29~63 (2008)
- Jackson, D. O., "The Cognition Hypothesis : A synthesis and meta-analysis of research on second language task complexity", Language Learning, 63(2), pp.330~367 (2013)
- Kuiken, F., "Second language task complexity:Researching the Cognition Hypothesis of language learning and performance", pp.91~104 (2011), John Benjamins
- Leaper, D. A., "The influence of prompt on group oral tests", Language Testing, 31(2), pp.177~204 (2014)
- Leech, G. N., "Word frequencies in written and spoken English" (2001), Taylor & Francis
- Lennon, P., "Investigating fluency in EFL : A quantitative approach", Language Learning, 40(3), pp.387~417 (2006)
- Lunz, M. E., "Judge consistency and severity across grading periods", Evaluation & The Health Professions, 13(4), pp.425~444 (1990)
- Mackey, A., "Second language research: Methodology and design" (2016), Routledge
- McNamara, T. F., "Measuring second language performance" (1996), Longman
- Michel, M. C., "Second language task complexity: Researching the Cognition Hypothesis of language learning and performance", pp.141~173 (2011), John Benjamins
- Michel, M. C., "The influence of complexity in monologic versus dialogic tasks in Dutch L2", IRAL, 45(3), pp.241~259 (2007)
- Myford, C., "Detecting and measuring rater effects using many-facet Rasch measurement : Part II", Journal of Applied Measurement, 5(2), pp.189~227 (2004)
- Nation, P., "How large a vocabulary is needed for reading and listening?", The Canadian Modern Language Review, 63(1), pp.59~82 (2006)
- Norris, J. M., "Towards an organic approach to investigating CAF in instructed SLA : The case of complexity", Applied Linguistics, 30(4), pp.555~578 (2009)
- Ockey, G. J., "New and not so new methods for assessing oral communication", Language Value, 7(1), pp.1~21 (2015)
- Ong, J., "Effects of task complexity on the fluency and lexical complexity in EFL students’ argumentative writing", Journal of Second Language Writing, 19(4), pp.218~233 (2010)
- Plough, I. C., "A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors", Language Testing, 27(2), pp.235~260 (2010)
- Prabhu, N. S., "Second language pedagogy" (1987), Oxford University Press
- Révész, A., "Teachers’ perspectives on second language task difficulty : Insights from think-alouds and eye tracking", Annual Review of Applied Linguistics, 36, pp.182~204 (2016)
- Riggenbach, H., "Toward an understanding of fluency : A microanalysis of nonnative speaker conversations", Discourse Processes, 14(4), pp.423~441 (1991)
- Robinson, P., "Task complexity and second language narrative discourse", Language Learning, 45(1), pp.99~140 (1995)
- Robinson, P., "The cognitive hypothesis of adult task-based second language learning : Theory and research", Journal of the Korea English Education Society, 1(2), pp.1~25 (2002)
- Robinson, P., "Cognitive complexity and task sequencing : Studies in a componential framework for second language task design", IRAL, 43(1), pp.1~32 (2005)
- Robinson, P., "Second language task complexity: Researching the Cognition Hypothesis of language learning and performance", pp.3~37 (2011), John Benjamins
- Robinson, P., "Task-based language learning: A review of issues", Language Learning, 61(s1), pp.1~36 (2011)
- Saal, F. E., "Rating the ratings : Assessing the psychometric quality of rating data", Psychological Bulletin, 88(2), pp.413~428 (1980)
- Skehan, P., "A cognitive approach to language learning" (1998), Oxford University Press
- Skehan, P., "Modelling second language performance : Integrating complexity, accuracy, fluency, and lexis", Applied Linguistics, 30(4), pp.510~532 (2009)
- Skehan, P., "Task type and task processing conditions as influences on foreign language performance", Language Teaching Research, 1(3), pp.185~211 (1997)
- Towell, R., "Relative degrees of fluency : A comparative case study of advanced learners of French", IRAL, 40(2), pp.117~150 (2002)
- Towell, R., "The development of fluency in advanced learners of French", Applied Linguistics, 17(1), pp.84~119 (1996)
- Van Moere, A., "The companion to language assessment", pp.1358~1374 (2013), John Wiley & Sons, Inc
- Verma, J. P., "Repeated measures design for empirical researchers" (2015), John Wiley & Sons, Inc
- Wilcoxon, F., "Individual comparisons by ranking methods", Biometrics Bulletin, 1(6), pp.80~83 (1945)
- Won, Y., "The effect of task complexity on rater severity in an adaptive performance-based second language oral communication test", Doctor, pp.2019
- Yang, C. C. R., "Teacher questions in second language classrooms : An investigation of three case studies", Asian EFL Journal, 12(1), pp.181~201 (2010)
