The Interplay of Task, Rating Scale, and Rater Background in the Assessment of Korean EFL Students’ Writing
2
조회
0
다운로드
6
인용
0
참고문헌
키워드
초록
This study examines how native English speaking (NES) and Korean non-native English speaking (KNES) teachers assess L2 writing performance. More specifically, this study aims to investigate whether these two groups of raters evaluate writing samples differently when using different rating scales (holistic vs. analytic) and different task types (narrative vs. argumentative). Four NES and four KNES raters evaluated 78 narrative and 78 argumentative essays written by Korean EFL university students using both holistic and analytic rating rubrics. The comparison between the two rater groups indicated that the scores given by the two groups were statistically significantly different for both holistic and analytic ratings regardless of the two task types investigated. Overall, KNES teachers rated the essays more harshly than their NES counterparts, irrespective of task type and rating scale. Multiple regression analysis of five analytic sub-criteria revealed that the two rater groups demonstrated similar patterns in assessing argumentative essays, while for narrative essays, the relative influence of each analytic sub-criterion on overall writing quality differed for the two rater groups. Implications for L2 writing assessment are included.
Abstract
This study examines how native English speaking (NES) and Korean non-native English speaking (KNES) teachers assess L2 writing performance. More specifically, this study aims to investigate whether these two groups of raters evaluate writing samples differently when using different rating scales (holistic vs. analytic) and different task types (narrative vs. argumentative). Four NES and four KNES raters evaluated 78 narrative and 78 argumentative essays written by Korean EFL university students using both holistic and analytic rating rubrics. The comparison between the two rater groups indicated that the scores given by the two groups were statistically significantly different for both holistic and analytic ratings regardless of the two task types investigated. Overall, KNES teachers rated the essays more harshly than their NES counterparts, irrespective of task type and rating scale. Multiple regression analysis of five analytic sub-criteria revealed that the two rater groups demonstrated similar patterns in assessing argumentative essays, while for narrative essays, the relative influence of each analytic sub-criterion on overall writing quality differed for the two rater groups. Implications for L2 writing assessment are included.
참고문헌 (65)
- Bouwer, R., "Effect of genre on the generalizability of writing scores", Language Testing, 32(1), pp.83~100 (2014)
- Braddock, R., "Research in written composition." (1963), National Council of Teachers of English
- Carr, N. T., "A comparison of the effects of analytic and holistic rating scale types in the context of composition tests", Issues in Applied Linguistics, 11(2), pp.207~241 (2000)
- Carrell, P. L., "Reading and writing descriptive and persuasive texts", The Modern Language Journal, 75(3), pp.314~324 (1991)
- "An investigation of the impact of the 6+1 trait writing model on grade 5 student writing achievement final report" (2011)[URL]
- Connor-Linton, J., "Crosscultural comparison of writing standards : American ESL and Japanese EFL", World Englishes, 14(1), pp.99~115 (1995)
- Crowhurst, M., "Cohesion in argument and narration at three grade levels", Research in the Teaching of English, 21(2), pp.185~201 (1987)
- Crowhurst, M., "Research review: Patterns of development in writing persuasive", argumentative discourse, 1988
- Crowhurst, M., "Teaching and learning the writing of persuasive", argumentative discourse, 1990, pp.15(4), 348~359
- Cumming, A., "Scoring TOEFL essays and TOEFL 2000 prototype written tasks: An investigation into raters’ decision making and development of a preliminary analytic framework. (TOEFL Monograph Series No. 22)" (2001), Educational Testing Service
- Dunbar, S. B., "Quality control in the development and use of performance assessments", Applied Measurement in Education, 4(4), pp.289~303 (1991)
- Eckes, T., "Rater types in writing performance assessments : A classification approach to rater variability", Language Testing, 25(2), pp.155~185 (2008)
- Engelhard, Jr. G., "The influences of mode of discourse, experiential demand, and gender on the quality of student writing", Research in the Teaching of English, 26(3), pp.315~335 (1992)
- Foster, P., "The influence of planning and task type on second language performance", Studies in Second Language Acquisition, 18(3), pp.299~323 (1996)
- Freedman, A., "Writing in the college years : Some indices of growth", College Composition and Communication, 31(3), pp.311~324 (1984)
- Hafner, J. C., "Quantitative analysis of the rubric as an assessment tool : An empirical study of student peer-group rating", International Journal of Science Education, 25(12), pp.1509~1528 (2003)
- Hake, R., "Writing assessment: Issues and strategies", pp.153~167 (1986), Longman
- Hamp-Lyons, L., "Second language writing: Research insights for the classroom", pp.69~87 (1990), Cambridge University Press
- Hamp-Lyons, L., "Assessing second language writing in academic contexts", pp.87~107 (1991), Albex
- Hamp-Lyons, L., "Rating nonnative writing : The trouble with holistic scoring", TESOL Quarterly, 29(4), pp.759~762 (1995)
- Hamp-Lyons, L., "Exploring the dynamics of second language writing", pp.162~189 (2003), Cambridge University Press
- Hinkel, E, "The effects of essay topics on modal verb uses in L1 and L2 academic writing", Journal of Pragmatics, 41(4), pp.667~683 (2009)
- "Personal narrative collection"[URL]
- 장소영, "The Development of a Standardized Rater Training Program on Essay Scoring: A Case Study", 영어교육, 66(3), pp.201~231 (2011), 한국영어교육학회
- Johnson, S. J., "The influence of rater language background on writing performance assessment", Language Testing, 26(4), pp.485~505 (2009)
- Kegley, P. H., "The effect of mode of discourse on student writing performance : Implications for policy", Educational Evaluation and Policy Analysis, 8(2), pp.147~154 (1986)
- 김아영, "Scoring Behavior of Native vs. Non-native Speaker Raters of Writing Exams", 어학연구, 48(2), pp.319~342 (2012), 언어교육원
- Knoch, U., "Little coherence, considerable strain for reader : A comparison between two rating scales for the assessment of coherence", Assessing Writing, 12(2), pp.108~128 (2007)
- Knoch, U., "Diagnostic assessment of writing : A comparison of two rating scales", Language Testing, 26(2), pp.275~304 (2009)
- Knoch, U., "Investigating the effectiveness of individualized feedback to rating behavior—a longitudinal study", Language Testing, 28(2), pp.179~200 (2011)
- 이희경, "Native and nonnative rater behavior in grading Korean students' English essays", Asia Pacific Education Review, 10(3), pp.387~397 (2009), 교육연구소[DOI]
- Markham, L. R., "Influences of handwriting quality on teacher evaluation of written work", American Educational Research Journal, 13(4), pp.277~283 (1976)
- Matsuhashi, A., "Pausing and planning : The tempo of written discourse production", Research in the Teaching of English, 15(2), pp.113~134 (1981)
- McNamara, T. F., "Modelling performance : Opening Pandora’s box", Applied Linguistics, 16(2), pp.159~179 (1995)
- McNamara, T. F., "Measuring second language performance." (1996), Addison Wesley Longman
- Messick, S., "The interplay of evidence and consequences in the validation of performance assessments", Educational Researcher, 23(2), pp.13~23 (1994)
- Miller, M. D., "Alternative assessment in a high-stakes environment", Educational Measurement : Issues and Practice, 12(2), pp.9~15 (1993)
- Norment, M., "Contrastive analyses of organizational structures and cohesive elements in native and ESL Chinese, English, and Spanish writing", doctor, pp.1984
- Norris, J. M., "Designing second language performance assessments." (1998), University of Hawaii Press
- O’Loughlin, K., "The assessment of writing by English and ESL teachers", Australian Review of Applied Linguistics, 17(1), pp.23~44 (1994)
- O’Loughlin, K., "The impact of gender in oral proficiency testing", Language Testing, 19(2), pp.169~192 (2002)
- 박수경, "Lexical Analysis of Korean University Students’ Narrative and Argumentative Essays", 영어교육, 68(3), pp.131~157 (2013), 한국영어교육학회
- Soo-Kyung Park, "Korean EFL Learner s’ Use of Cohesive Devices in Narrative and Argumentative Essays", Studies in English Education, 18(2), pp.51~81 (2013), 글로벌영어교육학회
- Pollitt, A., "Calibrating graded assessments : Rasch partial credit analysis of performance in writing", Language Testing, 4(1), pp.72~92 (1987)
- Quellmalz, E. S., "Effects of discourse and response mode on the measurement of writing competence", Journal of Educational Measurement, 19(4), pp.241~258 (1982)
- Reed, D., "Experimenting with uncertainty: Language testing in honour of Alan Davies (Vol. 11)", pp.82~96 (2001), Cambridge University Press
- Rezaei, A. R., "Reliability and validity of rubrics for assessment through writing", Assessing Writing, 15(1), pp.18~39 (2010)
- Rigdon, E., "The performance of the polychoric correlation coefficient and selected fitting functions in confirmatory factor analysis with ordinal data", Journal of Marketing Research, 28(4), pp.491~497 (1991)
- Santos, T., "Professors’ reactions to the academic writing of nonnative-speaking students", TESOL Quarterly, 22(1), pp.69~90 (1988)
- Shi, L., "Native-and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18(3), pp.303~325 (2001)
- Sinclair, V. E., "The effect of mode variation on syntactic complexity in adult ESL composition writing", doctor, pp.1984
- Sloan, C., "The effect of handwriting on teachers’ grading of high school essays", Journal of Association for the Study of Perception, 17(2), pp.15~21 (1982)
- Stiggins, R. J., "Revitalizing classroom assessment : The highest instructional priority", Phi Delta Kappan, 69(5), pp.363~368 (1988)
- Tajeddin, Z., "Pragmatic rater training: Does it affect non-native L2teachers’ rating accuracy and bias?", Iranian Journal of Language Testing, 4(1), pp.66~83 (2014)
- Tarone, E., "Task‐related variation in interlanguage : The case of articles", Language Learning, 38(1), pp.21~44 (1988)
- "Report of the task force on writing assessment" (2009)[URL]
- "The R statistical environment. The R project for statistical computing" (2011)[URL]
- Vaughan, C, "Assessing second language writing in academic contexts", pp.11~125 (1991), Ablex
- Weigle, S. C., "Effects of training on raters of ESL compositions: Quantitative and qualitative approaches", doctor, pp.1994
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 5(2), pp.263~287 (1998)
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- Weir, C. J., "Communicative language testing." (1990), Prentice Hall
- White, E. M., "Holisticism", College Composition and Communication, 35(4), pp.400~409 (1984)
- Yang, H. C., "Second language writers’ strategy use and performance in an integrated reading‐listening‐writing task", TESOL Quarterly, 46(1), pp.80~103 (2012)
- Yau, M., "The influence of mode on the syntactic complexity of EFL students at three grade levels", TESL Canada Journal, 2(1), pp.65~76 (1984)
