The Role of Raters’ Personality Traits in the Rating Process of L2 Argumentative Writing: Interactions with Argument Positions
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The present study examines the influence of raters’personality traits on the rating process of L2 argumentative writing and their interactions with argument positions. Four Korean EFL raters took a personality test and scored four essays composed by Korean EFL university students, thinking aloud their rating processes. The raters’decision-making behaviors were analyzed to investigate the role of their personality during the rating process. Results of the study reveal that the raters not only shared certain rating strategies due to the nature of scoring L2 argumentative essays but also illustrated personalityspecific variations in decision-making behaviors and this appeared to interact with the essay argument positions. A pair of raters, found to be extroverted perceivers, favored supporting argument and attempted to interpret the writer’s intention or correct linguistic errors with more emotional engagement with the writer; the other pair, the introverted judgers, preferred the counter-argument, and were highly goal-oriented and time conscious, scoring more objectively and efficiently with more foci on task completion and the quantitative aspects of writing. The raters also demonstrated variations within the pair group. The study suggests raters’personality traits can be a source of rater variation, which influences scoring validity and reliability of writing assessment.
Abstract
The present study examines the influence of raters’personality traits on the rating process of L2 argumentative writing and their interactions with argument positions. Four Korean EFL raters took a personality test and scored four essays composed by Korean EFL university students, thinking aloud their rating processes. The raters’decision-making behaviors were analyzed to investigate the role of their personality during the rating process. Results of the study reveal that the raters not only shared certain rating strategies due to the nature of scoring L2 argumentative essays but also illustrated personalityspecific variations in decision-making behaviors and this appeared to interact with the essay argument positions. A pair of raters, found to be extroverted perceivers, favored supporting argument and attempted to interpret the writer’s intention or correct linguistic errors with more emotional engagement with the writer; the other pair, the introverted judgers, preferred the counter-argument, and were highly goal-oriented and time conscious, scoring more objectively and efficiently with more foci on task completion and the quantitative aspects of writing. The raters also demonstrated variations within the pair group. The study suggests raters’personality traits can be a source of rater variation, which influences scoring validity and reliability of writing assessment.
참고문헌 (34)
- Alaei, M. M., "The impact of rater’s personality traits on holistic and analytic scores: Does genre make any difference too?", Procedia -Social and Behavioral Science, 98, pp.1240~1248 (2014)
- Barkaoui, K., "Variability in ESL essay rating processes : The role of the rating scale and rater experience", Language Assessment Quarterly, 7, pp.54~74 (2010)
- Boroujeni, A. A. J., "The impact of extroversion and introversion personality types on EFL learners’ writing ability", Theory and Practice in Language Studies, 5, pp.212~218 (2015)
- Carrell, P., "The effect of writers’ personalities and raters’personalities on the holistic evaluation of writing", Assessing Writing, 2, pp.153~190 (1995)
- Choi, Y. H., "GENOVA and FACETS analysis of an English writing test : Tasks, raters, and scales", English Teaching, 56(2), pp.125~142 (2001)
- 최연희, "Roles of Receptive and Productive Vocabulary Knowledge in L2 Writing through the Mediation of L2 Reading Ability", 영어교육, 72(1), pp.3~24 (2017), 한국영어교육학회
- 최연희, "Interactions of Writers’ and Raters’ Argument Positions in Scoring L2 Argumentative Writing", 교과교육학연구, 22(5), pp.317~325 (2018), 교과교육연구소[DOI]
- Cumming, A., "Expertise in evaluating second language compositions", Language Testing, 7, pp.31~51 (1990)
- Cumming, A., "Decision making while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86, 67~96
- Davidson, M., "Effects of ethnicity and violent content on rubric scores in writing samples", Journal of Educational Research, 93, pp.367~373 (2000)
- Hamp-Lyons, L., "Second language writing:Research insights for the classroom", pp.69~87 (1990), Cambridge University Press
- Johnson, J. S., "The influence of rater language background on writing performance assessment", Language Testing, 26, pp.485~509 (2009)
- Susie Kim, "Exploring Rater Behaviors During a Writing Assessment Discussion", 영어교육, 70(1), pp.97~121 (2015), 한국영어교육학회[DOI]
- Knoch, U., "Investigating the effectiveness of individualized feedback to rating behavior : A longitudinal study", Language Testing, 28, pp.179~200 (2011)
- Leckie, G., "Rater effects on essay scoring : A multilevel analysis of severity drift, central tendency, and rater experience", Journal of Educational Measurement, 48, pp.399~418 (2011)
- 이성용, "Rating of Korean Students’ L2 Writing: Similarities and Differences Between Native and Non-Native Raters", 교과교육학연구, 16(3), pp.629~655 (2012), 교과교육연구소[DOI]
- Lim, G. S., "Prompt and rater effects in second language writing performance assessment", doctoral, pp.2009
- Lumley, T., "Assessment criteria in a large-scale writing test: What do they really mean to the raters?", Language Testing, 19, pp.246~276 (2002)
- Lumley, T., "Rater characteristics and rater bias : Implications for training", Language Testing, 12, pp.54~71 (1995)
- Marefat, F., "Student writing, personality type of the student and the rater: Any interrelationship?", The Reading Matrix, 6, pp.116~124 (2006)
- Mayer, J. D., "Personality: A systems approach" (2007), Allyn & Bacon
- Messick, S., "Validity of psychological assessment : Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning", American Psychologist, 50, pp.741~749 (1995)
- 박수경, "The Interplay of Task, Rating Scale, and Rater Background in the Assessment of Korean EFL Students’ Writing", 영어교육, 70(2), pp.55~82 (2015), 한국영어교육학회[DOI]
- Rezaee, A. A., "Essay raters’ personality types and rater reliability", International Journal of Language Studies, 5, pp.109~122 (2011)
- Schaefer, E., "Rater bias patterns in an EFL writing assessment", Language Testing, 25, pp.465~493 (2008)
- Shohamy, E., "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76, pp.27~33 (1992)
- Weigle, S. C., "Using FACETS to model rater training effects", Language Testing, 15, pp.263~287 (1998)
- Weigle, S. C., "Investigating rater", prompt interactions in writing assessment : Quantitative and qualitative approaches, 1999, pp.6, 145~178
- Weigle, S. C., "Assessing writing" (2002), Cambridge University Press
- Weigle, S. C., "Effects of task and rater background on the evaluation of ESL student writing : A pilot study", TESOL Quarterly, 37, pp.345~354 (2003)
- Wiseman, C. S., "Rater effects : Ego engagement in rater decision-making", Assessing Writing, 17, pp.150~173 (2012)
- Yeh, S. S., "Validation of a scheme for assessing argumentative writing of middle school students", Assessing Writing, 5, pp.123~150 (1998)
- Yun, W., "How raters’ and writers’ perceptions of a topic affect the scoring of compositions", College Teaching, 51, pp.115~118 (2003)
- Zhang, J., "Same text different processing? Exploring how raters’ cognitive and meta-cognitive strategies influence rating accuracy in essay scoring", Assessing Writing, 27, pp.37~53 (2016)