Analysis of rater effect in the evaluation of second language grammatical knowledge in the context of writing: Application of a generalized linear model
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Despite ample testing research involving the impact of rater background characteristics (e.g. raters’ native language) on assessment, relatively little has been known about how raters’ linguistic knowledge influences their scoring decision-making process. By associating second language acquisition research with regard to two types of linguistic knowledge (i.e. implicit and explicit knowledge), this study aims to explore the influence of raters’ linguistic knowledge and other rater factors (including teaching experience) on second language writing assessment, particularly in relation to the grammatical aspect of performance. Forty-two raters’ grammatical knowledge was measured with an untimed paper-and-pencil grammar test (explicit knowledge) and a timed computer-based experiment (implicit knowledge). The raters also responded to a background questionnaire. For scoring, the raters evaluated five examinees’ responses on two writing tasks (a personal essay and an argumentative essay), based on a six-point analytic scoring rubric. An analysis of a cumulative link mixed model revealed that the raters’ scores could be significantly related to their grammatical knowledge and teaching experience. In addition, different effects emerged across the examinees’ ability levels. These findings contribute to a better understanding of the predictors with respect to rating performance and provide practical implications for rater selection and training in second language performance assessment.
Abstract
Despite ample testing research involving the impact of rater background characteristics (e.g. raters’ native language) on assessment, relatively little has been known about how raters’ linguistic knowledge influences their scoring decision-making process. By associating second language acquisition research with regard to two types of linguistic knowledge (i.e. implicit and explicit knowledge), this study aims to explore the influence of raters’ linguistic knowledge and other rater factors (including teaching experience) on second language writing assessment, particularly in relation to the grammatical aspect of performance. Forty-two raters’ grammatical knowledge was measured with an untimed paper-and-pencil grammar test (explicit knowledge) and a timed computer-based experiment (implicit knowledge). The raters also responded to a background questionnaire. For scoring, the raters evaluated five examinees’ responses on two writing tasks (a personal essay and an argumentative essay), based on a six-point analytic scoring rubric. An analysis of a cumulative link mixed model revealed that the raters’ scores could be significantly related to their grammatical knowledge and teaching experience. In addition, different effects emerged across the examinees’ ability levels. These findings contribute to a better understanding of the predictors with respect to rating performance and provide practical implications for rater selection and training in second language performance assessment.
참고문헌 (70)
- Standards for educational and psychological testing, "American Educational Research Association", American Educational Research Association, ~ (1999)
- Bachman, Lyle F., "Fundamental considerations in language testing" (1990), Oxford University Press
- Bachman, Lyle F, "Some reflections on task-based language performance assessment", Language Testing, 19(4), pp.453~476 (2002)
- Bachman, Lyle F, "Statistical analyses for language assessment" (2004), Cambridge University Press
- Barrett, Steven, "The impact of training on rater variability", International Education Journal, 2(1), pp.49~58 (2001)
- Bialystok, Ellen, "On the relationship between knowing and using linguistic forms", Applied Linguistics, 3(3), pp.181~206 (1982)
- Brown, James D, "Do English and ESL faculties rate writing samples differently?", TESOL Quarterly, 25(4), pp.587~603 (1991)
- Cho, Dongwan, "A study of ESL writing assessment: Intra-rater reliability of ESL compositions", Melbourne Papers in Language Testing, 8(1), pp.1~24 (1999)
- Chomsky, Noam, "Aspects of the theory of syntax" (1965), The MIT Press
- Christensen, Rune H. B., "Analysis of ordinal data with cumulative link models-estimation with the R-package ordinal", R package version 2013, ~ (2013)
- Cumming, Alister, "Expertise in evaluating second language compositions", Language Testing, 7(1), pp.31~51 (1990)
- Cumming, Alister, "Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters’ decision making and development of a preliminary analytic framework (TOEFL Monograph Series, MS-22)" (2000), Educational Testing Service
- Cumming, Alister, "Decision making while rating ESL", EFL writing tasks: A descriptive framework, 2002, pp.86(1), 67~96
- De Boeck, Paul, "Explanatory item response models: A generalized linear and nonlinear approach" (2004), Springer
- Dienes, Zoltan, "A theory of implicit and explicit knowledge", Behavioral and Brain Sciences, 22(5), pp.735~808 (1999)
- Doran Harold, "Estimating the multilevel Rasch model: With the lme4 package", Journal of Statistical Software, 20(2), pp.1~18 (2007)
- Douglas, Dan, "Quantity and quality in speaking test performance", Language Testing, 11(2), pp.125~144 (1994)
- Du, Yurong, "Differential facet functioning detection in direct writing assessment", Paper presented at the Annual Meeting of the American Educational Research Association (1996)
- Dunbar, Stephen B., "Quality control in the development and use of performance assessments", Applied Measurement in Education, 4(4), pp.289~304 (1991)
- Eckes, Thomas, "Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis", Language Assessment Quarterly, 2(3), pp.197~221 (2005)
- Ellis, Rod, "Measuring implicit and explicit knowledge of a second language: A psychometric study", Studies in Second Language Acquisition, 27(2), pp.144~159 (2005)
- Ellis, Rod, "Modelling learning difficulty and second language proficiency: The differential contributions of implicit and explicit knowledge", Applied Linguistics, 27(3), pp.432~455 (2006)
- Ellis, Rod, "Implicit and explicit knowledge in second language learning, testing and teaching" (2009), Multilingual Matters
- Engelhard Jr, George, "Examining rater errors in the assessment of written composition with a many-faceted Rasch model", Journal of Educational Measurement, 31(2), pp.93~112 (1994)
- Green, Peter S, "Implicit and explicit grammar: An empirical study", Applied Linguistics, 13(2), pp.168~184 (1992)
- Gregg, Kevin R, "The state of emergentism in second language acquisition", Second Language Research, 19(2), pp.95~128 (2003)
- Hamp-Lyons, Liz, "Assessment of writing: Policies, politics, practice" (1996), Modern Language Association
- Hamp-Lyons, Liz, "The Englishes of English tests: Bias revisited", World Englishes, 27(1), pp.26~39 (2008)
- Han, Youngju, "Implicit knowledge, explicit knowledge and general language proficiency", Language Teaching Research, 2(1), pp.1~23 (1998)
- Hu, Guangwei, "Psychological constraints on the utility of metalinguistic knowledge in second language production", Studies in Second Language Acquisition, 24(3), pp.347~386 (2002)
- 장주현, "Comparing two types of explicit pronunciation instructions on second language accentedness", 언어연구, 32, pp.15~32 (2015), 언어정보연구소[DOI]
- Johnson, Jeff S, "The influence of rater language background on writing performance assessment", Language Testing, 26(4), pp.485~505 (2009)
- Johnson, Robert L., "The relation between score resolution methods and interrater reliability: An empirical study of an analytic rating rubric", Applied Measurement in Education, 13(2), pp.121~138 (2000)
- Kenyon, Dorry, "Introductory remarks at symposium on development and use of rating scales in language testing", Paper presented at the 14th Language Testing Research Colloquium (1992)
- Kim, Hyun Jung, "Investigating raters’ development of rating ability on a second language speaking assessment", pp.2011
- 김선영, "ESL College Learners’ Interactive Perspective and Its Influence on Reading-Writing Practices and Development", 언어연구, 34, pp.1~24 (2017), 언어정보연구소[DOI]
- Kim, Youn-Hee, "An investigation into native and non-native teachers’ judgements of oral English performance: A mixed methods approach", Language Testing, 26(2), pp.187~217 (2009)
- Kondo-Brown, Kimi, "A FACETS analysis of rater bias in measuring Japanese L2 writing performance", Language Testing, 19(1), pp.3~31 (2002)
- Directions for the revisions of the national English ability test and English national curriculum, "Korea Institute for Curriculum and Evaluation", Korea Institute for Curriculum and Evaluation, ~ (2011)
- 이준규, "The Activations of Relational Structures in Processing Second Language Noun-noun Compound", 언어연구, 28(1), pp.143~157 (2011), 언어정보연구소[DOI]
- Lim, Gad S, "The development and maintenance of rating quality in performance writing assessment: A longitudinal study of new and experienced raters", Language Testing, 28(4), pp.543~560 (2011)
- Linacre, John M., "Many-faceted Rasch measurement" (1989), MESA Press
- Lumley, Tom, "Assessment criteria in a large-scale writing test: What do they really mean to the raters?", Language Testing, 19(3), pp.246~276 (2002)
- Lumley, Tom, "Assessing second language writing: The rater’s perspective" (2005), Peter Lang
- Lumley, Tom, "Rater characteristics and rater bias:Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- McNamara, Tim F, "Modelling performance: Opening pandora’s box", Applied Linguistics, 16(2), pp.159~179 (1995)
- McNamara, Tim F., "Measuring second language performance" (1996), Longman
- McNamara, Tim F, "‘Interaction’ in second language performance assessment: Whose performance?", Applied Linguistics, 18(4), pp.446~466 (1997)
- Meiron, Beryl E, "Fairness and validation in language assessment" (2000), Cambridge University Press
- Milanovic, Michael, "Studies in language testing 3: Performance testing, cognition and assessment" (1996), Cambridge University Press
- An introduction to the elementary English national curriculum, "Ministry of Education, Science and Technology", Ministry of Education, Science and Technology, ~ (2011)
- Popham, James, "Modern educational measurement: A practitioner’s perspective" (1990), Prentice-Hall
- Reed, Daniel J, "Studies in language testing 11: Experimenting with uncertainty: Essays in honor of Allan Davies" (2001), Cambridge University Press
- Rijmen, Frank, "A nonlinear mixed model framework for item response theory", Psychological Methods, 8(2), pp.185~205 (2003)
- Rumelhart, David E, "Parallel distributed processing: Explorations in the microstructure of cognition. Volume I" (1986), The MIT Press
- Sakyi, Alfred A, "Fairness and validation in language assessment" (2000), Cambridge University Press
- Schaefer, Edward, "Rater bias patterns in an EFL writing assessment", Language Testing, 25(4), pp.465~493 (2008)
- Schoonen, Rob, "The assessment of writing ability: Expert readers versus lay readers", Language Testing, 14(2), pp.157~184 (1997)
- Shi, Ling, "Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing", Language Testing, 18(3), pp.303~325 (2001)
- Shohamy, Elana, "The effect of raters’ background and training on the reliability of direct writing tests", The Modern Language Journal, 76(1), pp.27~33 (1992)
- Smith, David, "Studies in immigrant English language assessment (vol. 1)" (2000), National Centre for English Language Teaching and Research, Macquarie University
- Sweedler-Brown, Carol O, "ESL essay evaluation: The influence of sentence-level and rhetorical features", Journal of Second Language Writing, 2(1), pp.3~17 (1993)
- Vaughan, Christopher L, "Assessing second language writing in academic contexts" (1991), Ablex
- Wallace, Michael J, "Training foreign language teachers - A reflective approach" (1991), Cambridge University Press
- Weigle, Sara Cushing, "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Wigglesworth, Gillian, "Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction", Language Testing, 10(3), pp.305~335 (1993)
- Wigglesworth, Gillian, "Patterns of rater behaviour in the assessment of an oral interaction test", Australian Review of Applied Linguistics, 17(2), pp.77~103 (1994)
- Wiseman, Cynthia S, "Investigating selected facets in measuring second language writing ability using holistic and analytic scoring methods", pp.2008
- Wolfe, Edward W., "Cognitive differences in proficient and nonproficient essay scorers", Written Communication, 15(4), pp.465~492 (1998)
- Zhang, Ying, "Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?", Language Testing, 28(1), pp.31~50 (2011)