Developing and Validating Empirically-Derived Diagnostic Descriptors in ESL Academic Writing
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Despite the increasing interest in and need for developing empirical and diagnostic assessment schemes in L2 assessment and testing, very little research has been devoted to it. The literature in this area is scant, and only a handful of studies (e.g., Fulcher, 1993; Knoch, 2009; North & Schneider, 1998; Upshur & Turner, 1995) have attempted to explore such possibility. In response to this need for research, this two-phase study developed and validated empirically-derived diagnostic descriptors for use in small-scale classroom assessment in English as a second language (ESL) academic writing. In Phase 1, descriptors that are relevant to the construct of ESL academic writing were developed, and in Phase 2, the use of the descriptors was validated through the multiple analyses. The results indicated that 35 empirically-derived descriptors addressed all aspects of ESL writing skills, including content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. In addition, teachers were able to use the descriptors in a consistent manner. These findings demonstrate the effectiveness of an empirical approach to assessment scheme development, and underscore the importance of its use for diagnostic purposes.
Abstract
Despite the increasing interest in and need for developing empirical and diagnostic assessment schemes in L2 assessment and testing, very little research has been devoted to it. The literature in this area is scant, and only a handful of studies (e.g., Fulcher, 1993; Knoch, 2009; North & Schneider, 1998; Upshur & Turner, 1995) have attempted to explore such possibility. In response to this need for research, this two-phase study developed and validated empirically-derived diagnostic descriptors for use in small-scale classroom assessment in English as a second language (ESL) academic writing. In Phase 1, descriptors that are relevant to the construct of ESL academic writing were developed, and in Phase 2, the use of the descriptors was validated through the multiple analyses. The results indicated that 35 empirically-derived descriptors addressed all aspects of ESL writing skills, including content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. In addition, teachers were able to use the descriptors in a consistent manner. These findings demonstrate the effectiveness of an empirical approach to assessment scheme development, and underscore the importance of its use for diagnostic purposes.
참고문헌 (59)
- Alderson, J. C., "Language testing in the 1990s", pp.71~86 (1991), Macmillan
- Barkaoui, K., "Effects of scoring method and rater experience on ESL essay rating processes and outcomes", doctoral, pp.2008
- Brindley, G., "Interfaces between second language acquisition and language testing research", pp.112~140 (1998), Cambridge University Press
- Brown, J. D., "A categorical instrument for scoring second language writing skills", Language Learning, 34, pp.21~42 (1984)
- Canale, M., "Language and communication", pp.2~27 (1983), Longman
- Chalhoub Deville, M., "Theoretical models, assessment fra meworks and test construction", Language Testing, 14, pp.16~33 (1997)
- Cohen, A. D., "Second language writing: Research insights for the classroom", pp.155~177 (1990), Cambridge University Press
- Cumming, A., "Expertise in evaluating second language composit ions", Language Testing, 7, pp.31~51 (1990)
- Cumming, A., "On second language writing", pp.209~229 (2001), Lawrence Erlbaum
- TOEFL 2000 writing framework: A working paper. TOEFL Monograph Series, "Cumming, A.", Educational Testing Service, ~, pp.Report No. 18 (2000)
- Cumming, A., "Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters decision making and development of a preliminary analytic framework" (2001), Educational Testing Service
- Cumming, A., "Decision mak ing while rating ESL", EFL writing tasks : A descriptive framework, 2002, pp.86, 67~96
- de Jong, J., "Rating scales and listening comprehension", Australian Review of Applied Linguistics, 11, pp.73~87 (1988)
- Engber, C., "The relationsh ip of lexical proficiency to the quality of ESL compositions", Journal of Second Language Writing, 4, pp.139~155 (1995)
- Ericsson, K. A., "Protocol analysis: Verbal reports as data" (1993), MIT Press
- Ferris, D., "Student reaction s to teacher response in multiple draft composition classrooms", TESOL Quarterly, 29, pp.3~53 (1995)
- Fulcher, G., "The construction and validation of rating scales for oral tests in English as a foreign language", doctoral, pp.1993
- Fulcher, G., "Testing second language speaking" (2003), Pearson Longman
- Glaser, B., "The discovery of grounded theory: Strategies for qualitative research" (1967), Aldine
- Grabe, W., "Theory and practice of writing" (1996), Longman
- Grove, E., "Tasks and criteria in a test of oral communication skills for first year health science students", Melbourne Papers in Language Testing, 10, pp.37~47 (2001)
- Halliday, M. A. K., "Cohesion in English" (1976), Longman
- HampLyons, L., "Communicative writing profiles: An investigation of the transferability of a multiple trait scoring instrument across ESL writing assessment contexts", Languag e Learning, 41, pp.337~373 (1991)
- Hedgcock, J., "Feedback on feedback: Assessing learner receptivity to teacher response in L2 composing", Journal of Second Language Writing, pp.141~163 (1994)
- Jacobs, H., "Testing ESL composition: A practical approach" (1981), Newbury House
- Jang, E. E., "Demystifying a Q Matrix for making diagnostic inferences about L2 reading skills", Language Assessment Quarterly, 6, pp.210~238 (2009)
- Kepner, C., "An experiment in the relationship of types of writt en feedback to the development of second language writing skills", The Modern Language Journal, 75, pp.305~313 (1991)
- Kim, Y H., "An argument based validity inquiry into the Empirically derived Descripto r based Diagnostic (EDD) assessment in ESL academic writing", doctoral, pp.2010
- Kim, Y H., "Diagnosing EAP writing ability using the Reduced Reparameterized Unified Model", Language Testing, 28, pp.509~541 (2011)
- Knoch, U., "Diagnostic a ssessment of w riting: The development and validation of a rating scale" (2009), Peter Lang
- Lantolf, J. P., "Oral proficiency testing : A critical analysis", The Modern Language Journal, 69, pp.337~345 (1985)
- Laufer, B., "The development of L2 lexis in the expression of the advanced learner", The Modern Language Journal, 75, pp.440~448 (1991)
- Laufer, B., "Vocabulary size and use : Lexical richness in L2 written production", Applied Linguistics, 16, pp.307~322 (1995)
- Lee, Y W., "Cognitive diagnosis and Q matrices in language assessment", Language Assessment Quarterly, 6, pp.169~171 (2009)
- Lee, Y W., "Cognitive diagnosis approaches to language assessment : An overview", Lang uage Assessment Quarterly, 6, pp.172~189 (2009)
- Leki, I., "The preferences of ESL students for error correction in college level writing classes", Foreign Language Annals, 24, pp.203~218 (1991)
- Lumley, T., "Assessment criteria in a large scale writing test: What do they really mean to the raters?", Language Testing, 19, pp.246~276 (2002)
- Lumley, T., "Assessing second language writing: The raters’ perspective" (2005), Peter Lang
- Lunz, M. E., "Judge severity and consistency across grading periods", Evaluation and the Health Professions, 13, pp.425~444 (1990)
- Luoma, S., "Assessing speaking" (2004), Cambridge University Press
- Matthews, M., "The measurement of productive skills: Doubts concerning the assessment criteria of certain public examinations", ELT Journal, pp.117~121 (1990)
- Milanovic, M., "Studies in language testing 3:Performance testing, cognition and assessment", pp.92~111 (1996), Cambridge University Press
- Mullen, K. A., "On TESOL 77:Teaching and learning English as a second language: Trends in research and practice", pp.309320~ (1977), TESO
- Myford, C. M., "Introduction to Rasch measurement", pp.460~517 (2004), JAM Press
- The development of descriptors on scales of language proficiency, "North, B.", National Foreign Language Center, ~ (1993)
- North, B., "The development of a common framework scale of descriptors of language proficiency based on a theory of measurement", System, 23, pp.445~465 (1995)
- North, B., "The development of a common framework scale of descriptors of language proficiency based on a theory of measurement", doctoral, pp.1996
- North, B., "Scaling descriptors for language proficiency scales", Language Testing, 15, pp.217~263 (1998)
- Pienemann, M., "Constructing an acquisition based procedure for second language assessm ent", Studies in Se cond Language Acquisition, 10, pp.217~243 (1988)
- Polio, C., "On second language writing", pp.91~116 (2001), Erlbaum
- Pollitt, A., "Studies in language testing 3: Performance testing, cognition and assessment", pp.74~91 (1996), Cambridge University Press
- Sakyi, A. A., "Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions", Fairn ess and validation in language assessment: Selected papers from the 19th Language Te sting Research Colloquium, pp.129~121 (2000), Cambridge University Press
- Sawaki, Y., "Q matrix construct ion : Defining the link between constructs and test items in large scale reading and listening comprehension assessments", Language Assessment Quarterly, 6, pp.172~189 (2009)
- Smith, D., "Studies in immigrant English language assessment", pp.159~189 (2000), National Centr e for English Language Teaching and Research, Macquarie University
- Turner, C. E., "The language testing cycle: From in ception to washback", pp.55~79 (1996), Australian Review of Applied Linguistics
- Upshur, J. A., "Constructing rating scales for second language tests", ELT Journal, 49, pp.3~12 (1995)
- Vaughan, C., "Assessing second language writing in academic co ntexts", pp.111~125 (1991), Ablex Publishing Corporation
- Second language development in w riting: Measures of fluency, accuracy and complexity, "Wolfe Quintero, K.", University of Hawai’i Press, ~, pp.Technical Report No. 17 (1998)