Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model
2
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
The aim of this study was to explore the possibility of diagnosing second language (L2) learners’ writing skills through a multiple-choice examination by applying a CDM (cognitive diagnosis model). L2 learners’ writing abilities have mostly been assessed using direct methods such as essay tests. However, this method not only requires a large amount of human raters’ labor, but also faces several issues such as intra-rater and inter-rater reliabilities. Essay examinations are also hard to validate because they are limited in checking the internal structure (i.e., conducting factor analysis). To compensate for these limitations, this study developed a multiple-choice examination and validated it with factor analysis and a CDM. The examination consisted of 20 items on key L2 writing components: content, organization, grammar, vocabulary, and mechanics. It was conducted on 109 college students. The CDM analysis revealed that the test was valid based on response processes. It provided substantial information about each test-taker’s various skills. These results imply that L2 learners’ writing abilities can be assessed in an indirect way and that the test can be conducted with a large body of stu-dents to provide useful information regarding their writing abilities at the same time.
Abstract
The aim of this study was to explore the possibility of diagnosing second language (L2) learners’ writing skills through a multiple-choice examination by applying a CDM (cognitive diagnosis model). L2 learners’ writing abilities have mostly been assessed using direct methods such as essay tests. However, this method not only requires a large amount of human raters’ labor, but also faces several issues such as intra-rater and inter-rater reliabilities. Essay examinations are also hard to validate because they are limited in checking the internal structure (i.e., conducting factor analysis). To compensate for these limitations, this study developed a multiple-choice examination and validated it with factor analysis and a CDM. The examination consisted of 20 items on key L2 writing components: content, organization, grammar, vocabulary, and mechanics. It was conducted on 109 college students. The CDM analysis revealed that the test was valid based on response processes. It provided substantial information about each test-taker’s various skills. These results imply that L2 learners’ writing abilities can be assessed in an indirect way and that the test can be conducted with a large body of stu-dents to provide useful information regarding their writing abilities at the same time.
참고문헌 (35)
- Ackerman, T. A., "A comparison of the information provided by essay, multiple-choice, and free-response writing test", Applied Psychological Measurement, 12(2), pp.117~128 (1988)
- Akaike, H., "Second international symposium on information theory", pp.267~281 (1973), Akademai Kiado
- American Educational Research Association, "Standards for educational and psychological testing", American Educational Research Association, ~ (2014)
- Behizadeh, N., "Historical view of the influences of measurement and writing theories on the practice of writing assessment in the United States", Assessing Writing, 16(3), pp.189~211 (2011)
- Bejar, I., "Rater cognition : Implications for validity", Educational Measurement : Issues and Practice, 16(3), pp.189~211 (2012)
- Bozdogan, H., "Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions", Psychometrika, 52(3), pp.345~370 (1987)
- de la Torre, J., "DINA model and parameter estimation : A didactic", Journal of Educational and Behavioral Statistics, 34(1), pp.115~130 (2009)
- de la Torre, J., "The generalized DINA model framework", Psychometrika, 76(2), pp.179~199 (2011)
- Eckes, T., "Handbook of second language assessment", pp.147~164 (2016), De Gruyter Mouton
- Giraud, C., "Introduction to high-dimensional statis tics" (2015), CRC
- Hartz, S. M., "A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality", doctoral, pp.2002
- Huddleston, E. M., "Measurement of writing ability at the college-entrance level : Objective vs. subjective testing techniques", Journal of Experimental Education, 22(3), pp.165~213 (1954)
- Kim, Youn-Hee, "Diagnosing EAP writing ability using the reduced reparameterized unified model", Language Testing, 28(4), pp.509~541 (2011)
- Knoch, U., "Investigating the effectiveness of individualized feedback to rating behaviour : A longitudinal study", Language Testing, 28(2), pp.179~200 (2011)
- Lumley, T., "Assessing second language writing: The rater’s perspective" (2005), Lang
- Lumley, T., "Rater characteristics and rater bias: Implications for training", Language Testing, 12(1), pp.54~71 (1995)
- Ma, W., "GDINA : An R package for cognitive diagnosis modeling", Journal of Statistical Software, 93(14), pp.1~26 (2020)
- Shaun J. Manning, "Engagement With Automated Writing Feedback in Mandated vs. Voluntary Conditions", 현대영어교육, 20(4), pp.18~30 (2019), 현대영어교육학회[DOI]
- McDonald, R. P., "Test theory: A unified treatment" (1999), Lawrence Erlbaum Associates Publishers
- Muthén, L. K., "Mplus user’s guide" (2017), Muthén & Muthén
- Palmer, E. J., "Assessment of higher order cognitive skills in undergraduate education:Modified essay or multiple choice questions? Research paper", BMC Medical Education, 7(49), pp.1~7 (2007)
- 박찬호, "Cognitive Diagnostic Assessment of English Grammar for Korean EFL Learners", 영어교육, 66(4), pp.101~117 (2011), 한국영어교육학회[DOI]
- 박찬호, "An Exploratory Analysis of Compensatory Cognitive Diagnosis in EFL Reading Comprehension", 영어학, 17(1), pp.85~104 (2017), 한국영어학회[DOI]
- 박찬호, "Cognitive Diagnostic Writing Assessment for Korean Learners of English", 외국어교육연구, 34(2), pp.1~24 (2020), 외국어교육연구소[DOI]
- Justin Paulsen, "Examining cognitive diagnostic modeling in classroom assessment conditions", The Journal of Experimental Education, pp.1~18 (2021), Informa UK Limited[DOI]
- "R: A language and environment for statistical computing" (2021)[URL]
- "CDM: Cognitive diagnosis modeling. R package version 7.5-15" (2020)[URL]
- Rupp, A. A., "Diagnostic measurement: Theory, methods, and applications" (2010), Guilford
- Sawaki, Y., "Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessment", Language Assessment Quarterly, 6(3), pp.190~209 (2009)
- Schwarz, G., "Estimating the dimension of a model", Annals of Statistics, 6(2), pp.461~464 (1978)
- von Davier, M., "Handbook of diagnostic classification models: Models and model extensions, applications, software packages" (2019), Springer
- Wall, D., "The Routledge handbook of language testing", pp.79~92 (2012), Routledge
- Weigle, S., "Using FACETS to model rater training effects", Language Testing, 15(2), pp.263~287 (1998)
- Weigle, S., "Investigating rater", prompt interactions in writing assessment : Quantitative and qualitative approaches, 1999, pp.6(2), 145~178
- Xie, Q., "Diagnosing university students’ academic writing in English: Is cognitive diagnostic modelling the way forward?", Educational Psychology, 37(1), pp.26~47 (2016)