소규모 시험데이터에 대한 베이지언 인지진단모형의 적용
An Application of Bayesian Cognitive Diagnostic Modeling to Small Size Data
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Recently various specific and general cognitive diagnostic models(CDM) have been used in modeling data from language tests. CDMs have huge potential to provide learners and teachers with fine-grained diagnostic information about their learning and teaching beyond traditional total scores. Despite its advantages, current CDM applications in language testing are confined to analyses of large-sample data from national or testing agencies for educational decision making. For CDMs to be useful in practical classroom assessment contexts, they should be able to overcome constraints due to small sample conditions. Meanwhile the bayesian estimation procedures open a new path for modeling complex entities over the frequentist approaches: It does not depend on large-sample theory; it is flexible and robust in handling complicated models; it can incorporate extra information about the model into processing data; and it provides reasonable model-data fit statistics. However, bayesian approaches to CDMs are not well known to researchers in language testing as well as in SLA. This study introduces the bayesian approach to cognitive diagnostic modeling by applying the new method to a set of data from a listening comprehension test developed for measuring high school students’ listening ability. The study then compares the results with those from the conventional method and discusses its implications in modeling CDMs in language testing.
Abstract
Recently various specific and general cognitive diagnostic models(CDM) have been used in modeling data from language tests. CDMs have huge potential to provide learners and teachers with fine-grained diagnostic information about their learning and teaching beyond traditional total scores. Despite its advantages, current CDM applications in language testing are confined to analyses of large-sample data from national or testing agencies for educational decision making. For CDMs to be useful in practical classroom assessment contexts, they should be able to overcome constraints due to small sample conditions. Meanwhile the bayesian estimation procedures open a new path for modeling complex entities over the frequentist approaches: It does not depend on large-sample theory; it is flexible and robust in handling complicated models; it can incorporate extra information about the model into processing data; and it provides reasonable model-data fit statistics. However, bayesian approaches to CDMs are not well known to researchers in language testing as well as in SLA. This study introduces the bayesian approach to cognitive diagnostic modeling by applying the new method to a set of data from a listening comprehension test developed for measuring high school students’ listening ability. The study then compares the results with those from the conventional method and discusses its implications in modeling CDMs in language testing.
참고문헌 (45)
- Alderson, J. C., "Diagnosing Foreign Language Proficiency: The Interface between Learning and Assessment" (2005), Continuum
- Alderson, J. C., "Towards a Theory of Diagnosis in Second and Foreign Language Assessment: Insights from Professional Practice Across Diverse Fields", Applied Linguistics, 36(2), pp.236~260 (2015)
- T. Ando, "Bayesian predictive information criterion for the evaluation of hierarchical Bayesian and empirical Bayes models", Biometrika, 94(2), pp.443~458 (2007), Oxford University Press (OUP)[DOI]
- Vahid Aryadoust, "A Cognitive Diagnostic Assessment Study of the Listening Test of the Singapore–Cambridge General Certificate of Education O-Level: Application of DINA, DINO, G-DINA, HO-DINA, and RRUM", International Journal of Listening, 35(1), pp.29~52 (2018), Informa UK Limited[DOI]
- Buck, G., "The Subskills of Reading: Rule Space Analysis of a Multiple‐Choice Test of Second Language Rading Comprehension", Language Learning, 47(3), pp.423~466 (1997)
- Culpepper, S. A., "Bayesian Estimation of the DINA Model with Gibbs Sampling", Journal of Educational and Behavioral Statistics, 40(5), pp.454~476 (2015)[DOI]
- De la Torre, J., "The Generalized DINA Model Framework", Psychometrika, 76(2), pp.179~199 (2011)
- De la Torre, J., "Model Evaluation and Multiple Strategies in Cognitive Diagnosis : An Analysis of Fraction Subtraction Data", Psychometrika, 73(4), pp.595~624 (2008)
- De La Torre, J., "Higher-order Latent Trait Models for Cognitive Diagnosis", Psychometrika, 69(3), pp.333~353 (2004)
- Jimmy de la Torre, "Analysis of Clinical Data From a Cognitive Diagnosis Modeling Framework", Measurement and Evaluation in Counseling and Development, 51(4), pp.281~296 (2017), Informa UK Limited[DOI]
- DiBello, L. V., "Cognitively Diagnostic Assessment", pp.361~389 (1995), Erlbaum
- Dong, Y., "An Optimal Choice of Cognitive Diagnostic Model for Second Language Listening Comprehension Test", Frontiers in Psychology, 12 (2021)[DOI]
- Wenbo Du, "Probing what’s behind the test score: application of multi-CDM to diagnose EFL learners’ reading performance", Reading and Writing, 34(6), pp.1441~1466 (2021), Springer Science and Business Media LLC[DOI]
- W. Holmes Finch, "Using Fit Statistic Differences to Determine the Optimal Number of Factors to Retain in an Exploratory Factor Analysis", Educational and Psychological Measurement, 80(2), pp.217~241 (2019), SAGE Publications[DOI]
- Fox, J. P., "Bayesian Item Response Modeling: Theory and Applications" (2010), Springer
- Luke Harding, "Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles", Language Testing, 32(3), pp.317~336 (2015), SAGE Publications[DOI]
- Hartz, S. M., "A Bayesian Framework for the Unified Model for Assessing Cognitive Abilities: Blending Theory with Practicality", Ph. D., pp.2002
- Eli Hinkel, "Current Perspectives on Teaching the Four Skills", TESOL Quarterly, 40(1), pp.109~ (2006), Wiley[DOI]
- Henson, R., "Defining a Family of Cognitive Diagnosis Models Using Loglinear Models with Latent Variables", Psychometrika, 74(2), pp.191~210 (2009)
- Huang, H-Y., "Diagnostic Classification Model for Forced-Choice Items and Noncognitive Tests", Educational and Psychological Measurement, 83(1), pp.146~180 (2023)
- Jang, E. E., "A Review of Cognitive Diagnostic Assessment for Education : Theory and Application", International Journal of Testing, 8(3), pp.290~295 (2008)
- Matthew S. Johnson, "Measures of Agreement to Assess Attribute‐Level Classification Accuracy and Consistency for Cognitive Diagnostic Assessments", Journal of Educational Measurement, 55(4), pp.635~664 (2018), Wiley[DOI]
- Junker, B. W., "Cognitive Assessment Models with Few Assumptions and Connections with Nonparametric Item Response Theory", Applied Psychological Measurement, 25(3), pp.258~272 (2001)
- Kim, A. Y. (Alicia), "Exploring Ways to Provide Diagnostic Feedback with an ESL Placement Test: Cognitive Diagnostic Assessment of L2 Reading Ability", Language Testing, 32(2), pp.227~258 (2015)
- 김성훈, "Q-행렬에 근거하는 치유교습 순서", 교육평가연구, 33(2), pp.353~376 (2020), 한국교육평가학회
- Yong-Won Lee, "Cognitive Diagnosis and Q-Matrices in Language Assessment", Language Assessment Quarterly, 6(3), pp.169~171 (2009), Informa UK Limited[DOI]
- Lee, S. Y., "Basic and Advanced Bayesian Structural Equation Modeling: With Applications in the Medical and Behavioral Sciences" (2012), John Wiley and Sons
- Lee, M. D., "Bayesian Cognitive Modeling:A Practical Course" (2014), Cambridge University Press
- Leighton, J. P., "Cognitive Diagnostic Assessment for Education: Theory and Applications" (2007), Cambridge University Press
- Levy, R., "Bayesian Psychometric Modeling", pp.2017, Chapman and Hall
- Liu, Y., "An Application of M2 Statistic to Evaluate the Fit of Cognitive Diagnostic Models", Journal of Educational and Behavioral Statistics, 41(1), pp.3~26 (2016)
- Rupp, A. A., "The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the DINA Model", Educational and Psychological Measurement, 68(1), pp.78~96 (2008)
- Rupp, A. A., "Diagnostic Measurement:Theory, Methods, and Applications" (2010), Guilford Press
- Sawaki, Y., "Q-Matrix Construction : Defining the Link between Constructs and Test Items in Large-scale Reading and Listening Comprehension Assessments", Language Assessment Quarterly, 6(3), pp.190~209 (2009)
- Sessoms, J., "Applications of Diagnostic Classification Models : A Literature Review and Critical Commentary", Measurement : Interdisciplinary Research and Perspectives, 16(1), pp.1~17 (2018)
- Sinharay, S., "Assessing Fit of Cognitive Diagnostic Models A Case Study", Educational and Psychological Measurement, 67(2), pp.239~257 (2007)
- Sorrel, M. A., "Inferential Item-fit Evaluation in Cognitive Diagnosis Modeling", Applied psychological measurement, 41(8), pp.614~631 (2017)
- Stone, C. A., "Bayesian Analysis of Item Response Theory Models Using SAS" (2015), SAS Institute Inc
- Jonathan L. Templin, "Measurement of psychological disorders using cognitive diagnosis models.", Psychological Methods, 11(3), pp.287~305 (2006), American Psychological Association (APA)[DOI]
- von Davier, M., "Handbook of Diagnostic Classification Models: Models and Model Extensions, Applications, Software Packages", pp.1~17 (2019)
- Yeon-Sook Yi, "In Search of Optimal Cognitive Diagnostic Model(s) for ESL Grammar Test Data", Applied Measurement in Education, 30(2), pp.82~101 (2017), Informa UK Limited[DOI]
- Larry Vandergrift, "Recent developments in second and foreign language listening comprehension research", Language Teaching, 40(3), pp.191~210 (2007), Cambridge University Press (CUP)[DOI]
- Ling Ling Wang, "An Improved Parameter-Estimating Method in Bayesian Networks Applied for Cognitive Diagnosis Assessment", Frontiers in Psychology, 12 (2021), Frontiers Media SA[DOI]
- Watanabe, S, "Asymptotic Equivalence of Bayes across Validation and Widely Applicable Information Criterion in Singular Learning Theory", Journal of Machine Learning Research, 11(116), pp.3571~3594 (2010)
- Zhan, P., "Using JAGS for Bayesian Cognitive Diagnosis Modeling : A Tutorial", Journal of Educational and Behavioral Statistics, 44(4), pp.473~503 (2019)[DOI]