본문 바로가기
KERD
논문KCI 등재

베이지언 네트워크에 기반한 인지진단모형의 영어 읽기 시험에 대한 적용

Application of bayesian network-based cognitive diagnostic modeling to small sample English reading comprehension test data.

최세일
사단법인 한국언어학회 (The Linguistic Society of Korea)|2023.12|언어학 (EONEOHAG)|제31권 제4호|pp. 83-111|DOI 10.24303/lakdoi.2023.31.4.83

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

인지진단모형classification accuracy베이지언 네트워크bayesian networkscognitive diagnostic models진단정보문항모수분류정확도diagnostic informationitemparameters
주제분류언어학

초록

Cognitive diagnostic models (CDMs), a family of classification models developed to provide fine-grained diagnostic information for learning and teaching in education, have increasingly been used in language testing. However, most of the previous CDM studies in language testing have mainly been conducted based on large samples from professional testing agencies. This trend makes it difficult for practitioners to apply the models in classroom assessment contexts for which the models were originally developed. Realizing this limitation, researchers working in CDMs have recently begun to turn their attention to the conditions in which CDMs can work for classroom assessments, especially with small sample sizes. Bayesian networks (BN) provide an efficient and intuitive framework for modeling complex systems of observable or latent variables and have been extensively employed in the data science as well as in intelligent tutoring systems for modeling students’ learning progress. The framework has also huge potential to be well suited for diagnostic modeling of students’ learning in classroom contexts. This study was to examine whether BN can be applied in cognitive diagnostic modeling for classroom assessments. After constructing a set of small test data (N=100, 150, 200) from a large real test, the study applied a BN-based CDM model to the data sets and compared with conventional CDMs its item parameter and attribute classification recovery. The results show that the BN-based CDMs yielded uniformly better estimates in all testing conditions than the conventional methods. The study then discusses its implications for the CDM applications in language testing.

Abstract

Cognitive diagnostic models (CDMs), a family of classification models developed to provide fine-grained diagnostic information for learning and teaching in education, have increasingly been used in language testing. However, most of the previous CDM studies in language testing have mainly been conducted based on large samples from professional testing agencies. This trend makes it difficult for practitioners to apply the models in classroom assessment contexts for which the models were originally developed. Realizing this limitation, researchers working in CDMs have recently begun to turn their attention to the conditions in which CDMs can work for classroom assessments, especially with small sample sizes. Bayesian networks (BN) provide an efficient and intuitive framework for modeling complex systems of observable or latent variables and have been extensively employed in the data science as well as in intelligent tutoring systems for modeling students’ learning progress. The framework has also huge potential to be well suited for diagnostic modeling of students’ learning in classroom contexts. This study was to examine whether BN can be applied in cognitive diagnostic modeling for classroom assessments. After constructing a set of small test data (N=100, 150, 200) from a large real test, the study applied a BN-based CDM model to the data sets and compared with conventional CDMs its item parameter and attribute classification recovery. The results show that the BN-based CDMs yielded uniformly better estimates in all testing conditions than the conventional methods. The study then discusses its implications for the CDM applications in language testing.

참고문헌 (79)

  1. Afflerbach, P., "Reading: What else matters besides strategies and skills?", The Reading Teacher, 66(6), pp.440~448 (2013)
  2. Akbay, L., "Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured", Psicothema, 32(1), pp.122~129 (2020)
  3. Alderson J. C., "Testing reading comprehension skills(Part Two)", Reading in a Foreign Language, 7, pp.465~503 (1990)
  4. Alderson J. C., "Testing reading comprehension skills(Part One)", Reading in a Foreign Language, 6, pp.425~438 (1990)
  5. Alderson J. C., "Assessing reading" (2000), Cambridge University Press
  6. Alderson, J. C., "Assessing reading" (2005), Cambridge University Press
  7. Alderson, J. C., "Cognition and reading: cognitive levels as embodied in test questions", Reading in a Foreign Language, 5, pp.253~270 (1989)
  8. Almond, R. G., "Bayesian networks in educational assessment" (2015), Springer
  9. Bernhardt, E., "Progress and procrastination in second language reading", Annual Review of Applied Linguistics, 25, pp.133~150 (2005)
  10. Bolt, D. M., "Estimation of compensatory and noncompensatory multidimensional item response models using Markov Chain Monte Carlo", Applied Psychological Measurement, 27, pp.395~414 (2003)
  11. Bradshaw, C. P., "The School Assessment for Environmental Typology(SAfETy) : An observational measure of the school environment", American Journal of Community Psychology, 56, pp.280~292 (2015)
  12. Development, selection and validation of a set of cognitive and linguistic attributes for the SAT I verbal: Sentence completion section, "Buck, G.", Educational Testing Service, ~, pp.ETS Research Report No. RR-98-23 (1998)
  13. Buck, G., "Application of the rule-space procedure to language testing: Examining attributes of a free response listening test", Language Testing, 15(2), pp.119~157 (1998)
  14. Carver, R. P., "Reading rate: Theory, research, and practical implications", Journal of Reading, 36(2), pp.84~95 (1992)
  15. Chen, J., "An extension of the G-DINA model for polytomous attributes", Paper presented at the annual meeting of American Educational Research Association (2012)
  16. Choi, Y., "Development of adaptive formative assessment system using computerized adaptive testing and dynamic bayesian networks", Applied Sciences, 10(22), pp.8196~ (2020)
  17. Choi, Y., "The relative role of vocabulary and grammatical knowledge in L2 reading comprehension: A systematic review of literature", International Review of Applied Linguistics in Language Teaching, 59, pp.1~30 (2021)
  18. Cowell, R. G., "Probabilistic networks and expert systems: Exact computational methods for Bayesian networks" (2007), Springer Science & Business Media
  19. Culbertson, M. J., "Bayesian networks in educational assessment: The state of the field", Applied Psychological Measurement, 40(1), pp.3~21 (2016)
  20. Davey, B., "Factors affecting the difficulty of reading comprehension items for successful and unsuccessful readers", The Journal of Experimental Education, 56(2), pp.67~76 (1988)
  21. DeCarlo, L. T., "Recognizing uncertainty in the Q-Matrix via a Bayesian extension of the DINA model", Applied Psychological Measurement, 36(6), pp.447~468 (2012)
  22. de la Torre, J., "DINA model and parameter estimation : A didactic", Journal of Educational and Behavioral Statistics, 34, pp.115~130 (2009)
  23. DiBello, L. V., "Cognitively diagnostic assessment", pp.361~389 (1995), Routledge
  24. Epskamp, S., "Estimating psychological networks and their accuracy: A tutorial paper", Behavior Research Methods, 50(1), pp.195~212 (2018)
  25. Finkelman, M., "Cognitive diagnostic models and computerized adaptive testing : Two new item-selection methods that incorporate response times", Journal of Computerized Adaptive Testing, 2(4), pp.59~76 (2014)
  26. Alexandra Gottardo, "Are first- and second-language factors related in predicting second-language reading comprehension? A study of Spanish-speaking children acquiring English as a second language from first to second grade.", Journal of Educational Psychology, 101(2), pp.330~344 (2009), American Psychological Association (APA)[DOI]
  27. Gough, P. B., "Decoding, reading, and reading disability", Remedial and Special Education, 7(1), pp.6~10 (1986)
  28. Grabe, W., "Reading in a second language: moving from theory to practice" (2009), Cambridge University Press
  29. Harding, L., "Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles", Language Testing, 32(3), pp.317~336 (2015)
  30. Hoover, W. A., "The simple view of reading", Reading and Writing: An Interdisciplinary Journal, 2(2), pp.127~160 (1990)
  31. Hu, M., "Unknown vocabulary density and reading comprehension", Reading in a Foreign Language, 23, pp.403~430 (2000)
  32. Hughes, A., "Testing for Language Teachers" (2003), Arthur Hughes
  33. Jang, E. E., "Cognitive diagnostic assessment of L2 reading comprehension ability : Validity arguments for applying Fusion Model to LanguEdge assessment", Language Testing, 26(1), pp.31~73 (2009)
  34. Jang, E. E., "Elementary school ELLs' reading skill profiles using cognitive diagnosis modeling : Roles of length of residence and home language environment", Language Learning, 63(3), pp.400~436 (2013)
  35. Kasai, M., "Application of the rule space model to the reading comprehension section of the test of English as a foreign language (TOEFL)", pp.1997
  36. Kim, Y. H., "Diagnosing EAP writing ability using the reduced reparameterized unified model", Language Testing, 28(4), pp.509~541 (2011)
  37. Kim, A. Y., "Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability", Language Testing, 32(2), pp.227~258 (2015)
  38. Koller, D., "Probabilistic graphical models: principles and techniques" (2009), MIT press
  39. Laufer, B., "Special language: From humans to thinking machines", pp.316~323 (1989), Multilingual Matters
  40. Lee, J. F., "Background knowledge and L2 reading", The Modern Language Journal, 70(4), pp.350~354 (1986)
  41. 이정원, "The Role of Vocabulary and Grammar in Different L2 Reading Comprehension Measures", 영어교육, 71(3), pp.79~97 (2016), 한국영어교육학회[DOI]
  42. Lee Boh Young, "Doable and Practical: A Validation Study of Classroom Diagnostic Tests", Journal of Asia TEFL, 17(2), pp.363~378 (2020), 아시아테플[DOI]
  43. Lee, Y-W., "Application of three cognitive diagnosis models to ESL reading and listening assessments", Language Assessment Quarterly, 6(3), pp.239~263 (2009)
  44. Leighton, J. P., "Cognitive diagnostic assessment for education: Theory and applications" (2007), Cambridge University Press
  45. Levy, R., "Bayesian psychometric modeling" (2017), CRC Press
  46. Li, Y., "Diagnosing primary students’ reading progression: Is cognitive diagnostic computerized adaptive testing the way forward?", Journal of Educational and Behavioral Statistics, 48(6), pp.842~865 (2023)
  47. Li, Y., "Validating a reading assessment within the cognitive diagnostic assessment framework: Q-matrix construction and model comparisons for different primary grades", Frontiers in Psychology, 12, pp.786612~ (2021)
  48. Li, H., "The selection of cognitive diagnostic models for a reading comprehension test", Language Testing, 33, pp.391~409 (2016)
  49. Qiao Lin, "Measuring Skill Growth and Evaluating Change: Unconditional and Conditional Approaches to Latent Growth Cognitive Diagnostic Models", Frontiers in Psychology, 11, pp.2205~ (2020), Frontiers Media SA[DOI]
  50. Li, H., "A cognitive diagnostic analysis of the MELAB reading test", Spaan Fellow, 9, pp.17~46 (2011)
  51. Lumley, T., "The notion of subskills in reading comprehension tests: An EAP example", Language Testing, 10(3), pp.211~234 (1993)
  52. Ma, W., "Estimating cognitive diagnosis models in small samples: Bayes modal estimation and monotonic constraints", Applied Psychological Measurement, 45(2), pp.95~111 (2021)
  53. McNeish, D., "On using Bayesian methods to address small sample problems", Structural Equation Modeling, 23(5), pp.750~773 (2016)
  54. Morvay, G., "The relationship between syntactic knowledge and reading comprehension in EFL learners", Studies in Second Language Learning and Teaching, 2(3), pp.415~438 (2012)
  55. Nájera, P., "The restricted DINA model : A comprehensive cognitive diagnostic model for classroom-level assessments", Journal of Educational and Behavioral Statistics, 48(6), pp.719~749 (2023)
  56. Nation, I. S. P., "Learning vocabulary in another language" (2001), Cambridge University Press
  57. Nation, I. S. P., "How large a vocabulary is needed for reading and listening?", Canadian Modern Language Review, 63, pp.59~82 (2006)
  58. Neapolitan, R. E., "Learning Bayesian Networks" (2004), Prentice Hall
  59. Brevik, L. M., "The complexity of second language reading: Investigating the L1-L2 relationship", Reading in a Foreign Language, 28(2), pp.161~182 (2016)
  60. Hamdollah Ravand, "Diagnostic Classification Models: Recent Developments, Practical Issues, and Prospects", International Journal of Testing, 20(1), pp.24~56 (2019), Informa UK Limited[DOI]
  61. Yanfang Pan, "The Impact of Sample Attrition on Longitudinal Learning Diagnosis: A Prolog", Frontiers in Psychology, 11, pp.1051~ (2020), Frontiers Media SA[DOI]
  62. Ravand, H., "Application of a cognitive diagnostic model to a high-stakes reading comprehension test", Journal of Psychoeducational Assessment, 34(8), pp.782~799 (2016)
  63. Rupp, A. A., "Diagnostic measurement: Theory, methods, and applications" (2010), Guilford Press
  64. Sawaki, Y., "Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments", Language Assessment Quarterly, 6(3), pp.190~209 (2009)
  65. Sedat Sen, "Sample Size Requirements for Applying Diagnostic Classification Models", Frontiers in Psychology, 11, pp.621251~ (2021), Frontiers Media SA[DOI]
  66. Sessoms, J., "Applications of diagnostic classification models: a literature review and critical commentary", Measurement Interdisciplinary Research and Perspectives, 16, pp.1~17 (2018)
  67. Shiotsu, T., "The relative significance of syntactic knowledge and vocabulary breadth in the prediction of reading comprehension test performance", Language Testing, 24(1), pp.99~128 (2007)
  68. Sinharay, S., "Assessing fit of cognitive diagnostic models a case study", Educational and Psychological Measurement, 67(2), pp.239~257 (2007)
  69. Sorrel, M. A., "Improving accuracy and usage by correctly selecting : The effects of model selection in cognitive diagnosis computerized adaptive testing", Applied Psychological Measurement, 45(2), pp.112~129 (2021)
  70. Sorrel, M. A., "Inferential item-fit evaluation in cognitive diagnosis modeling", Applied Psychological Measurement, 41(8), pp.614~631 (2017)
  71. Stanovich, K. E., "Toward an interactive-compensatory model of individual differences in the development of reading fluency", Reading Research Quarterly, 16, pp.32~71 (1980)
  72. Xiaojian Sun, "Binary Restrictive Threshold Method for Item Exposure Control in Cognitive Diagnostic Computerized Adaptive Testing", Frontiers in Psychology, 12, pp.517155~ (2021), Frontiers Media SA[DOI]
  73. Zafer Susoy, "The Role of Vocabulary vs. Syntactic Knowledge in L2 Reading Comprehension", Eurasian Journal of Applied Linguistics, 5, pp.113~130 (2019), Eurasian Journal of Applied Linguistics[DOI]
  74. Yi, Y. -S., "In search of optimal cognitive diagnostic model(s) for ESL grammar test data", Applied Measurement in Education, 30(2), pp.82~101 (2017)
  75. Van Gelderen, A., "Roles of linguistic knowledge, metacognitive knowledge and processing speed in L3, L2 and L1 reading comprehension: A structural equation modeling approach", International Journal of Bilingualism, 7, pp.7~25 (2003)
  76. Wang, N., "Bayesian model checking in cognitive diagnostic models", Behaviormetrika, 46(2), pp.371~388 (2019)
  77. The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in their first year of study at a British …, "Weir, C.", ~, pp.British Council (2009), IDP Australia Research Reports. 9
  78. Zhan, P., "Using JAGS for Bayesian cognitive diagnosis modeling: A tutorial", Journal of Educational and Behavioral Statistics, 44(4), pp.473~503 (2019)
  79. Zhang, D., "Vocabulary and grammar knowledge in second language reading comprehension: A structural equation modeling study", Modern Language Journal, 96, pp.558~575 (2012)

관련 논문AI추천

소규모 시험데이터에 대한 베이지언 인지진단모형의 적용최세일한국언어과학회 · 언어과학2023 · 제30권 제3호베이지언 네트워크를 이용한 영어 관사 학습의 인지진단 평가최세일한국언어과학회 · 언어과학2023 · 제30권 제1호Cognitive Diagnostic Assessment of English Grammar for Korean EFL Learners박찬호, 조수경한국영어교육학회 · 영어교육2011 · 제66권 제4호For a More Useful EFL Cognitive Diagnostic Assessment: Using Small Attributes and an Item Pool이연숙언어과학회 · 언어과학연구2016 · 제76호인지진단모형의 국어 교육적 접근 방안 탐색최숙기청람어문교육학회 · 청람어문교육2017 · 제62호Effect of a Cognitive Diagnosis Model-Based Feedback in an English Writing Class: Focusing on Korean English Learners조수경현대영어교육학회 · 현대영어교육2023 · 제24권

이 간행물의 다른 논문 (10건)

딥러닝 언어 모델과 인공신경망 기계 번역을 활용한 담화 조응 현상과 한정 명사구 연구강아름, 이용훈pp. 1-34동사이동은 통사이동으로 부적절한가?김대익pp. 35-55한국어 교재에 나타난 ‘형태적 사동’ 연구유영관, 김건희, 유석pp. 57-82Bound Noun pep in Korean김태식pp. 113-136A Socio-historical Study of Vowel Raising in Australian and Canadian English이학문pp. 137-152
원문 보기