다층문항반응이론 기반의TIMSS 수학 평가 결과 분석
An Analysis of TIMSS Mathematics Assessment Results Based on Multilevel Item Response Theory
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
이 연구는 TIMSS 2015 4학년과 8학년 수학 평가에 다층문항반응이론 모형을 적용하여 분석하고 그 결과를 바탕으로 우리나라 수학교육의 발전 방향에 대한 시사점을 얻기 위하여 실시되었다. 분석 결과 각 인지요소가 국가별 문항 난이도에 기여하는 정도를 확인할 수 있었다. 또한 분석 결과는 집단수준의 진단을 가능하게 하는 프로파일로 해석되었다. 우리나라 4학년의 경우 ‘검색’에 가장 큰 강점을 보였으며 ‘분석’에 가장 큰 약점을 보였다. 8학년은 ‘표현’이 가장 강점을 보이는 요소였으며, ‘실행’이 가장 큰 약점을 보이는 요소였다. 이 분석 결과로 각 요소에 대한 국가 간 비교도 가능한데, 우리나라 4학년이 가장 큰 약점을 보인 ‘분석’에 강점을 보인 국가는 싱가포르였다. 그리고 8학년의 약점 요소인 ‘실행’에 대해서는 홍콩과 스웨덴이 강점을 보였다. 마지막으로 4학년과 8학년 결과를 비교해보면 강점이나 약점이 유지된 경우와 강점과 약점이 바뀐 경우 등으로 유형화할 수 있었다. ‘검색’의 경우 가장 큰 차이를 보여 강점 요소에서 약점 요소로 바뀌었으며 ‘계산’은 약점에서 강점으로 큰 변화를 보인 요소였다. 이 연구의 성과는 우선 다층문항반응이론 기반의 집단수준 진단평가 방법론을 적용함으로써 우리나라 학생들의 능력을 통제한 상태에서 강점과 약점 요소들을 정확하게 진단할 수 있었다는 것이다. 이 연구 결과에 교육과정 및 교과서 분석, 학교의 교수학습 과정에 대한 실태 분석이 더해지면 수학교육 발전을 위해 더욱 유용하게 활용될 수 있을 것이며, 국제 및 국내의 다른 집단수준 평가에도 적용할 수 있을 것으로 기대된다.
Abstract
The purpose of this study was to analyze TIMSS 2015 fourth and eighth grade mathematics assessment results by applying a multilevel item response theory model, and to provide implications for mathematics education in Korea. It was analyzed how much each cognitive attribute contributed to the difficulty level of each country. Also, the analysis results were interpreted as a group-level diagnosis profile. Fourth grade students in Korea had the greatest strength in ‘Retrieve’ and the biggest weakness in ‘Analyze.’ ‘Represent’ was the strongest factor in explaining item difficulty levels for Korean eighth graders, while ‘Implement’ was the weakest factor. The analysis results can be compared across countries for each attribute, and Singapore was the country having the greatest strength in ‘Analyze,’ which was the weakest attribute for the fourth grade students in Korea. Hong Kong and Sweden had strengths in ‘Implement’ while it was a weakness for the eighth grade students in Korea. Finally, the attributes could be classified by types of changes from the fourth to the eighth grade for each country. ‘Retrieve’ showed the biggest difference from a strong point in the fourth grade to a weak point in the eighth grade in Korea, and ‘Compute’ was an attribute showing a big change from weakness to strength. This study successfully applied a group-level diagnostic assessment methodology based on multilevel item response theory, and could thus accurately identify strong and weak attributes of Korean students by controlling for overall ability levels. When combined with other research studies with regard to the curriculum, textbooks, and teaching and learning, this study is expected to provide useful implications for the development of mathematics education in Korea. The methodology of this study is also expected to be applied to other international and domestic group-level assessments.
참고문헌 (57)
- 김희경, "인지진단모형을 활용한 수학 학업성취 결과 분석 -2011년 국가수준 학업성취도 평가 자료를 중심으로-", 학교수학, 15(2), pp.289~314 (2013), 대한수학교육학회
- 박찬호, "다층 구조를 이루는 이분문항 자료의 급내상관계수 추정 방안 비교", 교육평가연구, 26(2), pp.459~476 (2013), 한국교육평가학회
- 박찬호, "집단수준 진단평가를 활용한 학교 향상도 상승 및 하락 학교 프로파일 분석", 교육평가연구, 27(5), pp.1243~1264 (2014), 한국교육평가학회
- 박찬호, "다층문항반응이론 기반의 학교 간 차이 및 연도 간 변화 추이 분석 방안", 교육평가연구, 28(5), pp.1401~1422 (2015), 한국교육평가학회
- 반재천, "인지진단모형, 인지요소패턴 추정 방법, 사례수에 따른 인지요소패턴 추정의 분류 일관성 및 분류 정확성 비교", 교육평가연구, 29(3), pp.405~431 (2016), 한국교육평가학회
- 심재호, "국내외 학업성취도 평가의 서답형 문항 특성과 채점 체제 고찰", 열린교육연구, 20(4), pp.159~185 (2012), 한국열린교육학회
- 신기현, "PISA 의 학습 평가틀에 대한 비판적 분석", 열린교육연구, 14(1), pp.165~183 (2006), 한국열린교육학회
- Adams, R. J., "Multilevel item response models : An approach to errors in variables regression", Journal of Educational and Behavioral Statistics, 22, pp.47~76 (1997)
- Baker, F. B., "Item response theory: Parameter estimation techniques" (2004), Dekker
- (2010)[URL]
- Bock, R. D., "Multilevel analysis of educational data" (1989), Academic Press
- Bradshaw, L. P., "Combining item response theory and diagnostic classification models : A psychometric model for scaling ability and diagnosing misconceptions", Psychometrika, 79, pp.403~425 (2014)
- Cai, L., (2017), Vector Psychometric Group
- Camilli, G., "An aggregate IRT procedure for exploratory factor analysis", Journal of Educational and Behavioral Statistics, 40, pp.377~401 (2015)
- Camilli, G., "Summarizing item difficulty variation with parcel scores", Journal of Educational Measurement, 45, pp.363~389 (2008)
- Chiu, C. -Y., "Consistency of cluster analysis for cognitive diagnosis : The DINO model and the DINA model revisited", Applied Psychological Measurement, 39, pp.465~479 (2015)
- Cui, Y., "Evaluating person fit for cognitive diagnostic assessment", Applied Psychological Measurement, 39, pp.223~238 (2015)
- de Ayala, R. J., "The theory and practice of item response theory" (2009), Guilford
- De Boeck, P., "Explanatory item response models" (2004), Springer
- Embretson, S. E., "Item response theory for psychologists" (2000), Lawrence Erlbaum
- Fox, J. -P., "Multilevel item response theory: A Bayesian perspective on estimating parameters and testing statistical hypotheses", doctor, pp.2001
- Fox, J. -P., "Applications of multilevel IRT modeling", School Effectiveness and School Improvement, 15, pp.261~280 (2004)
- Fox, J. -P., "Multilevel IRT modeling in practice with the package mlirt", Journal of Statistical Software, 20, pp.1~16 (2007)
- Glas, C., "Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis", pp.97~115 (2014), CRC
- Hambleton, R. K., "Item response theory: Principles and applications" (1985), Kluwer-Nijhoff
- Heyneman, S. P., "Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis", pp.37~72 (2014), CRC
- Holland, P. W., "Differential item functioning" (1993), Lawrence Erlbaum
- Hong, H., "Efficient models for cognitive diagnosis with continuous and mixed-type latent variables", Applied Psychological Measurement, 39, pp.31~43 (2015)
- Kamata, A., "Item analysis by the hierarchical generalized linear model", Journal of Educational Measurement, 38, pp.79~93 (2001)
- Kaplan, M., "New item selection methods for cognitive diagnosis computerized adaptive testing", Applied Psychological Measurement, 39, pp.167~188 (2015)
- Li, X., "Assessment of differential item functioning under cognitive diagnosis models : The DINA model example", Journal of Educational Measurement, 52, pp.28~54 (2015)
- Lord, F. M., "Applications of item response theory to practical testing problems" (1980), Lawrence Erlbaum
- Lunn, D., "The BUGS books: A practical introduction to Bayesian analysis" (2012), CRC
- Madison, M. J., "The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model", Educational and Psychological Measurement, 75, pp.491~511 (2015)
- "TIMSS 2015international results in science. Retrieved from Boston College" (2016)[URL]
- Martin, M. O., "TIMSS 2011international results in science" (2012), TIMSS & PIRLS International Study Center, Boston College
- "TIMSS 2015 assessment frameworks" (2013)[URL]
- Mullis, I. V. S., "TIMSS 2011international results in mathematics" (2012), TIMSS & PIRLS International Study Center, Boston College
- "TIMSS 2015international results in mathematics" (2016)[URL]
- Natesan, P., "Bayesian estimation of graded response multilevel models using Gibbs sampling : Formulation and illustration", Educational and Psychological Measurement, 70, pp.420~439 (2010)
- Park, C., "Application of multilevel IRT to investigate cross-national skill profiles on TIMSS 2003", Issues and Methodologies in Large-Scale Assessments, 1, pp.71~96 (2008)
- Park, C., "April). Application of multilevel IRT to multiple-form linking when common items are drifted", Paper presented at the annual meeting of the National Council on Measurement in Education (2007)
- Porter, A. C., "Methodological advances in cross-national surveys of educational achievement" (2002), National Research Council
- Raudenbush, S. W., "Hierarchical linear models:Applications and data analysis methods" (2002), Sage
- Raudenbush, S. W., (2006), Scientific Software International
- Reckase, M. D., "Multidimensional item response theory" (2009), Springer
- Rupp, A. A., "Diagnostic measurement:Theory, methods, and applications" (2010), Guilford
- Rutkowski, D., "Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis", pp.3~9 (2014), CRC
- Snijders, T., "Multilevel analysis: An introduction to basic and advanced multilevel modeling" (1999), Sage
- Spiegelhalter, D. J., (2003), MRC Biostatistics Unit
- Stapleton, L. M., "Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis", pp.363~388 (2014), CRC
- Stone, C. A., "Providing subscale scores for diagnostic information : A case study when the test is essentially unidimensional", Applied Measurement in Education, 23, pp.63~86 (2010)
- Tatsuoka, K. K., "Cognitive assessment" (2009), Routledge
- Tatsuoka, K. K., "Patterns of diagnosed mathematical content and process skills in TIMSS-R across a sample of 20 countries", American Educational Research Journal, 41, pp.901~926 (2004)
- von Davier, M., "Handbook of statistics (Vol. 26): Psychometrics" (2006), Elsevier
- Wagemaker, H., "Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis", pp.11~36 (2014), CRC
- Xin, T., "Local equating of cognitively diagnostic modeled observed scores", Applied Psychological Measurement, 39, pp.44~61 (2015)