본문 바로가기
KERD
논문KCI 등재

고부담 읽기 평가의 DIF(차별문항기능) 분석: 계열 선택에 따른 배경지식이 읽기 평가 결과에 영향을 미치는가?

Analysis of Differential Item Functioning in High-Stakes Reading Assessments: Does Background Knowledge Based on Selection of Educational Curriculum Majors Influence Reading Assessment Results?

오규설
국어교육학회(since1969) (The Society Of Korean Language And Literature Education)|2023.12|국어교육연구 (Journal of Korean Language Education)|제52호|pp. 119-148|DOI 10.17313/jkorle.2023..52.119

2

조회

0

다운로드

0

인용

0

참고문헌

키워드

공정성Reading Assessment배경지식FairnessKorean SAT읽기 평가Differential Item Functioning차별문항기능DIF수능 국어 영역Background Knowledge

초록

고부담 읽기 평가에서 공정성은 타당도와 더불어 핵심적인 요소이다. 특정 하위 집단에게 문항이 유리하거나 불리하다면 이는 동일한 읽기 능력을 가진 학생들의 점수가 달리 나타나게 되므로 공정성의 문제는 평가 결과의 타당한 해석에 장애가 될 수 있다. 하위 집단 간 문항 반응의 편향은 DIF(차별문항기능)을 통해 식별할 수 있다. 따라서 이 연구는 한국의 수능 모의평가 국어 영역의 독서 문항을 대상으로 DIF 분석을 시행하여 계열 선택에 따라 다른 배경지식을 가진 하위 집단의 DIF를 식별하였다. 이후 그 원인을 분석하고, DIF를 감소시키는 대응 방안을 제시하였다.

Abstract

In high-stakes reading assessments, fairness is a critical component, along with validity. Suppose that an item is advantageous or disadvantageous to specific subgroups. In that case, it can lead to disparities in scores among students with the same reading ability, thus raising concerns about the fairness of the assessment and hindering valid interpretation of the results. Differential item functioning (DIF) can identify item bias among subgroups. Consequently, this study conducted a DIF analysis of the reading comprehension questions comprising the Korean SAT mock exams and identified DIF among subgroups with different background knowledge based on their chosen academic tracks. Subsequently, the study analyzed the causes of these differences, and this paper proposes measures to reduce DIF.

참고문헌 (54)

  1. 주세형, "국어과 읽기 평가 맥락에서 과학 지문의 특성 연구", 교육과정평가연구, 18(3), pp.313~342 (2015), 한국교육과정평가원
  2. Education(NCME), "Standards for educational and psychological testing" (2014), American Educational Research Association
  3. Bachman, L. F., "Building and supporting a case for test use", Language Assessment Quarterly, 2(1), pp.1~34 (2005)
  4. Bachman, L. F., "Language assessment in practice" (2010), Oxford University Press
  5. Bond, T., "Applying the Rasch model: Fundamental measurement in the human sciences" (2015), Routledge
  6. Cai, L., (2012), Vector Psychometric Group, LLC
  7. Camilli, G., "Educational measurement", pp.220~256 (2006), American Council on Education
  8. Carlson, J. E., "Advancing human assessment: The methodological, psychological and policy contributions of ETS", pp.133~178 (2017)
  9. de Groot, A., "Thought and Choice in Chess" (1965), Mouton Publishers
  10. Debelak, R., "Testing the local independence assumption of the Rasch model with Q 3-based nonparametric model tests", Applied Psychological Measurement, 44(2), pp.103~117 (2020)
  11. Dorans, N. J., "Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the scholastic aptitude test", Journal of Educational Measurement, 23, pp.355~368 (1986)
  12. Douglas, J., "Item-bundle DIF hypothesis testing : Identifying suspect bundles and assessing their differential functioning", Journal of Educational Measurement, 33, pp.465~484 (1996)
  13. Ferne, T., "A synthesis of 15 years of research on DIF in language testing : Methodological advances, challenges, and recommendations", Language Assessment Quarterly, 4(2), pp.113~148 (2007)
  14. A componential theory of reading skills and their interactions, "Frederiksen, J. R.", University of Illinois at Urbana-Champaign, Center for the Study of Reading, ~, pp.Center for the Study of Reading Technical Report; no. 227 (1982)
  15. Gierl, M. J., "Using dimensionality-based DIF analyses to identify and interpret constructs that elicit group differences", Educational Measurement : Issues and Practice, 24(1), pp.3~14 (2005)
  16. Goodman, K. S., "Reading : a psycholinguistic guessing game", Journal of the Reading Specialist, 6(1), pp.126~135 (1967)
  17. Hirsch, E. D., "Reading comprehension requires knowledge of words and the world", American educator, 27(1), pp.10~13 (2003)
  18. Hofstetter, C. R., "Knowledge, literacy, and power", Communication Research, 26(1), pp.58~80 (1999)
  19. Holland, P. W., "Test validity", pp.76~86 (1988), Routledge
  20. Holland, P. W., "Differential item functioning" (1993), Lawrence Erlbaum
  21. Jencks, C., "The black-white test scope gap : Why it persists and what can be done", The Brookings Review, 16(2), pp.24~27 (1998)
  22. Karami, H., "Differential item functioning : Current problems and future directions", International Journal of Language Studies, 5(3), pp.133~142 (2011)
  23. Kendeou, P., "The effects of prior knowledge and text structure on comprehension processes during reading of scientific texts", Memory & cognition, 35(7), pp.1567~1577 (2007)
  24. Kintsch, W., "Comprehension: A paradigm for cognition" (1998), Cambridge University Press
  25. Koo, J., "Examining differential item functioning trends for English language learners in a reading test: A meta-analytical approach", Language Testing, 31(1), pp.89~109 (2014)
  26. Linacre, J. M., "Item bias: Mantel-Haenszel and the Rasch model", Memorandum, (39) (1987), ERIC Clearinghouse
  27. Lord, F. M., "Applications of item response theory to practical testing problems" (1980), Erlbaum
  28. Magis, D., "A general framework and an R package for the detection of dichotomous differential item functioning", Behavior research methods, 42(3), pp.847~862 (2010)
  29. Mantel, N., "Chi-Square Tests With One Degree of Freedom : Extensions of the Mantel-Haenszel Procedure", Journal of the American Statistical Association, 58, pp.690~700 (1963)
  30. Mantel N., "Statistical Aspects of the Analysis of Data From Retrospective Studies of Disease", Journal of the National Cancer Institute, 22, pp.719~748 (1959)
  31. Marzano, R. J, "Building background knowledge for academic achievement:Research on what works in schools" (2004), Ascd
  32. PIRLS 2006International Report: IEA’s Progress in International Reading Literacy Study in primary schools in 40 countries, "Mullis, I. V. S.", TIMSS & PIRLS International Study Center, Boston College, ~ (2007)
  33. Pae, T. -I., "DIF for examinees with different academic backgrounds", Language Testing, 21(1), pp.53~73 (2004)
  34. Perfetti, C. A., "Reading ability" (1985), Oxford University Press
  35. Raquel, M., "The Rasch measurement approach to differential item functioning(DIF)analysis in language assessment research", Quantitative data analysis for language assessment, 1, pp.103~131 (2019)
  36. Robitzsch, A., "TAM : Test Analysis Modules", R package version, 4, pp.1~4 (2022)
  37. Roussos, L. A., "Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel type I error Performance", Journal of Educational measurement, 33(2), pp.215~230 (1996)
  38. Rumelhart, D., "Theoretical Issues in Reading Comprehension", pp.33~58 (1980), Erlbaum Associates
  39. Schaeffer, G. A., "Psychometric Evaluation of the New GRE® Writing Assessment", ETS Research Report Series, 2001(1), pp.1~53 (2001)
  40. Shealy, R., "A model-based standardization approach that separates true bias", DIF from group ability differences and detects test bias, DIF, pp.Psychometrika, 1993[DOI]
  41. Smith, R., "The role of background knowledge in reading comprehension : A critical review", Reading Psychology, 42(3), pp.214~240 (2021)
  42. Song, X., "DIF investigations across groups of gender and academic background in a large-scale high-stakes language test", Language Testing and Assessment, 4(1), pp.97~124 (2015)
  43. Swaminathan H., "Detecting differential item functioning using logistic regression procedures", Journal of Educational Measurement, 27(4), pp.361~370 (1990)
  44. Wind, S., "Rasch measurement theory analysis in R" (2022), CRC Press
  45. Zieky, M. J., "Looking back: Proceedings of a conference in honor of Paul W. Holland", pp.115~127 (2011), Springer
  46. Zumbo, B. D., "Three generations of DIF analyses : Considering where it has been, where it is now, and where it is going", Language assessment quarterly, 4(2), pp.223~233 (2007)
  47. Zwick, R., "A review of ETS differential item functioning assessment procedures: Flagging rules, minimum sample size requirements, and criterion refinement", ETS Research Report Series, 2012(1), pp.1~30 (2012)
  48. (2015), 교육부[URL]
  49. (2019), 교육부[URL]
  50. "“역대 최악 국어, 난이도 조절 실패”…교사·입시전문가한목소리" (2018)[URL]
  51. "수능 국어, 결국 ‘과학’에서 결판났다" (2015)[URL]
  52. "DIF-DPF-bias-interactions concepts. Help for Winsteps Rasch Measurement and Rasch Analysis Software" (2018)[URL]
  53. "The Mantel-Haenszel Procedure" (2009)[URL]
  54. "The SIBTEST Procedure" (2008)[URL]

관련 논문AI추천

읽기 선다형 문항의 변별도 결정 요인에 관한 연구 - 국가수준 학업성취도 고등학교 읽기 문항을 중심으로 -길호현국어교육학회(since1969) · 국어교육연구2015 · 제59호차별기능의 원인파악을 위한 차별기능문항군 기법의 적용손원숙사범대학부속중등교육연구소 · 중등교육연구2012 · 제60권 제4호한국어능력시험 읽기 영역의 언어권에 따른 차별 기능 문항 분석김미옥, 김정은언어연구교육원 한국어학당 · 외국어로서의 한국어교육2013 · 제38권 대단위 읽기 평가의 테스트렛 효과 진단 및 대안 탐색 －Bifactor 모형을 중심으로오규설청람어문교육학회 · 청람어문교육2025 · 제108호차별적 검사 기능 분석을 통한 컴퓨터 기반 검사의 양식 효과 연구김선미, 박찬호교육문제연구소 · 교육연구2025 · 제92권 An Application of the Rasch Model to Assessing an English Achievement Test in a High School Setting심재우, Keiko Samimy현대영어교육학회 · 현대영어교육2010 · 제11권 제2호

이 간행물의 다른 논문 (11건)

원어민 교사와 비원어민 교사의 한국어(KSL) 수업의 어려움 연구김여진, 김호정, 강 성가pp. 1-54비판적 언어 연구와 국어교육: 쟁점과 전망김규훈pp. 55-88구술성 변화에 따른 의사소통의 관점 탐색박창균, 조재윤pp. 89-118김윤식의 문학교육론우신영pp. 149-177시조 감상 교육에서 학습자의 감각 체험 양상 연구: <오우가>를 중심으로이루리pp. 179-223
원문 보기