Comparisons of item difficulty and passing scores by test equating in a basic medical education curriculum
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose: Test equating studies in medical education have been conducted only for high-stake exams or to compare two tests given in a single course. Based on item response theory, we equated computer-based test (CBT) results from the basic medical education curriculum at the College of Medicine, the Catholic University of Korea and evaluated the validity of using fixed passing scores. Methods: We collected 232 CBTs (28,636 items) for 40 courses administered over a study period of 9 years. The final data used for test equating included 12 pairs of tests. After test equating, Wilcoxon rank-sum tests were utilized to identify changes in item difficulty between previous tests and subsequent tests. Then, we identified gaps between equated passing scores and actual passing scores in subsequent tests through an observed-score equating method. Results: The results of Wilcoxon rank-sum tests indicated that there were no significant differences in item difficulty distribution by year for seven pairs. In the other five pairs, however, the items were significantly more difficult in subsequent years than in previous years. Concerning the gaps between equated passing scores and actual passing scores, equated passing scores in 10 pairs were found to be lower than actual passing scores. In the other two pairs, equated passing scores were higher than actual passing scores. Conclusion: Our results suggest that the item difficulty distributions of tests taught in the same course during successive terms can differ significantly. It may therefore be problematic to use fixed passing scores without considering this possibility.
Abstract
Purpose: Test equating studies in medical education have been conducted only for high-stake exams or to compare two tests given in a single course. Based on item response theory, we equated computer-based test (CBT) results from the basic medical education curriculum at the College of Medicine, the Catholic University of Korea and evaluated the validity of using fixed passing scores. Methods: We collected 232 CBTs (28,636 items) for 40 courses administered over a study period of 9 years. The final data used for test equating included 12 pairs of tests. After test equating, Wilcoxon rank-sum tests were utilized to identify changes in item difficulty between previous tests and subsequent tests. Then, we identified gaps between equated passing scores and actual passing scores in subsequent tests through an observed-score equating method. Results: The results of Wilcoxon rank-sum tests indicated that there were no significant differences in item difficulty distribution by year for seven pairs. In the other five pairs, however, the items were significantly more difficult in subsequent years than in previous years. Concerning the gaps between equated passing scores and actual passing scores, equated passing scores in 10 pairs were found to be lower than actual passing scores. In the other two pairs, equated passing scores were higher than actual passing scores. Conclusion: Our results suggest that the item difficulty distributions of tests taught in the same course during successive terms can differ significantly. It may therefore be problematic to use fixed passing scores without considering this possibility.
참고문헌 (19)
- Bandaranayake RC, "Setting and maintaining standards in multiple choice examinations: AMEE guide no. 37", Med Teach, 30(9-10), pp.836~845 (2008)
- MacDougall M, "Variation in assessment and standard setting practices across UK undergraduate medicine and the need for a benchmark", Int J Med Educ, 6, pp.125~135 (2015)
- Yousef MK, "Challenging the arbitrary cutoff score of 60%: standard setting evidence from preclinical operative dentistry course", Med Teach, 39(sup1), pp.S75~S79 (2017)
- Cohen-Schotanus J, "A standard setting method with the best performing students as point of reference : practical and affordable", Med Teach, 32(2), pp.154~160 (2010)
- Tekian A, "Overcome the 60% passing score and improve the quality of assessment", GMS Z Med Ausbild, 32(4), pp.Doc43~ (2015)
- Yim MK, "Test equating of the medical licensing examination in 2003 and 2004 based on the item response theory", J Educ Eval Health Prof, 3, pp.2~ (2006)
- Baker FB, "The basics of item response theory" (2001), ERIC Clearinghouse on Assessment and Evaluation
- Downing SM, "Item response theory : applications of modern test theory in medical education", Med Educ, 37(8), pp.739~745 (2003)
- De Champlain AF, "A primer on classical test theory and item response theory for assessments in medical education", Med Educ, 44(1), pp.109~117 (2010)
- Guilera G, "Item response theory test equating in health sciences education", Adv Health Sci Educ Theory Pract, 13(1), pp.3~10 (2008)
- 허선, "의과대학 한 과목 시험에서 문항반응이론에 따른 검사 동등화 예", Korean Journal of Medical Education, 17(1), pp.15~28 (2005), 한국의학교육학회
- Liao SW, "Comparison of proficiency in an anesthesiology course across distinct medical student cohorts : psychometric approaches to test equating", J Chin Med Assoc, 77(3), pp.150~154 (2014)
- Rizopoulos D, "ltm : an R package for latent variable modeling and item response theory analyses", J Stat Softw, 17(5), pp.1~25 (2006)
- Teker GT, "An investigation of goodness of model data fit", Procedia Soc Behav Sci, 106, pp.394~400 (2013)
- Stocking ML, "Developing a common metric in item response theory", Appl Psychol Meas, 7(2), pp.201~210 (1983)
- Battauz M, "equateIRT : an R package for IRT test equating", J Stat Softw, 68(7), pp.1~22 (2015)
- 김건상, "우리는 어떤 의사를 양성해야 하는가?", Korean Journal of Medical Education, 23(1), pp.3~5 (2011), 한국의학교육학회
- Park YS, "Three controversies over item disclosure in medical licensure examinations", Med Educ Online, 20, pp.28821~ (2015)
- Michaelides MP, "A review of the effects on IRT item parameter estimates with a focus on misbehaving common items in test equating", Front Psychol, 1, pp.167~ (2010)