Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination
2
조회
0
다운로드
6
인용
0
참고문헌
키워드
초록
Purpose: The Korea Medical Licensing Exam (KMLE) typically contains a large number of items. The purpose of this study was to investigate whether there is a difference in the cut score between evaluating all items of the exam and evaluating only some items whenconducting standard-setting. Methods: We divided the item sets that appeared on 3 recent KMLEs for the past 3 years into 4 subsets of each year of 25% each basedon their item content categories, discrimination index, and difficulty index. The entire panel of 15 members assessed all the items (360items, 100%) of the year 2017. In split-half set 1, each item set contained 184 (51%) items of year 2018 and each set from split-half set 2contained 182 (51%) items of the year 2019 using the same method. We used the modified Angoff, modified Ebel, and Hofstee methods in the standard-setting process. Results: Less than a 1% cut score difference was observed when the same method was used to stratify item subsets containing 25%,51%, or 100% of the entire set. When rating fewer items, higher rater reliability was observed. Conclusion: When the entire item set was divided into equivalent subsets, assessing the exam using a portion of the item set (90 out of360 items) yielded similar cut scores to those derived using the entire item set. There was a higher correlation between panelists’ individual assessments and the overall assessments.
Abstract
Purpose: The Korea Medical Licensing Exam (KMLE) typically contains a large number of items. The purpose of this study was to investigate whether there is a difference in the cut score between evaluating all items of the exam and evaluating only some items whenconducting standard-setting. Methods: We divided the item sets that appeared on 3 recent KMLEs for the past 3 years into 4 subsets of each year of 25% each basedon their item content categories, discrimination index, and difficulty index. The entire panel of 15 members assessed all the items (360items, 100%) of the year 2017. In split-half set 1, each item set contained 184 (51%) items of year 2018 and each set from split-half set 2contained 182 (51%) items of the year 2019 using the same method. We used the modified Angoff, modified Ebel, and Hofstee methods in the standard-setting process. Results: Less than a 1% cut score difference was observed when the same method was used to stratify item subsets containing 25%,51%, or 100% of the entire set. When rating fewer items, higher rater reliability was observed. Conclusion: When the entire item set was divided into equivalent subsets, assessing the exam using a portion of the item set (90 out of360 items) yielded similar cut scores to those derived using the entire item set. There was a higher correlation between panelists’ individual assessments and the overall assessments.
참고문헌 (12)
- 박훈기, "의사국가시험에서 실기시험의 도입이 의학교육에 미친 영향", 대한의사협회지, 55(2), pp.116~123 (2012), 대한의사협회
- Mikyoung Yim, "Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination", Korean Journal of Medical Education, 30(4), pp.347~357 (2018), 한국의학교육학회[DOI]
- Abdullah A. Ferdous, "The Use of Subsets of Test Questions in an Angoff Standard-Setting Method", Educational and Psychological Measurement, 65(2), pp.185~201 (2016), SAGE Publications[DOI]
- A study about standard setting for national licensing exams for health professions, "Ahn DS", Korea Health Personnel Licensing Examination, ~ (2018)
- Abdullah A. Ferdous, "Item Selection Strategy for Reducing the Number of Items Rated in an Angoff Standard Setting Study", Educational and Psychological Measurement, 67(2), pp.193~206 (2016), SAGE Publications[DOI]
- "Using stratified item selection to reduce the number of items rated in standard setting" (2011)[URL]
- Buckendahl CW, "Recommending cut scores with a subset of items: an empirical illustration", Pract Assess Res Eval, 15, pp.6~ (2010)[DOI]
- Angoff WH, "Scales, norms, and equivalent scores", pp.144~ (1984), Educational Testing Service
- Janghee Park, "Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee", Journal of Educational Evaluation for Health Professions, 15, pp.32~ (2018), Korea Health Personnel Licensing Examination Institute[DOI]
- Hofstee WK, "On educational testing", pp.109~127 (1983), Jossey-Bass
- Priya Kannan, "Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework", Applied Measurement in Education, 28(3), pp.169~186 (2015), Informa UK Limited[DOI]
- Hakan KARA, "Comparison of Passing Scores Determined by The Angoff Method in Different Item Samples", International Journal of Assessment Tools in Education, pp.80~97 (2020), International Journal of Assessment Tools in Education[DOI]