Performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose: This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact ofraters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores. Methods: Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and thePearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whethercorrect answers were provided or not and differences between internists and other specialists using the t-test. Results: The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices waslow when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answerswere provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists. Conclusion: Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore,raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting
Abstract
Purpose: This study aimed to assess the performance of the Ebel standard-setting method for the spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisting of multiple-choice questions. Specifically, the following parameters were evaluated: inter-rater agreement, the correlations between Ebel scores and item facility indices, the impact ofraters’ knowledge of correct answers on the Ebel score, and the effects of raters’ specialty on inter-rater agreement and Ebel scores. Methods: Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. The Ebel method was applied to 203 multiple-choice questions by 49 raters. Facility indices came from 194 candidates. We computed the Fleiss kappa and thePearson correlations between Ebel scores and item facility indices. We investigated differences in the Ebel score according to whethercorrect answers were provided or not and differences between internists and other specialists using the t-test. Results: The Fleiss kappa was below 0.15 for both facility and relevance. The correlation between Ebel scores and facility indices waslow when correct answers were provided and negligible when they were not. The Ebel score was the same whether the correct answerswere provided or not. Inter-rater agreement and Ebel scores were not significantly different between internists and other specialists. Conclusion: Inter-rater agreement and correlations between item Ebel scores and facility indices were consistently low; furthermore,raters’ knowledge of the correct answers and raters’ specialty had no effect on Ebel scores in the present setting
참고문헌 (15)
- American Educational Research Association, "Standards for educational and psychological testing", American Educational Research Association, 230~ (2014)
- Cizek GJ, "Standard setting", pp.352~ (2007), SAGE Publications Inc
- John J Norcini, "Setting standards on educational tests", Medical Education, 37(5), pp.464~469 (2003), Wiley[DOI]
- Ebel RL, "Essentials of educational measurement", pp.634~ (1972), Prentice-Hall
- M. G. Goldenberg, "Systematic review to establish absolute standards for technical performance in surgery", British Journal of Surgery, 104(1), pp.13~21 (2017), Oxford University Press (OUP)[DOI]
- Janghee Park, "Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee", Journal of Educational Evaluation for Health Professions, 15, pp.32~ (2018), Korea Health Personnel Licensing Examination Institute[DOI]
- Shoukri MM, "Measures of interobserver agreement and reliability", pp.269~ (2011), CRC Press
- J. Richard Landis, "The Measurement of Observer Agreement for Categorical Data", Biometrics, 33(1), pp.159~ (1977), JSTOR[DOI]
- M D Cusimano, "Standard setting in medical education", Academic Medicine, 71(10), pp.S112~20 (1996), Ovid Technologies (Wolters Kluwer Health)[DOI]
- Steven M. Downing, "RESEARCH METHODOLOGY: Procedures for Establishing Defensible Absolute Passing Scores on Performance Examinations in Health Professions Education", Teaching and Learning in Medicine, 18(1), pp.50~57 (2006), Informa UK Limited[DOI]
- Matt Homer, "Psychometric characteristics of integrated multi-specialty examinations: Ebel ratings and unidimensionality", Assessment & Evaluation in Higher Education, 37(7), pp.787~804 (2012), Informa UK Limited[DOI]
- Steven M. Downing, "Establishing Passing Standards for Classroom Achievement Tests in Medical Education: A Comparative Study of Four Methods", Academic Medicine, 78(Supplement), pp.S85~S87 (2003), Ovid Technologies (Wolters Kluwer Health)[DOI]
- D B Swanson, "Setting content-based standards for national board exams", Academic Medicine, 65(9), pp.S17~8 (1990), Ovid Technologies (Wolters Kluwer Health)[DOI]
- Matt Homer, "Setting standards in knowledge assessments: Comparing Ebel and Cohen via Rasch", Medical Teacher, 38(12), pp.1267~1277 (2016), Informa UK Limited[DOI]
- Adam E. Wyse, "A Method for Detecting Regression of Hard and Easy Item Angoff Ratings", Journal of Educational Measurement, 56(1), pp.28~50 (2019), Wiley[DOI]