Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to theset cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed fromthe norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school. Methods: This was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikinon July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales. Results: The cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was nosignificant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and thehighest proportion of the borderline group showed the largest cut score difference in standard setting methods. Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance canvary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensuson the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.
Abstract
Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to theset cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed fromthe norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school. Methods: This was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikinon July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales. Results: The cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was nosignificant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and thehighest proportion of the borderline group showed the largest cut score difference in standard setting methods. Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance canvary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensuson the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.
참고문헌 (18)
- De Champlain AF, "Understanding medical education: evidence, theory, and practice", pp.347~359 (2018), Wiley-Blackwell
- M.T. Kane, "Designing and evaluating standard-setting procedures for licensure and certification tests", Advances in Health Sciences Education, 4(3), pp.195~207 (1999), Springer Science and Business Media LLC[DOI]
- David M. Kaufman, "A Comparison of Standard-setting Procedures for an OSCE in Undergraduate Medical Education", Academic Medicine, 75(3), pp.267~271 (2000), Ovid Technologies (Wolters Kluwer Health)[DOI]
- Reid KJ, "Comparing the borderline group and borderline regression approaches to setting objective structured clinical examination cut scores", J Contemp Med Educ, 2, pp.8~12 (2014)[DOI]
- Norcini JJ, "Assessment methods in medical education", Teach Teach Educ, 23, pp.239~250 (2007)[DOI]
- Min Liu, "Setting Pass Scores for Clinical Skills Assessment", The Kaohsiung Journal of Medical Sciences, 24(12), pp.656~663 (2008), Wiley[DOI]
- 박장희, "Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination", Journal of Educational Evaluation for Health Professions, 17, pp.1~10 (2020), 한국보건의료인국가시험원[DOI]
- Janghee Park, "Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee", Journal of Educational Evaluation for Health Professions, 15, pp.32~ (2018), Korea Health Personnel Licensing Examination Institute[DOI]
- 임미경, "Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination", Journal of Educational Evaluation for Health Professions, 17, pp.1~9 (2020), 한국보건의료인국가시험원[DOI]
- Charlotte Ringsted, "‘The research compass’: An introduction to research in medical education: AMEE Guide No. 56", Medical Teacher, 33(9), pp.695~709 (2011), Informa UK Limited[DOI]
- 박송이, "Clinical performance of medical students in Korea in a whole-task emergency station in the objective structured clinical examination with a standardized patient complaining of palpitations", Journal of Educational Evaluation for Health Professions, 17, pp.1~6 (2020), 한국보건의료인국가시험원[DOI]
- Matt Homer, "Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can ‘work’", Medical Teacher, 42(3), pp.306~315 (2019), Informa UK Limited[DOI]
- Wood TJ, "Standard setting in a small scale OSCE: a comparison of the Modified Borderline-Group Method and the Borderline Regression Method", Adv Health Sci Educ Theory Pract, 11, pp.115~122 (2006)[DOI]
- Danette W. McKinley, "How to set standards on performance-based examinations: AMEE Guide No. 85", Medical Teacher, 36(2), pp.97~110 (2013), Informa UK Limited[DOI]
- Matthew Homer, "The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method", Medical Teacher, 31(5), pp.420~425 (2009), Informa UK Limited[DOI]
- Vanda Yazbeck Karam, "Evaluating the validity evidence of an OSCE: results from a new medical school", BMC Medical Education, 18(1) (2018), Springer Science and Business Media LLC[DOI]
- Hejri SM, "Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination", J Res Med Sci, 18, pp.887~891 (2013)
- Rabab Abdel Ra’oof, "Developing a Model for Identification of the Borderline Group in Objective Structured Clinical Exam", Education in Medicine Journal, 12(3), pp.1~10 (2020), Penerbit Universiti Sains Malaysia[DOI]