Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method. Methods The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station. Results On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts. Conclusion The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.
Abstract
Purpose Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method. Methods The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station. Results On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts. Conclusion The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.
참고문헌 (14)
- Cizek GJ, "Setting performance standards: foundations, methods, and innovations" (2012), Routledge
- Shulruf B, "Insights into the Angoff method : results from a simulation study", BMC Med Educ, 16, pp.134~ (2016)[DOI]
- Chong L, "The sights and insights of examiners in objective structured clinical examinations", J Educ Eval Health Prof, 14, pp.34~ (2017)[DOI]
- Chong L, "Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia", J Educ Eval Health Prof, 15, pp.17~ (2018)[DOI]
- Klein Nulend R, "Predictive validity of a tool to resolve borderline grades in OSCEs", GMS J Med Educ (2020)[DOI]
- Shulruf B, "The Objective Borderline method(OBM) : a probability-based model for setting up an objective pass", fail cut-off score in medical programme assessments, 2013, pp.18, 231~244
- Shulruf B, "Cut-scores revisited : feasibility of a new method for group standard setting", BMC Med Educ, 18, pp.126~ (2018)[DOI]
- Shulruf B, "Standard setting made easy : validating the Equal Z-score(EZ)method for setting cut-score for clinical examinations", BMC Med Educ, 20, pp.167~ (2020)[DOI]
- Felthun JZ, "Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations : a systematic scoping review", J Educ Eval Health Prof, 18, pp.11~ (2021)[DOI]
- Naumann FL, "Exploring examiner judgement of professional competence in rater based assessment", Adv Health Sci Educ Theory Pract, 21, pp.775~788 (2016)[DOI]
- Fan X, "APA handbook of research methods in psychology. Vol. 2, Research designs: quantitative, qualitative, neuropsychological, and biological" (2012), American Psychological Association
- 박장희, "Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination", Journal of Educational Evaluation for Health Professions, 17, pp.1~10 (2020), 한국보건의료인국가시험원[DOI]
- Tavakol M, "Modelling the Hofstee method reveals problems", Med Teach, 36, pp.181~182 (2014)[DOI]
- Wyse AE, "Comparing cut scores from the Angoff method and two variations of the Hofstee and Beuk methods", Appl Meas Educ, 33, pp.159~173 (2020)[DOI]