Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Purpose The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE. Methods The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results. Results The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method. Conclusion The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.
Abstract
Purpose The percent Angoff (PA) method has been recommended as a reliable method to set the cutoff score instead of a fixed cut point of 60% in the Korean Medical Licensing Examination (KMLE). The yes/no Angoff (YNA) method, which is easy for panelists to judge, can be considered as an alternative because the KMLE has many items to evaluate. This study aimed to compare the cutoff score and the reliability depending on whether the PA or the YNA standard-setting method was used in the KMLE. Methods The materials were the open-access PA data of the KMLE. The PA data were converted to YNA data in 5 categories, in which the probabilities for a “yes” decision by panelists were 50%, 60%, 70%, 80%, and 90%. SPSS for descriptive analysis and G-string for generalizability theory were used to present the results. Results The PA method and the YNA method counting 60% as “yes,” estimated similar cutoff scores. Those cutoff scores were deemed acceptable based on the results of the Hofstee method. The highest reliability coefficients estimated by the generalizability test were from the PA method and the YNA method, with probabilities of 70%, 80%, 60%, and 50% for deciding “yes,” in descending order. The panelist’s specialty was the main cause of the error variance. The error size was similar regardless of the standard-setting method. Conclusion The above results showed that the PA method was more reliable than the YNA method in estimating the cutoff score of the KMLE. However, the YNA method with a 60% probability for deciding “yes” also can be used as a substitute for the PA method in estimating the cutoff score of the KMLE.
참고문헌 (12)
- Yim MK, "Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination", J Educ Eval Health Prof, 17, pp.14~ (2020)[DOI]
- Park SY, "Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea", J Educ Eval Health Prof, 18, pp.25~ (2021)[DOI]
- Mikyoung Yim, "Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination", Korean Journal of Medical Education, 30(4), pp.347~357 (2018), 한국의학교육학회[DOI]
- Cizek GJ, "Setting performance standards: foundations, methods, and innovations" (2012), Routledge
- Yudkowsky R, "Simpler standards for local performance examinations : the yes", no Angoff and whole-test Ebel, 2008, pp.20, 212~217
- Ahn DS, "A study about standard setting for national licensing exams for health professions" (2018), Korea Health Personnel Licensing Examination
- 박장희, "Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination", Journal of Educational Evaluation for Health Professions, 17, pp.1~10 (2020), 한국보건의료인국가시험원[DOI]
- Ferdous AA, "The use of subsets of test questions in an Angoff standard-setting method", Educ Psychol Meas, 65, pp.185~201 (2005)[DOI]
- Brennan RL, "Generalizability theory" (2001), Springer-Verlag
- Kim J, "How to improve reliability of cut-off scores in dental competency exam : a comparison of rating methods in standard setting", Eur J Dent Educ, 24, pp.734~740 (2020)[DOI]
- Bloch R, "G String V user manual" (2018), Ralph Bloch & Geoff Norman
- "Annual report of the Korea Health Personnel Licensing Examination Institute: technical report of the item analysis"[URL]