Comparison of results between modified-Angoff and bookmark methods for estimating cut score of the Korean medical licensing examination
1
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
Purpose: The purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE. Methods: The process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed. Results: The modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen’s κ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists’ survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method. Conclusion: The modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.
Abstract
Purpose: The purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE. Methods: The process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed. Results: The modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen’s κ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists’ survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method. Conclusion: The modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.
참고문헌 (23)
- Cizek GJ, "Standard setting: a guide to establishing and evaluating performance standards on tests" (2007), Sage Publication Inc.
- 박훈기, "의사국가고시에서 실기시험", Korean Journal of Medical Education, 20(4), pp.309~312 (2008), 한국의학교육학회
- A study of standard setting for Korea dentist clinical skill test, "Shim JS", Korea Health Personnel Licensing Examination Institute, ~, pp.RE02-1411-02 (2014)
- A study of adoption of clinical skill test as Korea nurse licensing examination, "Shin SJ", Korea Health Personnel Licensing Examination institute, ~, pp.RE02-1407-05 (2014)
- Ahn DS, "Standard setting in student assessment by criterion referenced evaluation", Korean J Med Educ, 13(1), pp.41~45 (2001)
- Lee G, "A psychometric approach to setting a passing score on Korean National Medical Licensing Examination", J Educ Eval Health Prof, 1(1), pp.5~14 (2004)
- Jager RM, "Educational Measurement", pp.485~514 (1989), American Council on Education
- Clauser BE, "Educational Measurement", pp.701~731 (2006), Praeger Publishers
- Angoff WH, "Educational Measurement", pp.508~600 (1971), American Council on Education
- Mitzel HC, "Setting Performance Standards: Concepts, Methods, and Perspectives", pp.249~281 (2001), Lawrence Erlbaum Associates
- Buckendahl CW, "Recommending cut scores with a subset of items: an empirical illustration", Pract Assess Res Eval, 15(6), pp.1~10 (2010)
- Peterson CH, "Reliability and validity of bookmark‐based methods for standard setting:comparisons to Angoff‐based methods in the National Assessment of Educational Progress", Educ Meas Issues Pract, 30(2), pp.3~14 (2011)
- Kim TE, "The comparison of applicability: Angoff and bookmark methods in standard setting", pp.2004
- Kim NJ, "The standard setting of clinical performance examination (CPX) by modified-Angoff, bookmark, and item-descriptor matching (IDM) Method", pp.2010
- Çetin S, "A comparison of bookmark and Angoff standard setting methods", Educ Sci Theory Pract, 13(4), pp.2169~2175 (2013)
- Yin P, "A comparison of cut scores and cut score variability from Angoff-based and bookmark-based procedures in standard setting", Paper presented at: the annual meeting of the National Council on Measurement in Education (2005)
- Kane MT, "Setting Performance Standards: Concepts, Methods, and Perspectives", pp.53~88 (2001), Lawrence Erlbaum Associates
- Zimowski MF, "BILOG-MG3.0" (2003), Scientific Software International
- Carraccio CL, "From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills", Acad Med, 83(8), pp.761~767 (2008)
- Ten Cate O, "Medical competence:the interplay between individual ability and the health care environment", Med Teach, 32(8), pp.669~675 (2010)
- Hambleton RK, "Setting Performance Standards:Concepts, Methods, and Perspectives", pp.89~116 (2001), Lawrence Erlbaum Associates
- Karantonis A, "The bookmark standard‐setting method: a literature review", Educ Meas Issues Pract, 25(1), pp.4~12 (2006)
- 장윤선, "문항반응이론에 기초한 준거설정 방법 비교", 교육평가연구, 22(3), pp.659~680 (2009), 한국교육평가학회