진료수행시험에서 동일한 사례를 연기한 다수의 표준화 환자 간 채점결과 신뢰도
Inter-rater Reliability in a Clinical Performance Examination Using Multiple Standardized Patientsfor the Same Case
2
조회
0
다운로드
8
인용
0
참고문헌
키워드
초록
Purpose: The “standardization” of standardized patients (SP) is one of the most crucial factors for a successful clinical performance examination (CPX). This study aimed to examine the inter-rater reliability among SPs who portrayed the same case during a CPX. Methods: The context was a CPX conducted under the supervision of CPX Seoul-Gyeonggi Consortium in K medical school in August 2007. K medical school ran 12 stations consisting of duplicated sets of 6 cases. In total, thirty SPs participated with 5 SPs acting each of the 6 cases. The SPs evaluated the student’s performances in addition to portraying the cases. ANCOVA (analysis of covariance) was used to compare scores rated by the different SPs. The dependent variables were the case scores and the 4 subcomponent (history taking, physical examination, Clinical courtesy, and Patient-physician interaction) scores for each case; the independent variable was the SPs; and the covariate was the CPX total score. Results: The Headache and Cough stations showed an acceptable level of reliability. Otherwise, Weight Loss and Facial Flushing failed to show consistent scores in all 4 subcomponents. Diarrhea and Lt. hemiparesis showed partial consistency. In terms of the subcomponents, the physical exam scores were most consistent and the patient-physician interaction scores were most inconsistent. Conclusion: This study tested the level of “standardization” of one set of CPX cases with mixed results. The authors hope that our results will contribute to quality assurance of CPX.
Abstract
Purpose: The “standardization” of standardized patients (SP) is one of the most crucial factors for a successful clinical performance examination (CPX). This study aimed to examine the inter-rater reliability among SPs who portrayed the same case during a CPX. Methods: The context was a CPX conducted under the supervision of CPX Seoul-Gyeonggi Consortium in K medical school in August 2007. K medical school ran 12 stations consisting of duplicated sets of 6 cases. In total, thirty SPs participated with 5 SPs acting each of the 6 cases. The SPs evaluated the student’s performances in addition to portraying the cases. ANCOVA (analysis of covariance) was used to compare scores rated by the different SPs. The dependent variables were the case scores and the 4 subcomponent (history taking, physical examination, Clinical courtesy, and Patient-physician interaction) scores for each case; the independent variable was the SPs; and the covariate was the CPX total score. Results: The Headache and Cough stations showed an acceptable level of reliability. Otherwise, Weight Loss and Facial Flushing failed to show consistent scores in all 4 subcomponents. Diarrhea and Lt. hemiparesis showed partial consistency. In terms of the subcomponents, the physical exam scores were most consistent and the patient-physician interaction scores were most inconsistent. Conclusion: This study tested the level of “standardization” of one set of CPX cases with mixed results. The authors hope that our results will contribute to quality assurance of CPX.
참고문헌 (26)
- Ahn, D., "Standard setting in student assessment by criterion referenced evaluation", Korean Journal of Medical Education, 13(1), pp.41~45 (2001)
- Auewarakul, C., "Item analysis to improve reliability for an internal medicine undergraduate OSCE", Adv Health Sci Educ Theory Pract, 10(2), pp.105~113 (2005)
- Boulet, J.R., "Quality assurance methods for performance-based assessments", Adv Health Sci Educ Theory Pract, 8(1), pp.27~47 (2003)
- Brennan, R., "Performance assessments from the perspective of generalizability theory", Applied Psychological Measurement, 24(4), pp.339~353 (2000)
- Chesser, A.M., "Factor analysis can be a useful standard setting tool in a high stakes OSCE assessment", Med Educ, 38(8), pp.825~831 (2004)
- "CPX Seoul-Gyeonggi Consortium"[URL]
- Crossley, J., "Generalisability: a key to unlock professional assessment", Med Educ, 36(10), pp.972~978 (2002)
- de Champlain, A.F., "Standardised patients' accuracy in recording examinees' behaviours using checklists", Acad Med, 72(10), pp.S85~S87 (1997)
- Downing, S., "Reliability: on the reproducibility if assessment data", Med Educ, 38, pp.1006~1012 (2004)
- Heine, N., "An analysis of standardized patient checklist errors and their effect on student scores", Med Educ, 37, pp.99~104 (2003)
- Im, H., "A study of investigating error sources and reliability for clinical performance examination (CPX).", Journal of Educational Evaluation, 18(1), pp.27~46 (2005)
- Kim, J., "Analysis of the evaluation for clinical performance examination using standardized patients in one medical school", Korean Journal of Medical Education, 16(1), pp.51~61 (2004)
- Kim, K., "Comparison of interrater consistence and accuracy of estimation for ability parameters among multiple scoring scales", Journal of Educational Evaluation, 14(1), pp.327~347 (2001)
- Kim, S., "The appropriateness of using standardized patients (SPs) assessment scores in clinical performance examination (CPX)", Korean Journal of Medical Education, 17(2), pp.163~172 (2005)
- Kwon, I., "Comparison of the evaluation results of faculty with those of standardized patients in a clinical performance examination experience", Korean Journal of Medical Education, 17(2), pp.173~183 (2005)
- Martin, J.A., "Who should rate candidates in an objective structured clinical examination?", Acad Med, 71(2), pp.170~175 (1996)
- Mazor, K.M., "The relationship between checklist scores on a communication OSCE and analogue patients’ perceptions of communication", Adv Health Sci Educ Theory Pract, 10(1), pp.37~51 (2005)
- Nieman, L.Z., "Training and validating the use of geriatric simulated patients", Res Med Educ, 27, pp.154~159 (1988)
- Pangaro, L.N., "Performance of “standardized examinees” in a standardizedpatient examination of clinical skills", Acad Med, 72(11), pp.1008~1011 (1997)
- Park, H., "Sharing of information among students and its effect on the scores of clinical performance examination", Korean Journal of Medical Education, 17(2), pp.185~195 (2005)
- Park, W., "Correlation of CPX scores with the scores of the clinical clerkship assessments and written examinations", Korean Journal of Medical Education, 17(3), pp.297~303 (2005)
- Solomon, D.J., "Source of measurement error in an ECG examination:implications for performance-based assessments", adv Health Sci Educ Theory Pract, 9(4), pp.283~290 (2004)
- Solomon, D.J., "Global rating of student performance in a standardized patient examination: is the whole more than the sum of the parts?", Adv Health Sci Educ Theory Pract, 5(2), pp.131~140 (2000)
- Sung, T.(, "Nonsulhyung gosawa yecheneuinggye silgigosareul wihan chejumjagan silroedo chujeong", Journal of Educational Evaluation, 7(1), pp.43~56 (1994)
- Tamblyn, R., "Sources of unreliability and bias in standardised patient rating", Teaching Learning Med, 3, pp.74~85 (1991)
- Wang, W., "The effect of fatigue on the accuracy of standardised patients' checklist recording", Teaching Learning Med, 8, pp.148~151 (1996)