표준화 환자를 이용한 진료수행시험에서 교수와 표준화 환자의 채점 정확도
Faculty Observer and Standardized Patient Accuracy in Recording Examinees’ Behaviors Using Checklists in the Clinical Performance Examination
2
조회
0
다운로드
9
인용
0
참고문헌
키워드
초록
Purpose: The purpose of the study was to examine the recording accuracy of faculty observers and standardized patients (SPs) on a clinical performance examination (CPX). Methods: This was a cross-sectional study of a fourth-year medical students’ CPX that was held at a medical school in Seoul, Korea. The CPX consisted of 4 cases and was administered to 118 examinees, with the participation of 52 SP and 45 faculty observers. For the study we chose 15 examinees per case, and analyzed 60 student-SP encounters in total. To determine the recording accuracy level, 2 SP trainers developed an answer key for each encounter. First, we computed agreement rates (P) and kappa coefficient (K) values between the answer key–SPs and the answer key–faculty observers. Secondly, we analyzed variance (ANOVA) with repeated measures to determine whether the mean percentage of the correct checklist score differed as a function of the rater, the case, or the interaction between both factors. Results: Mean P rates ranged from 0.72 to 0.86, while mean K values varied from 0.39 to 0.59. The SP checklist accuracy was higher than that of faculty observersat the level of item comparison. Results from ANOVA showed that there was no significant difference between the percentage of correct scores by the answer key, faculty observers and SPs. There was no significant interaction between rater and case factors. Conclusion: Acceptable levels of recording accuracy were obtained in both rater groups. SP raters can replace faculty raters in a large-scale CPX with thorough preparation.
Abstract
Purpose: The purpose of the study was to examine the recording accuracy of faculty observers and standardized patients (SPs) on a clinical performance examination (CPX). Methods: This was a cross-sectional study of a fourth-year medical students’ CPX that was held at a medical school in Seoul, Korea. The CPX consisted of 4 cases and was administered to 118 examinees, with the participation of 52 SP and 45 faculty observers. For the study we chose 15 examinees per case, and analyzed 60 student-SP encounters in total. To determine the recording accuracy level, 2 SP trainers developed an answer key for each encounter. First, we computed agreement rates (P) and kappa coefficient (K) values between the answer key–SPs and the answer key–faculty observers. Secondly, we analyzed variance (ANOVA) with repeated measures to determine whether the mean percentage of the correct checklist score differed as a function of the rater, the case, or the interaction between both factors. Results: Mean P rates ranged from 0.72 to 0.86, while mean K values varied from 0.39 to 0.59. The SP checklist accuracy was higher than that of faculty observersat the level of item comparison. Results from ANOVA showed that there was no significant difference between the percentage of correct scores by the answer key, faculty observers and SPs. There was no significant interaction between rater and case factors. Conclusion: Acceptable levels of recording accuracy were obtained in both rater groups. SP raters can replace faculty raters in a large-scale CPX with thorough preparation.
참고문헌 (19)
- 고진경, "진료수행시험에서 동일한 사례를 연기한 다수의 표준화 환자 간 채점결과 신뢰도", 한국의학교육, 20(1), pp.61~72 (2008), 한국의학교육학회
- Martin JA, "Who should rate candidates in an objective structured clinical examination?", Acad Med, 71, pp.170~175 (1996)
- McLaughlin K, "Can standardized patients replace physicians as OSCE examiners?", BMC Med Educ, 6, pp.12~ (2006)
- Kopp KC, "Checklist agreement between standardized patients and faculty", J Dent Educ, 59, pp.824~829 (1995)
- MacRae HM, "Comparing checklists and databases with physicians' ratings as measures of students' history and physical-examination skills", Acad Med, 70, pp.313~317 (1995)
- 권복규, "표준화 환자를 이용한 임상수행능력평가시험 (CPX)에서 교수와 표준화 환자의 평가 결과 비교", 한국의학교육, 17(2), pp.173~184 (2005), 한국의학교육학회
- Vu NV, "Standardized (simulated) patients' accuracy in recording clinical performance check-list items", Med Educ, 26, pp.99~104 (1992)
- De Champlain AF, "Standardized patients' accuracy in recording examinees' behaviors using checklists", Acad Med, 72, pp.S85~S87 (1997)
- Heine N, "An analysis of standardised patient checklist errors and their effect on student scores", Med Educ, 37, pp.99~104 (2003)
- Wallace P, "Effect of varying amounts of feedback on standardized patient checklist accuracy in clinical practice examinations", Teach Learn Med, 11, pp.148~152 (1999)
- Colliver JA, "Effects of using two or more standardized patients to simulate the same case on case means and case failure rates", Acad Med, 66, pp.616~618 (1991)
- Cohen J, "A coefficient of agreement for nominal scales", Educ Psychol Meas, 20, pp.37~46 (1960)
- Kim KH, "Comparison of inter-rater consistence and accuracy of estimation for ability parameters among multiple scoring scales", J Educ Eval, 14, pp.327~347 (2001)
- Janjua NZ, "Estimates of intraclass correlation coefficient and design effect for surveys and cluster randomized trials on injection use in Pakistan and developing countries", Trop Med Int Health, 11, pp.1832~1840 (2006)
- Cha JS, "An analytical review of interrater reliability & agreement", J Bus Res, 23, pp.75~102 (1994)
- Landis JR, "The measurement of observer agreement for categorical data", Biometrics, 33, pp.159~174 (1977)
- Fleiss JL, "Design and analysis of clinical experiments" (1986), Wiley
- Vu NV, "Use of standardized patients in clinical assessments: recent developments and measurement findings", Educational Researcher, 23, pp.23~30 (1994)
- Williams RG, "Cognitive, social and environmental sources of bias in clinical performance ratings", Teach Learn Med, 15, pp.270~292 (2003)