임상수행능력평가(CPX)에서 학교효과가 신뢰도에 미치는 영향
The School Effect on the Reliability of Clinical Performance Examination in Medical Schools
2
조회
0
다운로드
8
인용
0
참고문헌
키워드
초록
Purpose: The purpose of this study is to test the reliability of the clinical performance examination (CPX) using Generalizability theory (G-theory). Through G-theory, the effects of not only students and tasks but also the school will be analyzed as primary sources of error, which can affect the interpretation of the reliability of the CPX. Methods: One thousand three hundred nineteen students from 16 medical schools that participated in the Seoul-Gyeonggi CPX Consortium 2008 were enrolled. In our research design, we suppose that student is nested within school and crossed with task. Data analysis was conducted with urGenova. Results: According to our analysis, the percentage of error variance was 6.2% for school, 14.9% for student nested within school,14.4% for task, and 3% for interaction between school and task. An effect of school on students was observed, but the interaction between task and school was insignificant. When student is nested within school, the universe score decreased and the g-coefficient was less than the g-coefficient of the p × t (p: studentm, t: task) design. Conclusion: The results show that generalizability theory is useful in detecting various error components in the CPX. Using the generalizability theory to improve the technical quality of performance assessments provides us with greater information compared with traditional test theories.
Abstract
Purpose: The purpose of this study is to test the reliability of the clinical performance examination (CPX) using Generalizability theory (G-theory). Through G-theory, the effects of not only students and tasks but also the school will be analyzed as primary sources of error, which can affect the interpretation of the reliability of the CPX. Methods: One thousand three hundred nineteen students from 16 medical schools that participated in the Seoul-Gyeonggi CPX Consortium 2008 were enrolled. In our research design, we suppose that student is nested within school and crossed with task. Data analysis was conducted with urGenova. Results: According to our analysis, the percentage of error variance was 6.2% for school, 14.9% for student nested within school,14.4% for task, and 3% for interaction between school and task. An effect of school on students was observed, but the interaction between task and school was insignificant. When student is nested within school, the universe score decreased and the g-coefficient was less than the g-coefficient of the p × t (p: studentm, t: task) design. Conclusion: The results show that generalizability theory is useful in detecting various error components in the CPX. Using the generalizability theory to improve the technical quality of performance assessments provides us with greater information compared with traditional test theories.
참고문헌 (19)
- Seong TJ, "Educational evaluation" (2002), Hakjisa
- "Seoul-Gyeonggi CPX Consortium"[URL]
- Kim JH, "Role and developmental direction of Seoul-Gyeonggi CPX Consortium", 4th CPX Symposium (2007)
- Kim SS, "Generalizability theory" (2001), Kyoyookbook
- Brennan RL, "Generalizability theory: statistics for social science and public policy" (2001), Springer
- Cronbach LJ, "The generalizability of behavioral measurement: the theory of generalizability for scores and profiles" (1972), John Wiley
- Nam MH, "An application of the generalizability theory to performance assessment", J Educ Eval, 9, pp.73~93 (1996)
- 강애남, "학생들의 동료평가를 활용한 수행평가 결과의 일반화가능도 분석", 교육평가연구, 19(3), pp.107~121 (2006), 한국교육평가학회
- Kim CH, "A generalizability theory approach to investigating rater effects with a national exam composed of constructed-response items", J Curr Eval, 11, pp.141~159 (2008)
- 노혜린, "의료윤리 능력 평가로서의 객관구조화 구술시험 개발 경험", 한국의학교육, 21(1), pp.23~33 (2009), 한국의학교육학회
- 임형, "임상수행능력평가(CPX)의 오차요인 탐색과 신뢰도 연구", 교육평가연구, 18(1), pp.27~46 (2005), 한국교육평가학회
- Lee YM, "Investigation of error factor of CPX using generalizability theory", 2nd CPX Symposium (2005), Seoul National University Hospital
- Park JH, "A psychometric evaluation of CPX in relation to validity and reliability", pp.2008
- Brennan RL, "The conventional wisdom about group mean scores", J Educ Meas, 32, pp.385~396 (1995)
- Brennan RL, "Manual for urGENOVA version 2.1" (2001), Iowa Testing Programs, University of Iowa
- Crick JE, "Manual for GENOVA: a gerneralized analysis of variation system" (1983), Research and Development Division, American College Testing Program
- Korean Society for Education Evaluation, "Educational evaluation thesaurus" (2004), Hakjisa
- 김 선, "진료수행시험 평가자로서 표준화 환자 활용의 타당성", 한국의학교육, 17(2), pp.163~172 (2005), 한국의학교육학회
- Kang SJ, "Regression analysis" (2003), Kyoyookbook