Messick의 타당도 틀을 활용한 임상실습 전 실기시험의 타당도 평가
Assessing the Validity of the Preclinical Objective Structured Clinical Examination Using Messick’s Validity Framework
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Students must be familiar with clinical skills before starting clinical practice to ensure patients’ safety and enable efficient learning. However, performance is mainly tested in the third or fourth years of medical school, and studies using the validity framework have not been reported in Korea. We analyzed the validity of a performance test conducted among second-year students classified into content, response process, internal structure, relationships with other variables, and consequences according to Messick’s framework. As results of the analysis, content validity was secured by developing cases according to a pre-determined blueprint. The quality of the response process was controlled by training and calibrating raters. The internal structure showed that (1) reliability by generalizability theory was acceptable (coefficients of 0.724 and 0.786, respectively, for day 1 and day 2), and (2) the relevant domains had proper correlations, while the clinical performance examination (CPX) and objective structured clinical examination (OSCE) showed weaker relationships. OSCE/CPX scores were correlated with other variables, especially grade point average and oral structured exam scores. The consequences of this assessment were (1) making students learn clinical skills and study themselves, while causing too much stress for students due to lack of motivation; (2) reminding educators of the need to apply practical teaching methods and to give feedback on the test results; and (3) providing an opportunity for faculty to consider developing support programs. It is necessary to develop the blueprint more precisely according to students’ level and to verify the validity of the response process with statistical methods.
Abstract
Students must be familiar with clinical skills before starting clinical practice to ensure patients’ safety and enable efficient learning. However, performance is mainly tested in the third or fourth years of medical school, and studies using the validity framework have not been reported in Korea. We analyzed the validity of a performance test conducted among second-year students classified into content, response process, internal structure, relationships with other variables, and consequences according to Messick’s framework. As results of the analysis, content validity was secured by developing cases according to a pre-determined blueprint. The quality of the response process was controlled by training and calibrating raters. The internal structure showed that (1) reliability by generalizability theory was acceptable (coefficients of 0.724 and 0.786, respectively, for day 1 and day 2), and (2) the relevant domains had proper correlations, while the clinical performance examination (CPX) and objective structured clinical examination (OSCE) showed weaker relationships. OSCE/CPX scores were correlated with other variables, especially grade point average and oral structured exam scores. The consequences of this assessment were (1) making students learn clinical skills and study themselves, while causing too much stress for students due to lack of motivation; (2) reminding educators of the need to apply practical teaching methods and to give feedback on the test results; and (3) providing an opportunity for faculty to consider developing support programs. It is necessary to develop the blueprint more precisely according to students’ level and to verify the validity of the response process with statistical methods.
참고문헌 (25)
- Kim BS, "Evaluation of introduction to clinical medicine by objective structured clinical examination", Korean J Med Educ, 13(2), pp.289~298 (2001)
- Godefrooij MB, "Students’ perceptions about the transition to the clinical phase of a medical curriculum with preclinical patient contacts; a focus group study", BMC Med Educ, 10, pp.28~ (2010)
- Graham R, "Reliability and predictive validity of a comprehensive preclinical OSCE in dental education", J Dent Educ, 77(2), pp.161~167 (2013)
- Cook DA, "A contemporary approach to validity arguments : a practical guide to Kane’s framework", Med Educ, 49(6), pp.560~575 (2015)
- Cook DA, "Validation of educational assessments : a primer for simulation and beyond", Adv Simul(Lond), 1, pp.31~ (2016)
- Messick S, "Standards of validity and the validity of standards in performance assessment", Educ Meas Issues Pract, 14(4), pp.5~8 (1995)
- American Educational Research Association, "Standards for educational and psychological testing", American Educational Research Association, ~ (2014)
- Downing SM, "Validity : on meaningful interpretation of assessment data", Med Educ, 37(9), pp.830~837 (2003)
- Cook DA, "What counts as validity evidence? : examples and prevalence in a systematic review of simulation-based assessment", Adv Health Sci Educ Theory Pract, 19(2), pp.233~250 (2014)
- Borgersen NJ, "Gathering validity evidence for surgical simulation : a systematic review", Ann Surg, 267(6), pp.1063~1068 (2018)
- Pugh D, "A procedural skills OSCE : assessing technical and non-technical skills of internal medicine residents", Adv Health Sci Educ Theory Pract, 20(1), pp.85~100 (2015)
- Brennan RL, "Generalizability theory" (2001), Springer-Verlag
- Dancey CP, "Statistics without maths for psychology" (2004), Pearson Prentice Hall
- Liao SC, "Comparison between inter-rater reliability and inter-rater agreement in performance assessment", Ann Acad Med Singap, 39(8), pp.613~618 (2010)
- 김성해, "감정지능이 진료수행시험의 환자-의사관계 점수에 미치는 영향", Korean Journal of Medical Education, 23(3), pp.159~165 (2011), 한국의학교육학회
- Kiyohara LY, "The patient-physician interactions as seen by undergraduate medical students", Sao Paulo Med J, 119(3), pp.97~100 (2001)
- Downing SM, "Assessment in health professions education" (2009), Routledge
- Iramaneerat C, "Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement", Adv Health Sci Educ Theory Pract, 13(4), pp.479~493 (2008)
- Wimmers PF, "The impact of case specificity and generalisable skills on clinical performance : a correlated traits-correlated methods approach", Med Educ, 42(6), pp.580~588 (2008)
- Bakhsh TM, "Comparison of students’ performance in the traditional oral clinical examination and the objective structured clinical examination", Saudi Med J, 30(4), pp.555~557 (2009)
- Remmen R, "Correlation of a written test of skills and a performance based test : a study in two traditional medical schools", Med Teach, 23(1), pp.29~32 (2001)
- Dadgar SR, "OSCE as a tool for evaluation of practical semiology in comparison to MCQ & oral examination", J Pak Med Assoc, 58(9), pp.506~507 (2008)
- Cilliers FJ, "A model of the pre-assessment learning effects of summative assessment in medical education", Adv Health Sci Educ Theory Pract, 17(1), pp.39~53 (2012)
- 윤소정, "의과대학생은 시험을 준비하기 위해 어떻게 공부하는가: 평가 전 인지 및 메타인지 활동", 의학교육논단, 21(1), pp.51~58 (2019), 연세대학교 의과대학[DOI]
- 김종훈, "재교육으로서 비디오를 이용한 피드백이 임상수행평가 성적에 미치는 영향", 의학교육논단, 14(1), pp.51~56 (2012), 연세대학교 의과대학[DOI]