Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT). Methods This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada). Results All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%. Conclusion The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.
Abstract
Purpose This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT). Methods This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada). Results All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%. Conclusion The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.
참고문헌 (12)
- Schuwirth LW, "Understanding medical education: evidence, theory, and practice", pp.275~289 (2018), Wiley-Blackwell
- Crossley J, "Generalisability : a key to unlock professional assessment", Med Educ, 36, pp.972~978 (2002)[DOI]
- Brennan RL, "Generalizability theory", Educ Meas, 11, pp.27~34 (1992)[DOI]
- Vaughan B, "Reliability of a viva assessment of clinical reasoning in an Australian pre-professional osteopathy program assessed using generalizability theory", J Educ Eval Health Prof, 14, pp.1~ (2017)[DOI]
- Downing SM, "Reliability : on the reproducibility of assessment data", Med Educ, 38, pp.1006~1012 (2004)[DOI]
- Brannick MT, "A systematic review of the reliability of objective structured clinical examination scores", Med Educ, 45, pp.1181~1189 (2011)[DOI]
- Swanson DB, "Factors influencing reproducibility of tests using standardized patients", Teach Learn Med, 1, pp.158~166 (1989)[DOI]
- Kreiter CD, "Case specificity : empirical phenomenon or measurement artifact", Teach Learn Med, 19, pp.378~381 (2007)[DOI]
- Newble D, "Techniques for measuring clinical competence : objective structured clinical examinations", Med Educ, 38, pp.199~203 (2004)[DOI]
- Schuwirth LW, "The use of clinical simulations in assessment", Med Educ, (37 Suppl 1), pp.65~71 (2003)[DOI]
- Norman G, "How specific is case specificity", Med Educ, 40, pp.618~623 (2006)[DOI]
- Blood AD, "Neurology objective structured clinical examination reliability using generalizability theory", Neurology, 85, pp.1623~1629 (2015)[DOI]