Inter-rater reliability and content validity of the measurement tool for portfolio assessments used in the Introduction to Clinical Medicine course at Ewha Womans University College of Medicine: a methodological study
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose This study aimed to examine the reliability and validity of a measurement tool for portfolio assessments in medical education. Specifically, it investigated scoring consistency among raters and assessment criteria appropriateness according to an expert panel. Methods A cross-sectional observational study was conducted from September to December 2018 for the Introduction to Clinical Medicine course at the Ewha Womans University College of Medicine. Data were collected for 5 randomly selected portfolios scored by a gold-standard rater and 6 trained raters. An expert panel assessed the validity of 12 assessment items using the content validity index (CVI). Statistical analysis included Pearson correlation coefficients for rater alignment, the intraclass correlation coefficient (ICC) for inter-rater reliability, and the CVI for item-level validity. Results Rater 1 had the highest Pearson correlation (0.8916) with the gold-standard rater, while Rater 5 had the lowest (0.4203). The ICC for all raters was 0.3821, improving to 0.4415 after excluding Raters 1 and 5, indicating a 15.6% reliability increase. All assessment items met the CVI threshold of ≥0.75, with some achieving a perfect score (CVI=1.0). However, items like “sources” and “level and degree of performance” showed lower validity (CVI=0.72). Conclusion The present measurement tool for portfolio assessments demonstrated moderate reliability and strong validity, supporting its use as a credible tool. For a more reliable portfolio assessment, more faculty training is needed.
Abstract
Purpose This study aimed to examine the reliability and validity of a measurement tool for portfolio assessments in medical education. Specifically, it investigated scoring consistency among raters and assessment criteria appropriateness according to an expert panel. Methods A cross-sectional observational study was conducted from September to December 2018 for the Introduction to Clinical Medicine course at the Ewha Womans University College of Medicine. Data were collected for 5 randomly selected portfolios scored by a gold-standard rater and 6 trained raters. An expert panel assessed the validity of 12 assessment items using the content validity index (CVI). Statistical analysis included Pearson correlation coefficients for rater alignment, the intraclass correlation coefficient (ICC) for inter-rater reliability, and the CVI for item-level validity. Results Rater 1 had the highest Pearson correlation (0.8916) with the gold-standard rater, while Rater 5 had the lowest (0.4203). The ICC for all raters was 0.3821, improving to 0.4415 after excluding Raters 1 and 5, indicating a 15.6% reliability increase. All assessment items met the CVI threshold of ≥0.75, with some achieving a perfect score (CVI=1.0). However, items like “sources” and “level and degree of performance” showed lower validity (CVI=0.72). Conclusion The present measurement tool for portfolio assessments demonstrated moderate reliability and strong validity, supporting its use as a credible tool. For a more reliable portfolio assessment, more faculty training is needed.
참고문헌 (9)
- Duque G, "Learning while evaluating : the use of an electronic evaluation portfolio in a geriatric medicine clerkship", BMC Med Educ, 6, pp.4~ (2006)[DOI]
- Van Tartwijk J, "Portfolios for assessment and learning: AMEE guide no. 45", Med Teach, 31, pp.790~801 (2009)[DOI]
- Bae HS, "An exploitative review of quality control on portfolio evaluation", J Educ Eval, 10, pp.75~104 (1997)
- van der Vleuten CP, "Assessing professional competence : from methods to programmes", Med Educ, 39, pp.309~317 (2005)[DOI]
- Roberts C, "The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement", BMC Med Educ, 14, pp.197~ (2014)[DOI]
- Grant AJ, "Exploring students’ perceptions on the use of significant event analysis, as part of a portfolio assessment process in general practice, as a tool for learning how to use reflection in learning", BMC Med Educ, 7, pp.5~ (2007)[DOI]
- "Measuring and promoting inter-rater agreement of teacher and principal performance ratings" (2012)[URL]
- Driessen E, "Portfolios in medical education : why do they meet with mixed success? : a systematic review", Med Educ, 41, pp.1224~1233 (2007)[DOI]
- Yoo DM, "Development and validation of a portfolio assessment system for medical schools in Korea", J Educ Eval Health Prof, 17, pp.39~ (2020)[DOI]