Development and validation of a portfolio assessment system for medical schools in Korea
2
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
Purpose: Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability. Methods: We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020. Results: The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher. Conclusion: We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.
Abstract
Purpose: Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability. Methods: We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020. Results: The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher. Conclusion: We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.
참고문헌 (11)
- 유동미, "Evaluation of a portfolio-based course on self-development for pre-medical students in Korea", Journal of Educational Evaluation for Health Professions, 16, pp.1~7 (2019), 한국보건의료인국가시험원[DOI]
- Heeneman S, "The use of a portfolio in postgraduate medical education: reflect, assess and account, one for each or all in one?", GMS J Med Educ, 34 (2017)[DOI]
- Jan Van Tartwijk, "Portfolios for assessment and learning: AMEE Guide no. 45", Medical Teacher, 31(9), pp.790~801 (2009), Informa UK Limited[DOI]
- Lee JW, "An analysis of the educational effectiveness and goodness of portfolio assessment", pp.2002
- Bae HS, "An exploitative review of quality control on portfolio evaluation", J Educl Eval, 10, pp.75~104 (1997)
- Cherie Lucas, "Inter-rater reliability of a reflective rubric to assess pharmacy students’ reflective thinking", Currents in Pharmacy Teaching and Learning, 9(6), pp.989~995 (2017), Elsevier BV[DOI]
- 이은하, "학문 목적 한국어 쓰기 수행 평가의 채점자 신뢰도 검증 - 다국면 Rasch 모형을 활용하여 -", 새국어교육, (103), pp.311~354 (2015), 한국국어교육학회[DOI]
- Celia Laird O’Brien, "Feasibility and Outcomes of Implementing a Portfolio Assessment System Alongside a Traditional Grading System", Academic Medicine, 91(11), pp.1554~1560 (2016), Ovid Technologies (Wolters Kluwer Health)[DOI]
- Margery H. Davis, "Examiner perceptions of a portfolio assessment process", Medical Teacher, 32(5), pp.e211~e215 (2010), Informa UK Limited[DOI]
- Cynthia C. Gadbury-Amyot, "Validity and Reliability of Portfolio Assessment of Student Competence in Two Dental School Populations: A Four-Year Study", Journal of Dental Education, 78(5), pp.657~667 (2014), Wiley[DOI]
- Chris Roberts, "The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement", BMC Medical Education, 14(1) (2014), Springer Science and Business Media LLC[DOI]