Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Purpose: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence onthe use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. Methods: Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residencyprograms to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites wasvideo-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). Results: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019). Conclusion: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.
Abstract
Purpose: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence onthe use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into 6 elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. Methods: Messick’s unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from 3 pediatric residencyprograms to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites wasvideo-recorded during 2018–2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). Results: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone (r=0.34, P=0.019). Conclusion: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.
참고문헌 (15)
- "Training requirements for the specialty of internal medicine" (2016)[URL]
- (2023)[URL]
- Fromme HB, "Pediatric resident-as-teacher curricula : a national survey of existing programs and future needs", J Grad Med Educ, 3, pp.168~175 (2011)[DOI]
- Post RE, "Residents-as-teachers curricula : a critical review", Acad Med, 84, pp.374~380 (2009)[DOI]
- Newman L, "Resident-as-teacher DVD series", MedEdPORTAL, 11, pp.10152~ (2015)[DOI]
- Zackoff M, "An observed structured teaching evaluation demonstrates the impact of a resident-as-teacher curriculum on teaching competency", Hosp Pediatr, 5, pp.342~347 (2015)[DOI]
- Frey-Vogel AS, "The development of an instrument for faculty to assess resident-led large group teaching", Acad Pediatr, 20, pp.442~447 (2020)[DOI]
- Newman LR, "Developing a peer assessment of lecturing instrument : lessons learned", Acad Med, 84, pp.1104~1110 (2009)[DOI]
- "The Pediatrics Milestone Project" (2012)[URL]
- Cicchetti DV, "Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology", Psychol Assess, 6, pp.284~290 (1994)[DOI]
- Yudkowsky R, "Assessment in health professions education" (2019), Routledge
- Park YS, "Competency evaluations in the next accreditation system : contributing to guidelines and implications", Teach Learn Med, 28, pp.135~145 (2016)[DOI]
- Li ST, "Competent for unsupervised practice: use of pediatric residency training milestones to assess readiness", Acad Med, 92, pp.385~393 (2017)[DOI]
- Newman LR, "Frame-of-reference training : establishing reliable assessment of teaching effectiveness", J Contin Educ Health Prof, 36, pp.206~210 (2016)[DOI]
- Dunnington GL, "A prospective randomized trial of a residents-as-teachers training program", Acad Med, 73, pp.696~700 (1998)[DOI]