The relationship between classical item characteristics and item response time on computer-based testing
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
Purpose: This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship. Methods: We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT. Results: iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination. Conclusion: The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.
Abstract
Purpose: This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship. Methods: We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT. Results: iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination. Conclusion: The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.
참고문헌 (16)
- Son CG, "Curriculum & educational evaluation" (2012), Teayong
- 임현선, "문항반응이론과 고전검사이론을 이용한 진료수행시험의 문항 분석", Korean Journal of Medical Education, 19(3), pp.185~196 (2007), 한국의학교육학회
- "Understanding item analyses"[URL]
- Huh S, "Preparing the implementation of computerized adaptive testing for high-stakes examinations", J Educ Eval Health Prof, 5, pp.1~ (2008)
- Wise SL, "An investigation of the differential effort received by items on a low-stakes computer-based test", Appl Meas Educ, 19(2), pp.95~114 (2006)
- Lee YH, "Introduction to an open source internet-based testing program for medical student examinations", J Educ Eval Health Prof, 6, pp.4~ (2009)
- Bergstrom BA, "Computerized adaptive testing exploring examinee response time using hierarchical linear modeling", Paper presented at: the Annual Meeting of the National Council on Measurement in Education (1994)
- "New applications in testing: using response time to increase the construct validity of a latent trait estimate" (2004)[URL]
- Halkitsis PN, "Estimating testing time: the effects of item characteristics on response latency", Paper presented at: the Annual Meeting of the American Educational Research Association (1996)
- Yang CL, "Examining item difficulty and response time on perceptual ability test items", J Appl Meas, 3(3), pp.282~299 (2002)
- Wise SL, "Response time effort : a new measure of examinee motivation in computer-based tests", Appl Meas Educ, 18(2), pp.163~183 (2005)
- Wise SL, "An investigation of the differential effort received by items on a low-stakes computer-based test", Appl Meas Educ, 19(2), pp.95~114 (2006)
- "Interpreting exam performance: what do those stats mean anyway?"[URL]
- Wolf LF, "Consequence of performance, test, motivation, and mentally taxing items", Appl Meas Educ, 8(4), pp.341~351 (1995)
- Schnipke DL, "Computer-Based Testing: Building the Foundation for Future Assessments", pp.244~ (2002), Lawrence Erlbaum Associates
- Kingsbury GG, "Procedures for using response latencies to identify unusual test performance in computerized adaptive tests", Paper presented at: the Annual Meeting of the National Council on Measurement in Education (1993)