The irtQ R package: a user-friendly tool for item response theory-based test data analysis and calibration
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Computerized adaptive testing (CAT) has become a widely adopted test design for high-stakes licensing and certification exams, particularly in the health professions in the United States, due to its ability to tailor test difficulty in real time, reducing testing time while providing precise ability estimates. A key component of CAT is item response theory (IRT), which facilitates the dynamic selection of items based on examinees' ability levels during a test. Accurate estimation of item and ability parameters is essential for successful CAT implementation, necessitating convenient and reliable software to ensure precise parameter estimation. This paper introduces the irtQ R package (http://CRAN.R-project.org/), which simplifies IRT-based analysis and item calibration under unidimensional IRT models. While it does not directly simulate CAT, it provides essential tools to support CAT development, including parameter estimation using marginal maximum likelihood estimation via the expectation-maximization algorithm, pretest item calibration through fixed item parameter calibration and fixed ability parameter calibration methods, and examinee ability estimation. The package also enables users to compute item and test characteristic curves and information functions necessary for evaluating the psychometric properties of a test. This paper illustrates the key features of the irtQ package through examples using simulated datasets, demonstrating its utility in IRT applications such as test data analysis and ability scoring. By providing a user-friendly environment for IRT analysis, irtQ significantly enhances the capacity for efficient adaptive testing research and operations. Finally, the paper highlights additional core functionalities of irtQ, emphasizing its broader applicability to the development and operation of IRT-based assessments.
Abstract
Computerized adaptive testing (CAT) has become a widely adopted test design for high-stakes licensing and certification exams, particularly in the health professions in the United States, due to its ability to tailor test difficulty in real time, reducing testing time while providing precise ability estimates. A key component of CAT is item response theory (IRT), which facilitates the dynamic selection of items based on examinees' ability levels during a test. Accurate estimation of item and ability parameters is essential for successful CAT implementation, necessitating convenient and reliable software to ensure precise parameter estimation. This paper introduces the irtQ R package (http://CRAN.R-project.org/), which simplifies IRT-based analysis and item calibration under unidimensional IRT models. While it does not directly simulate CAT, it provides essential tools to support CAT development, including parameter estimation using marginal maximum likelihood estimation via the expectation-maximization algorithm, pretest item calibration through fixed item parameter calibration and fixed ability parameter calibration methods, and examinee ability estimation. The package also enables users to compute item and test characteristic curves and information functions necessary for evaluating the psychometric properties of a test. This paper illustrates the key features of the irtQ package through examples using simulated datasets, demonstrating its utility in IRT applications such as test data analysis and ability scoring. By providing a user-friendly environment for IRT analysis, irtQ significantly enhances the capacity for efficient adaptive testing research and operations. Finally, the paper highlights additional core functionalities of irtQ, emphasizing its broader applicability to the development and operation of IRT-based assessments.
참고문헌 (30)
- Huh S, "Preparing the implementation of computerized adaptive testing for high-stakes examinations", J Educ Eval Health Prof, 5, pp.1~ (2008)[DOI]
- Seo DG, "Overview and current management of computerized adaptive testing in licensing", certification examinations, 2017, pp.14, 17~
- Seo DG, "Introduction to the LIVECAT web-based computerized adaptive testing platform", J Educ Eval Health Prof, 17, pp.27~ (2020)[DOI]
- Yan D, "Adaptive testing with regression trees in the presence of multidimensionality", J Educ Behav Stat, 29, pp.293~316 (2004)[DOI]
- R Development Core Team, "R: a language and environment for statistical computing", R Foundation for Statistical Computing, ~ (2023)
- Bock RD, "Marginal maximum likelihood estimation of item parameters : application of an EM algorithm", Psychometrika, 46, pp.443~459 (1981)[DOI]
- Kim S, "A comparative study of IRT fixed parameter calibration methods", J Educ Meas, 43, pp.355~381 (2006)[DOI]
- "Scale drift in on-line calibration. ETS Res Rep Ser 1988" (1988)[URL]
- Woods CM, "Empirical histograms in item response theory with ordinal data", Educ Psychol Meas, 67, pp.73~87 (2007)[DOI]
- Han KT, "Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests", Appl Psychol Meas, 40, pp.289~301 (2016)[DOI]
- Warm TA, "Weighted likelihood estimation of ability in item response theory", Psychometrika, 54, pp.427~450 (1989)[DOI]
- Hambleton RK, "Fundamentals of item response theory" (1991), Sage Publications
- Bock RD, "Adaptive EAP estimation of ability in a microcomputer environment", Appl Psychol Meas, 6, pp.431~444 (1982)[DOI]
- Thissen D, "Item response theory for scores on tests including polytomous items with ordered responses", Appl Psychol Meas, 19, pp.39~49 (1995)[DOI]
- Thissen D, "Test scoring", pp.73~140 (2001), Lawrence Erlbaum Associates Publishers
- Kolen MJ, "Psychometric properties of IRT proficiency estimates", Educ Meas Issues Pract, 29, pp.8~14 (2010)[DOI]
- Ban JC, "A comparative study of on-line pretest item : calibration", scaling methods in computerized adaptive testing, 2001, pp.38, 191~212
- Chen P, "A new online calibration method for multidimensional computerized adaptive testing", Psychometrika, 81, pp.674~701 (2016)[DOI]
- Wainer H, "Computerized adaptive testing: a primer", pp.65~102 (1990), Lawrence Erlbaum
- Bock RD, "Handbook of modern item response theory", pp.433~448 (1997), Springer
- Orlando M, "Likelihood-based item-fit indices for dichotomous item response theory models", Appl Psychol Meas, 24, pp.50~64 (2000)[DOI]
- Lord FM, "Comparison of IRT true-score and equipercentile observed-score"equatings", Appl Psychol Meas, 8, pp.453~461 (1984)[DOI]
- Lee WC, "Classification consistency and accuracy for complex assessments using item response theory", J Educ Meas, 47, pp.1~17 (2010)[DOI]
- Rudner LM, "Computing the expected proportions of misclassified examinees", Pract Assess Res Eval, 7, pp.14~ (2000)[DOI]
- Rudner LM, "Expected classification accuracy", Pract Assess Res Eval, 10, pp.13~ (2005)[DOI]
- Lim H, "Detecting differential item functioning in CAT using IRT residual DIF approach", J Educ Meas, 60, pp.626~650 (2023)[DOI]
- Lim H, "A residual-based differential item functioning detection framework in item response theory", J Educ Meas, 59, pp.80~104 (2022)[DOI]
- Nandakumar R, "Evaluation of the CATSIB DIF procedure in a pretest setting", J Educ Behav Stat, 29, pp.177~199 (2004)[DOI]
- Lim H, "A recursion-based analytical approach to evaluate the performance of MST", J Educ Meas, 58, pp.154~178 (2021)[DOI]
- Cappaert KJ, "Evaluating CAT-adjusted approaches for suspected item parameter drift detection", Meas Interdiscip Res Perspect, 16, pp.226~238 (2018)[DOI]