Feasibility of Using Prior Information about Predicted Item Difficulty in Increasing the Accuracy of Item Parameter Estimation and IRT Equating
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
For item banking or computerized adaptive testing to be successful, it is of vital importance to ensure the accuracy of item parameter estimation, especially when calibration needs to be conducted with the limited number of examinees for security reasons. This study investigated whether judgmental information about item difficulty would improve the accuracy of parameter estimation when used as prior information. Performance of using predictions of judges with various degrees of accuracy was evaluated in terms of item parameter invariance as well as effects on test equating, with reference to performances of other estimation methods under various simulation conditions. The findings of this study suggest that using priors based on judgmental information may increase the accuracy of b-parameter estimation and test equating in a considerable amount, unless predictions about item p-values are extremely inaccurate. The effects were even more obvious for the 1PL model and for smaller sample sizes. In estimating a-parameters and overall equating results for larger sample sizes, mixed results were found for the superiority of using judgmental information as priors.
Abstract
For item banking or computerized adaptive testing to be successful, it is of vital importance to ensure the accuracy of item parameter estimation, especially when calibration needs to be conducted with the limited number of examinees for security reasons. This study investigated whether judgmental information about item difficulty would improve the accuracy of parameter estimation when used as prior information. Performance of using predictions of judges with various degrees of accuracy was evaluated in terms of item parameter invariance as well as effects on test equating, with reference to performances of other estimation methods under various simulation conditions. The findings of this study suggest that using priors based on judgmental information may increase the accuracy of b-parameter estimation and test equating in a considerable amount, unless predictions about item p-values are extremely inaccurate. The effects were even more obvious for the 1PL model and for smaller sample sizes. In estimating a-parameters and overall equating results for larger sample sizes, mixed results were found for the superiority of using judgmental information as priors.
참고문헌 (20)
- Bock, R. D, "Marginal maximum likelihood estimation of item parameters:Applications of EM algorithm", Psychometrika, 46, pp.443~459 (1981)
- Chang, S.-W, "A comparison of the standardization and IRT methods of adjusting pretest item statistics using realistic data", Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. (2001)
- Hambleton, R. K, "Item response theory: Principles and applications" (1985), Kluwer-Nijhoff
- "PIE: A computer program for IRT equating"[URL]
- Harwell, M, "An empirical study of the effects of small data sets and varying prior variances on item parameter estimation in BILOG", Applied Psychological Measurement, 15(3), pp.279~291 (1991)
- 김종필, "Empirical Study of Using Collateral Informationfor Calibrating Pretest Items with Small Sample Size", 교육평가연구, 20(2), pp.199~219 (2007), 한국교육평가학회
- Kolen, M. J, "Test equating, scaling, and linking: Methods and practices" (2004), Springer
- Lim, R. G, "Evaluation of two methods for estimating item response theory parameters when assessing differential item functioning", Journal of Applied Psychology, 75, pp.164~174 (1990)
- Lord, F. M, "Applications of item response theory to practical testing problems" (1980), Lawrence Erlbaum
- Mislevy, R. J, "Eploiting auxiliary information about examinees in the estimation of item parameters", Applied Psychological Measurement, 11, pp.81~91 (1987)
- Mislevy, R. J, "How to equate tests with little or no data", Journal of Educational Measurement, 30, pp.55~78 (1993)
- Reckase,M.D, "Unifactor latent trait models applied to multi-factor tests: Results and implications", Journal of Educational Statistics, 4, pp.207~230 (1979)
- Seong,T.J, "Sensitivity of marginal maximum likelihood estimation of item and ability parameters to the characteristics of the prior ability distributions", Applied Psychological Measurement, 14(3), pp.299~311 (1990)
- Skaggs, G, "A comparison of Pseudo-Bayesian and joint maximum likelihood procedures for estimating item parameters in the three-parameter IRT model", Applied Psychological Measurement, 13(4), pp.391~402 (1989)
- On the use of collateral item response information to improve pretest item calibration, "Stout, W", LSAC Report Series. Law School Admission Council, ~ (2003)
- Swaminathan, H, "Bayesian estimation in the Rasch model", Journal of Educational Statistics, 7, pp.175~191 (1982)
- Swaminathan, H, "Bayesian estimation in the three-parameter logistic model", Psychometrika, 51, pp.581~601 (1986)
- Swaminathan, H, "Small sample estimation in dichotomous item response models: effect of priors based on judgmental information on the accuracy of item parameter estimates", Applied Psychological Measurement, 27, pp.27~51 (2003)
- Wright, B. D, "Best Test Design" (1979), MESA Press
- Zimowski, M. F, "BILOG-MG: Multi-group IRT analysis and test maintenance for binary items" (1996), Scientific Software International