The Validity of Subscores for a Reading Comprehension Test: A Bifactor Approach
1
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
This study examined the validity of subscores for a reading comprehension section in a TOEIC practice test for which subscores are routinely provided for test takers. The study employed a bifactor approach, a constrained version of full-information item factor analysis, in its investigation of the subscore characteristics. The results showed a) all the subscores have lower reliability indices compared to the reliability of the total score and b) that the information of the subscores is trivial and redundant after accounting for their contribution to the general reading ability. These results agree with most of the previous studies on subscore quality and again demonstrate that language testing programs should be cautious about reporting subscores despite the pervasive demand for subscores in the testing industry.
Abstract
This study examined the validity of subscores for a reading comprehension section in a TOEIC practice test for which subscores are routinely provided for test takers. The study employed a bifactor approach, a constrained version of full-information item factor analysis, in its investigation of the subscore characteristics. The results showed a) all the subscores have lower reliability indices compared to the reliability of the total score and b) that the information of the subscores is trivial and redundant after accounting for their contribution to the general reading ability. These results agree with most of the previous studies on subscore quality and again demonstrate that language testing programs should be cautious about reporting subscores despite the pervasive demand for subscores in the testing industry.
참고문헌 (45)
- American Educational Research Associatin, "Standards for educational and psychological testing (1999)" (1999), American Educational Research Association
- Ackerman, T., "Using confirmatory MIRT modeling to provide diagnostic information in large scale assessment", Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego, CA (2009)
- Alderson, J. C., "Diagnosing foreign language proficiency: The interface between learning and assessment" (2005), Continuum
- Alderson, J. C., "Assessing reading" (2000), Cambridge UP
- Alderson, J. C., "Testing reading comprehension skills", Reading in a Foreign Language, 6(2), pp.425~438 (1990)
- Alderson, J. C., "Cognition and reading: Cognitive levels as embodied in test questions", Reading in a Foreign language, 5, pp.253~270 (1989)
- Betts, J., "Investigating early literacy and numeracy: Exploring the utility of the bifactor model", School Psychology Quarterly, 26(2), pp.97~107 (2011)
- Bock, R. D., "Full-information item factor analysis", Applied Psychological Measurement, 12, pp.261~280 (1988)
- Brown, H. D., "Language assessment: principles and classroom practices" (2004), Pearson Education, Inc
- Cai, L., (2010), SSI International
- Davis, F. B., "Research in comprehension in reading", Reading Research Quarterly, 3, pp.499~545 (1968)
- de la Torre, J., "A comparison of four methods of IRT subscoring", Applied Psychological Measurement, 35(4), pp.296~316 (2011)
- Demars, C. E., "Application of the Bi-Factor multidimensional item response theory model to testlet-based tests", Journal of Educational Measurement, 43(2), pp.145~168 (2006)
- du Toit, M., "IRT from SSI: BILOG-MG, MULTILOG, PARSCALE, TESTFACT" (2003), Scientific Software International
- Freedenthal, S., "Evaluation of the psychometric properties of the interpersonal needs questionnaire-12 in samples of men and women", Journal of Clinical Psychology, 67(6), pp.609~623 (2011)
- Gibbons, R. D., "Full-information item bi-factor analysis", Psychometrika, 57, pp.423~436 (1992)
- Harris, D. J., "Methods f examining the usefulness of subscores", Paper presented at the auuual meeting of the National Council on Measurement in Eucation. Chicago, IL (1991)
- When can subscores have value?, "Haberman, J. S", ETS, ~, pp.ETS RR-05-08 (2005)
- Haberman, J. S., "Reporting subscores for institutions", British Journal of Mathematical and Statistical Psychology, 62, pp.79~95 (2009)
- Haladyna, T. M., "The validity of subscores for a credentialing test", Evaluation and the Health Professions, 27(4), pp.349~368 (2004)
- Hambleton, R. K., "Item response theory: Principles and applications" (1985), Kluwer Nijhoff Publishing
- Hyland, K., "Second language writing" (1996), Cambridge UP
- Kintsch, W., "Toward a model of text comprehension and production", Psychological Review, 85, pp.363~394 (1978)
- Ling, G., "Why the major field(business) test does not report subscores of individual test-takers - reliability and construct validity evidence", Paper presented at the annual meeting of American Educational Research Association and the National Council on Measurement in Education (2009)
- Lord, F. M., "Statistical theories of mental test scores" (1968), Addison-Welsley Publishing Company
- Lumley, T., "The notion of subskills in reading comprehension tests: an EAP example", Language Testing, 10, pp.211~234 (1993)
- McLeod, L. D., "Factor Analysis for Items Scored in Two Categories, In Test Scoring", pp.189~216 (2001), Erlbaum
- "Mead, R. (1982). Review of Mumby, J. (1978). Communicative syllabus design. Applied Linguistics, 3, 70-78", ~
- The facts about subscores, "Monaghan, W.", ETS, ~, pp.ETS RDC-04 (2006)
- EFL reading and writing, "Teaching ESL", pp.2009, Nation, I. S. P.
- Comparison of subscores based on classical test theory methods, "Puhan, G.", ETS, ~, pp.ETS RR-08-04 (2008)
- Reise, S. P., "The role of the bifactor model in resolving dimensionality issues in health outcomes measures", Quality of Life Research, 16, pp.19~31 (2007)
- When can subscores be expected to have added value? results from operational and simulated data, "Sinharay, S.", ETS, ~, pp.ETS RR-10-16 (2010)
- Reporting subscores: a survey, "Sinharay, S.", ETS, ~, pp.ETS RM-08-18 (2008)
- Sinharay, S., "Subscores based on classical test theory: to report of not to report", Educational Measurement: Issues and Practice, 26(4), pp.21~28 (2007)
- Sinharay, S., "Reporting diagnostic scores in educational testing: temtations, pitfalls, and some solutions", Multivariate Behavioral Research, 45, pp.553~573 (2010)
- Skehan, P., "Issues in the testing of English for specific purposes", Language Testing, 1, pp.202~220 (1984)
- Stone, C. A., "Providing subscale scores for diagnostic information: a case study when the test is essentially unidimensional", Applied Measurement in Education, 23, pp.63~86 (2010)
- Stout, W., "A nonparametric approach for assessing latent trait unidimensionality", Psychometrika, 52, pp.589~617 (1987)
- Tate, R., "Implications of multidimensionlity for total score and subscore performance", Applied Measurement in Education, 17(2), pp.89~112 (2004)
- von Davier, M., "A general diagnostic model applied to language testing data", British Journal of Mathematical and Statistical Psychology, 61, pp.287~307 (2008)
- Wainer, H., "Augmented scores-“Borrowing strength” to compute scores based on small numbers of items, In Test Scoring", pp.343~387 (2001), Lawrence Erlbaum
- Weigle, S. C., "Assessing writing" (2002), Cambridge UP
- Wood, R., (2003), Scientific Software International, Inc
- Zhang, J., "The theoretical DETECT index of dimensionality and its applicationto approximate simple structure", Psychometrika, 64, pp.213~249 (1999)
