The Comparison of Effects in 3PL Model and 3PL Testlet Model on the TOEIC
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Evaluating or measuring test takers’ language abilities is of central importance in the development of standardized language tests that foster meaningful assessment. The Test of English for International Communication(TOEIC) is a widely used standardized test to evaluate test takers’ receptive(reading and listen- ing skills) and productive(writing and speaking skills) language abilities. Althou- gh item response theory(IRT) models usually provide satisfactory answers for measuring parameters of items and test takers, they do not always lead to the correct solution when systematic biases are ignored in estimating items and test takers. Due to the paucity of validation studies of the TOEIC, the present study only seeks to explore psychometric properties of items in the simulated TOEIC test in the context of IRT and testlet response theory (TRT). In order to accomplish this purpose, this study describes how a set of items, sharing the same contents in the listening comprehension part of the TOEIC, be- haves under the two suggested models (3PL and 3PLT) with the test samples who enrolled in the TOEIC classes in the university in South Korea. Of 100 items on TOEIC listening comprehension session, 60 items (10 items for photographs, 30 items for question-response, 30 items for short conversion, and 30 items for short talks session, respectively) are clustered within each testlet which sometimes called item bundle. Results from this study may provide more precise, reliable, and valid information about estimates regarding items and examinees. Later, suggestions for future research are offered.
Abstract
Evaluating or measuring test takers’ language abilities is of central importance in the development of standardized language tests that foster meaningful assessment. The Test of English for International Communication(TOEIC) is a widely used standardized test to evaluate test takers’ receptive(reading and listen- ing skills) and productive(writing and speaking skills) language abilities. Althou- gh item response theory(IRT) models usually provide satisfactory answers for measuring parameters of items and test takers, they do not always lead to the correct solution when systematic biases are ignored in estimating items and test takers. Due to the paucity of validation studies of the TOEIC, the present study only seeks to explore psychometric properties of items in the simulated TOEIC test in the context of IRT and testlet response theory (TRT). In order to accomplish this purpose, this study describes how a set of items, sharing the same contents in the listening comprehension part of the TOEIC, be- haves under the two suggested models (3PL and 3PLT) with the test samples who enrolled in the TOEIC classes in the university in South Korea. Of 100 items on TOEIC listening comprehension session, 60 items (10 items for photographs, 30 items for question-response, 30 items for short conversion, and 30 items for short talks session, respectively) are clustered within each testlet which sometimes called item bundle. Results from this study may provide more precise, reliable, and valid information about estimates regarding items and examinees. Later, suggestions for future research are offered.
참고문헌 (36)
- Bachman, L. F., "Fundamental considerations in language testing" (1998), Oxford University Press
- Bachman, L. F., "Modern language testing at the turn of the century : assuring that what we count counts", Language Testing, 17(1), pp.1~42 (2000)
- Baker, F. B., "Item response theory : Parameter estimation techniques" (2004), Dekker
- Bell, R. B., "Conditional independence in a clustered item test", Applied Psychological Measurement, 12, pp.15~26 (1988)
- Scores on the TOEIC (Test of English for International Communication) test as a function of training time and type, "Bolt. R. F.", Educational Testing Service, ~, pp.TOEIC Research Report No. 3 (1998)
- Choi, I. C., "An application of item response theory to language testing: Theoretical studies in second language acquisition" (1992), Peter Lang
- Choi, I. C., "The impact of EFL testing on EFL education in Korea", Language Testing, 25(1), pp.39~62 (2008)
- Educational Testing Service, "TOEIC research summary" (1998), ETS
- Gilfert, S., "A review of TOEIC", The internet TESL Journal, 2(8) (1996)
- Hambleton, R. K., "Item response theory" (1985), Kluwer-Nijhoff
- In’nami, Y., "Factor structure of the revised TOEIC test : A multiple-sample analysis", Language Testing, 29(1), pp.131~152 (2011)
- Johnson, K., "Encyclopaedic dictionary of applied linguistics" (1998), Blackwell Publishers
- Keller, L. A., "Evaluating scoring procedures for context-dependent item sets", Applied Measurement in Education, 16(3), pp.207~222 (2003)
- Lord. F. M., "Applications of item response theory to practical testing problems" (1980), Lawrence Erlbaum
- Master, G. M., "A Rasch model for partial credit scoring", Psychometrika, 47(2), pp.149~174 (1982)
- Muraki, E., "A generalized partial credit model : Application of an EM algorithm", Applied Psychological Measurement, 16(2), pp.159~176 (1992)
- The redesigned TOEIC (Listening and Reading) test: Relations to test-taker perceptions of proficiency in English, "Powers, D. E.", Educational Testing Service, ~, pp.ETS Research Report No.ETS RR-09-56 (2008)
- Prapphal. K., "Issues and trends in language testing and assessment in Thailand", Language Testing, 25(1), pp.127~143 (2008)
- Samejima, F., "Estimation of latent ability using a response pattern of graded scores", Monograph Supplement, 17, pp.1~100 (1969)
- Background and goals of the Listening and Reading test redesign project, "Schedl, M.", Educational Testing Service, ~, pp.Compendium Study TC-10-02 (2010)
- Sireci, S. G., "On the reliability of testlet-based tests", Journal of Educational Measurement, 28(3), pp.237~247 (1991)
- Mapping English language proficiency test scores onto the Common European Framework, "Tannenbaum, R. J.", Educational Testing Service, ~, pp.ETS Research Report No.RR-05-18 (2005)
- Thissen, D., "Trace lines for testlets : A use of multiple-categorical-response models", Journal of Educational Measurement, 26(3), pp.247~260 (1989)
- van den Wollenberg, A. L., "Two new test statistics for the Rasch model", Psychometrika, 47(2), pp.123~140 (1982)
- Wainer, H., "Item clusters and computerized adaptive testing : A case for testlets", Journal of Educational Measurement, 24(2), pp.185~201 (1987)
- Wainer, H., "Using a new statistical model for testlets to score TOEFL", Journal of Educational Measurement, 37(3), pp.203~220 (2000)
- Wainer, H., "Testlet response theory: An analog for the 3PL model using in testlet-based adaptive testing, In Adaptive testing: Theory and pra", pp.245~269 (2000), Kluwer
- Wainer, H., "Testlet response theory and its applications" (2007), Cambridge University Press
- Wang, W. -C., "The Rasch testlet model", Applied psychological measurement, 29(2), pp.126~149 (2005)
- Enhancing the interpretation of a norm-referenced second-language test through criterion referencing: A research of experience in the TOEIC testing context, "Wilson, K. M.", Educational Testing Services, ~, pp.TOEIC Research Report No. 1 (1989)
- An explanatory dimensionality assessment of the test, "Wilson, K. M.", Educational Testing Services, ~, pp.TOEIC Research Report No.RR-00-14 (2000)
- Woodford, P., "The Test of English for International Communication (TOEIC), In English for international communication", pp.61~72 (1982), Pergamon Institute of English
- Yen, Y. M., "Effects of local item dependence on the fit and equating performance of the three-parameter logistic model", Applied Psychological Measurement, 8, pp.125~145 (1984)
- Yen, W. M., "Scaling performance assessment : Strategies for meaning local item dependence", Journal of Educational Measurement, 30(3), pp.187~213 (1993)
- Yen, W. M., "Item response theory, In Educational measurement", pp.111~154 (2006), American Council on Educational and Praeger Publishers
- Zhang, S., "Investigating the relative effects of persons, items, sections, and languages on TOEIC score dependability", Language Testing, 23(3), pp.351~369 (2006)