Test Fairness: DIF beyond Group Mean Differences
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The purpose of this study is to investigate whether an EFL test was associated with DIF across gender and to locate the source of DIF in question types and passage content. Findings showed that even though females obtained significantly higher scores than males in 48 items out of 50 in the group mean comparison, 15 items with six items in favor of males and nine items in favor of females showed DIF after matching ability levels. In the source of DIF, males had higher probabilities of getting items about inferential questions correct in listening comprehension, whereas females had higher probabilities of getting items about literal understanding and applications (indirect assessment of speaking ability) correct. Interestingly, the reverse held true in reading comprehension. That is, questions regarding literal understanding and application (indirect assessment of writing ability) differed in their functioning in favor of males, whereas questions regarding inferences, main ideas, and grammar differed in their functioning in favor of females. In passage content, more female-favoring contents than male-favoring contents in quality as well as in quantity were found, probably leading to the poor performance by males in the reading test. These findings imply that item developers take every effort to take a rational choice between question types and passage content in order to develop a fair test across subgroups such as gender.
Abstract
The purpose of this study is to investigate whether an EFL test was associated with DIF across gender and to locate the source of DIF in question types and passage content. Findings showed that even though females obtained significantly higher scores than males in 48 items out of 50 in the group mean comparison, 15 items with six items in favor of males and nine items in favor of females showed DIF after matching ability levels. In the source of DIF, males had higher probabilities of getting items about inferential questions correct in listening comprehension, whereas females had higher probabilities of getting items about literal understanding and applications (indirect assessment of speaking ability) correct. Interestingly, the reverse held true in reading comprehension. That is, questions regarding literal understanding and application (indirect assessment of writing ability) differed in their functioning in favor of males, whereas questions regarding inferences, main ideas, and grammar differed in their functioning in favor of females. In passage content, more female-favoring contents than male-favoring contents in quality as well as in quantity were found, probably leading to the poor performance by males in the reading test. These findings imply that item developers take every effort to take a rational choice between question types and passage content in order to develop a fair test across subgroups such as gender.
참고문헌 (32)
- Bernhardt,E, "Reading development in a second language" (1991), Ablex Publishing Corporation
- Brantmeier,C, "Beyond linguistic knowledge:Individual differences in second language reading", Foreign Language Annals, 36, pp.33~43 (2003)
- Characteristics associated with differential item functioning on the scholastic aptitude test: Gender and majority, "minority group comparisons", Carlton,S, 1992, Research Report No. 92: 60-70
- Chen,H-C, "Effects of previewing and providing background knowledge on Taiwanese college students’ comprehension of American short stories", TESOL Quarterly, 29, pp.663~686 (1995)
- Chiang,C, "The effect of speech modification,prior knowledge,and listening proficiency on EFL lecture learning", TESOL Quarterly, 26, pp.345~374 (1992)
- Clauser,B.E, "Using statistical procedures to identify differentially functioning test items", Educational Measurement: Issues and Practice, 17, pp.31~47 (1998)
- Dorans,N, "DIF detection and description:Mantel-Haenszel and standardization.In Differential item functioning", pp.35~66 (1993), Lawrence Erlbaum
- "Test and score data summary for TOEFL internet-based test.Retrieved December 20,2007"[URL]
- Elder,C, "What does test bias have to do with fairness?", Language Testing, 14, pp.261~277 (1997)
- Ellis,B, "Test and item bias:What they are,what they aren’t,and how to detect them", Eric Document Reproduction Service No. ED 480042, ~ (2003)
- Fincher-Kiefer,R, "The role of prior knowledge in inferential processing", Journal of Research in Reading, 15, pp.12~27 (1992)
- Gafni,N, "Differential item functioning:Performance by sex on reading comprehension tests", ERIC Document Reproduction Service No. ED 331844, ~ (1991)
- Hyde,J, "Gender differences in verbal ability:A meta-analysis", Psychological Bulletin, 104, pp.53~69 (1988)
- Jin, Kyung-Ae, "Factors Related to the Difficulty of College Scholastic Ability Test English Items", 영어교육, 60(3), pp.17~25 (2005), 한국영어교육학회
- Kunnan,A.J, "DIF in native language and gender groups in an ESL placement test", TESOL Quarterly, 24, pp.741~746 (1990)
- Lund,R, "A comparison of second language listening and reading comprehension", Modern Language Journal, 75, pp.196~204 (1991)
- Maller,S, "Best practices in detecting bias in nonverbal tests.In Handbook of nonverbal assessment", pp.2003, Kluwer Academic
- Mantel,N, "Statistical aspects of the analysis of data from retrospective studies of disease", Journal of the National Cancer Institute, 22, pp.719~748 (1959)
- Messick,S, "Validity.In Educational measurement (3rd ed.)", pp.13~103 (1989), American Council on Education & Macmillan
- Pae,T.-I, "Gender effect on reading comprehension with Korean EFL learners", System, 32, pp.265~281 (2004)
- Pae,T.-I, "DIF for examinees with different academic backgrounds", Language Testing, 21, pp.53~73 (2004)
- Pae,T.-I, "Examining the relationship between differential item and differential test functioning", Language Testing, 23, pp.475~496 (2006)
- Park,G.-P, "Comparison of L2 listening and reading comprehension by university students learning English in Korea", Foreign Language Annals, 37, pp.448~458 (2004)
- Park,G.-P, "Differential item functioning on an English listening test across gender", TESOL Quarterly, 42, pp.115~123 (2008)
- Ryan,K, "Differential item functioning on two tests of EFL proficiency", Language Testing, 9, pp.12~29 (1992)
- Shohamy,E, "Validation of listening comprehension tests:The effect of text and question type", Language Testing, 8, pp.23~40 (1991)
- Stanovich,K, "Toward an interactive-compensatory model of individual differences in the development of reading fluency", Reading Research Quarterly, 16, pp.32~71 (1980)
- Swaminathan,H, "Detecting differential item functioning using logistic regression procedures", Journal of Educational Measurement, 27, pp.361~370 (1990)
- Takala,S, "Test fairness:a DIF analysis of an L2 vocabulary test", Language Testing, 17, pp.323~340 (2000)
- "TOEIC technical manual.Retrieved December 19,2008"[URL]
- Thissen,D, "Beyond group mean differences:The concept of item bias", Psychological Bulletin, 99, pp.118~128 (1986)
- Uiterwijk,H, "Linguistic sources of item bias for second generation immigrants in Dutch tests", Language Testing, 22, pp.211~234 (2005)