Exploring Test Takers’ Cognition in a High-stakes Reading Test: An Eye-tracking Study
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Learning how test takers derive their answers, referred to as the test-taking process, is essential to test validation. However, research on test takers’ mental process has been impeded by the excessive popularity of quantitative analysis and the limitations of research methods. By using eye-tracking technology, this study investigated Chinese ESL learners’ reading processes in the iBT TOEFL reading section for the purpose of test validation. Ninety Chinese ESL students at a large Midwestern university were invited to an eye-tracking lab and asked to take a 20-minute reading test on a computer screen to which an eye tracker was attached. As in Bax and Weir (2012), this study analyzed only high scorers’ eye movement data (e.g., gaze plots and fixation durations) based on the predetermined criteria. The reading time spent on the intended reading types was also calculated as evidence of validity. The results showed that participants predominantly exercised careful reading skills below the paragraph level, vocabulary questions seemingly failed to elicit inferencing ability, and expeditious reading rarely occurred. Learners did not spend more time reading a text associated with implicit questions compared to a text associated with explicit questions. The implication of the findings for test validity and test development are also discussed.
Abstract
Learning how test takers derive their answers, referred to as the test-taking process, is essential to test validation. However, research on test takers’ mental process has been impeded by the excessive popularity of quantitative analysis and the limitations of research methods. By using eye-tracking technology, this study investigated Chinese ESL learners’ reading processes in the iBT TOEFL reading section for the purpose of test validation. Ninety Chinese ESL students at a large Midwestern university were invited to an eye-tracking lab and asked to take a 20-minute reading test on a computer screen to which an eye tracker was attached. As in Bax and Weir (2012), this study analyzed only high scorers’ eye movement data (e.g., gaze plots and fixation durations) based on the predetermined criteria. The reading time spent on the intended reading types was also calculated as evidence of validity. The results showed that participants predominantly exercised careful reading skills below the paragraph level, vocabulary questions seemingly failed to elicit inferencing ability, and expeditious reading rarely occurred. Learners did not spend more time reading a text associated with implicit questions compared to a text associated with explicit questions. The implication of the findings for test validity and test development are also discussed.
참고문헌 (45)
- Alderson, J. C., "Assessing reading" (2000), Cambridge University Press
- Alderson, J. C., "Test review: Test of English as a Foreign Language™: Internet-based test (TOEFL iBT®)", Language Testing, 26(4), pp.621~631 (2009)
- Bax, S, "The cognitive processing of candidates during reading tests: Evidence from eye-tracking", Language Testing, 30(3), pp.1~25 (2013)
- Bax, S., "Investigating learners’ cognitive processes during a computer-based CAE Reading test", University of Cambridge ESOL Examinations Research Notes, 47, pp.3~14 (2012)
- Gail P. Baxter, "Investigating the Cognitive complerrity of Science Assessments", Educational Measurement: Issues and Practice, 17(3), pp.37~45 (1998), Wiley-Blackwell[DOI]
- Bernhardt, E. B., "Understanding advanced second-language reading" (2011), Routledge
- Representing language use in the university: Analysis of the TOEFL 2000 spoken and written academic language corpus. TOEFL Monograph Series, "Biber, D.", Eduational Testing Service, ~ (2004)
- Borsboom, D., "Measuring the mind: Conceptual issues in contemporary psychometrics" (2005), Cambridge University Press
- Brien, E. J. O., "Elaborative inferences during reading: Do they occur on-line?", Journal of Experimental Psychology, 14(3), pp.410~420 (1988)
- Carver, R. P., "Reading rate : Theory, research, and practical implications", Journal of Reading, 36(2), pp.84~95 (1992)
- Chapelle, C., "Building a validity argument for TOEFL", pp.2008, Routledge
- Andrew D. Cohen, "`I want to go back to the text': Response strategies on the reading subtest of the new TOEFL®", Language Testing, 24(2), pp.209~250 (2016), SAGE Publications[DOI]
- Educational Testing Service, "The official guide to the TOEFL test" (2009), McGraw-Hill
- Enright, M. K., "Reading for a reason: Using reader purpose to guide test design" (2000), Educational Testing Service
- Field, J., "Into the mind of the academic listener", Journal of English for Specific Purposes, 10(2), pp.102~112 (2011)
- Fraser, C. A., "Reading rate in L1 Mandarin Chinese and L2 English across five reading tasks", The Modern Language Journal, 91(3), pp.372~394 (2007)
- Fuchs, L. S., "Oral reading fluency as an indicator of reading competence : A theoretical, empirical, and historical analysis", Scientific Studies of Reading, 5(3), pp.239~256 (2001)
- Glaser, R., "The future of testing : A research agenda for cognitive psychology and psychometrics", American Psychologist, 36(9), pp.923~936 (1981)
- Grabe, W., "Fluency in reading : Thirty-five years later", Reading in a Foreign Language, 22(1), pp.71~83 (2010)
- Henning, G., "A guide to language testing: Development, evaluation, research" (1987), Newbury House
- Eun Hee Jeon, "L2 Reading Comprehension and Its Correlates: A Meta-Analysis", Language Learning, 64(1), pp.160~212 (2014), Wiley-Blackwell[DOI]
- Judith, R., "Student responses to reading strategies instruction", Foreign Language Annals, 28(2), pp.262~273 (1995)
- Juhasz, B. J., "The role of age of acquisition and word frequency in reading: Evidence from eye fixation durations", Visual Cognition, 13(7-8), pp.846~863 (2006)
- Just, M. A., "A theory of reading : From eye fixations to comprehension", Psychological Review, 87(4), pp.329~354 (1980)
- Khalifa, H., "Examining reading: Research and practice in assessing second language reading" (2009), Cambridge University Press
- Lemaire, B., "A cognitive computational model of eye movements investigating visual strategies on textual material", 33rd annual meeting of the Cognitive Science Socieity, pp.1146~1151 (2011)
- Matin, E., "Saccadic suppression : A review and an analysis", Psychological Bulletin, 81(12), pp.899~917 (1974)
- Messick, S., "Meaning and values in test validation : The science and ethics of assessment", Educational researcher, 18(2), pp.5~11 (1989)
- Paulson, E. J., "Does the degrees of reading power assessment reflect the reading process? An eye-movement examination", Journal of Adolescent & Adult Literacy, 46(3), pp.234~244 (2002)
- Rayner, K, "Eye movements in reading and information processing: 20 years of research", Psychological Bulletin, 124(3), pp.372~422 (1998)
- Rayner, K., "Eye movements as reflections of comprehension processes in reading", Scientific Studies of Reading, 10(3), pp.241~255 (2006)
- Rayner, K., "Psychology of reading" (2012), Psychology Press
- Reichle, E. D., "Toward a model of eye movement control in reading", Psychological Review, 105(1), pp.125~157 (1998)
- Reichle, E. D., "Eye movements during mindless reading", Psychological Science, 21(9), pp.1300~1310 (2010)
- Roever, C., "Effects of language of administration on a self-assessment of language skills" (2005), Educational Testing Service
- Rupp, A., "How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective", Language Testing, 23(4), pp.441~474 (2006)
- Yasuyo Sawaki, "Factor structure of the TOEFL Internet-based test", Language Testing, 26(1), pp.5~30 (2009), SAGE Publications[DOI]
- Solheim, O. J., "Eye-tracking as a tool in process-oriented reading test validation", International Electronic Journal of Elementary Education, 4(1), pp.153~168 (2011)
- Keith E. Stanovich, "The Interactive-Compensatory Model of Reading", Remedial and Special Education, 5(3), pp.11~19 (2016), SAGE Publications[DOI]
- Suvorov, R., "The use of eye tracking in research on video-based second language(L2)listening assessment : A comparison of context videos and content videos", Language Testing, 32(4), pp.463~483 (2015)
- Amos van Gelderen, "Development of adolescent reading comprehension in language 1 and language 2: A longitudinal analysis of constituent components.", Journal of Educational Psychology, 99(3), pp.477~491 (2007), American Psychological Association (APA)[DOI]
- Verhoeven, L. T., "Components in early second language reading and spelling", Scientific Studies of Reading, 4(4), pp.313~330 (2000)
- Weir, C. J., "Language testing and validation: An evidence-based approach" (2005), Palgrave-Macmillan
- Weir, C., "The cognitive processes underlying the academic reading construct as measured by IELTS", IELTS Collected Papers, 2, pp.212~269 (2006)
- The effects of testwiseness and test-taking anxiety on L2 listening test performance: A visual (eye-tracking) and attentional investigation, "Winke, P.", IELTS, 1~30, pp.IELTS Research Reports Online Series, 3 (2014)