Predictive Processing of Multimodal Information by EFL Learners
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The present study investigates how multimodal discourses are processed in real-time in a variety of communication situations by conducting eye-tracking experiments with Korean EFL learners. It examines the interactions between linguistic and non-linguistic information and observes the patterns of incremental processing that emerge over the course of the interactions, exploring how they are related to the language proficiency levels of the participants in the experiment. Through the visual-world paradigm (VWP), which creates contexts for multimodal information, participants in the experiment were given both auditory inputs via questions (e.g., “What color is the upper left circle?”) and visual stimuli in the form of figures with various color features in different positions as a means to examine the participants’ reactions in real-time based on the timing and proportion of fixation movement. Participants with advanced language proficiency actually showed different fixation movements, even on the lexical level, according to the position of adjectives within a nominal phrase (e.g., the upper left circle vs. the left upper circle) at varying speeds of predictive comprehension of the figures’ position based on the context of visual images (e.g., 20 figures vs. ten figures). In contrast, participants with intermediate language proficiency did not show significantly different reactions to the conditions of utterance and image. This result shows that L2 speakers with different language proficiency levels implement distinct strategies for input processing, and that advanced speakers are capable of predictive comprehension of ambiguous contexts when non-linguistic information, such as visual images, is available.
Abstract
The present study investigates how multimodal discourses are processed in real-time in a variety of communication situations by conducting eye-tracking experiments with Korean EFL learners. It examines the interactions between linguistic and non-linguistic information and observes the patterns of incremental processing that emerge over the course of the interactions, exploring how they are related to the language proficiency levels of the participants in the experiment. Through the visual-world paradigm (VWP), which creates contexts for multimodal information, participants in the experiment were given both auditory inputs via questions (e.g., “What color is the upper left circle?”) and visual stimuli in the form of figures with various color features in different positions as a means to examine the participants’ reactions in real-time based on the timing and proportion of fixation movement. Participants with advanced language proficiency actually showed different fixation movements, even on the lexical level, according to the position of adjectives within a nominal phrase (e.g., the upper left circle vs. the left upper circle) at varying speeds of predictive comprehension of the figures’ position based on the context of visual images (e.g., 20 figures vs. ten figures). In contrast, participants with intermediate language proficiency did not show significantly different reactions to the conditions of utterance and image. This result shows that L2 speakers with different language proficiency levels implement distinct strategies for input processing, and that advanced speakers are capable of predictive comprehension of ambiguous contexts when non-linguistic information, such as visual images, is available.
참고문헌 (49)
- Abbott, J. T., "Focal colors across languages are representative members of color categories", Proceedings of the National Academy of Sciences, 113, pp.11178~11183 (2016)
- Altmann, G. T. M., "Interaction with context during human sentence processing", Cognition, 30(3), pp.191~238 (1988)
- Arnold, J. E., "The effects of thematic roles on pronoun use and frequency of reference", Discourse Processes, 31(2), pp.137~162 (2001)
- Arnold, J. E., "The immediate use of gender information: eyetracking evidence of the time-course of pronoun resolution", Cognition, 76, pp.B13~B26 (2000)
- Clahsen, H., "Grammatical processing in language lerners", Applied Psycholinguistics, 27(1), pp.3~42 (2006)
- Cook, M., "The development of comprehension in interlanguage pragmatics: The case of request strategies in English", Australian Review of Applied Linguistics, 25, pp.19~39 (2002)
- Cooper, R. M., "The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing", Cognitive Psychology, 6(1), pp.84~107 (1974)
- Cunnings, I., "Anaphora resolution and reanalysis during L2 sentence processing: evidence from the visual world paradigm", Studies in Second Language Acquisition, 39(4), pp.621652~ (2017)
- de Groot, F., "Language-induced visual and semantic biases in visual search are subject to task requirements", Visual Cognition, 25(1-3), pp.225~240 (2017)
- Dekydtspotter, L., "The comparative fallacy in L2 processing research", Proceedings of the 8th Generative Approaches to Second Language Acquisition Conference (GASLA 2006), pp.33~40 (2006), Cascadilla Press
- Felser, C., "The processing of ambiguous sentences by first and second language learners of English", Applied Psycholinguistics, 24, pp.453~489 (2003)
- Frazier, L., "Attention and performance XII", pp.559~586 (1987), Lawrence Erlbaum Associates
- Frazier, L., "The sausage machine: A new two-stage parsing model", Cognition, 6(4), pp.291~325 (1978)
- Frazier, L., "Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences", Cognitive Psychology, 14(2), pp.178~210 (1982)
- Harley, B., "Language aptitude and second language proficiency in classroom learners of different darting ages", Studies in Second Language Acquisition, 19, pp.379~400 (2002)
- Havik, E., "Processing subject-object ambiguities in the L2: A self-paced reading study with German L2 learners of Dutch", Language Learning, 59, pp.73~112 (2009)
- Hemforth, B., "Cognitive parsing in german: An introduction", German sentence processing, pp.1~23 (2000), Kluwer Academic Press
- Hemforth, B., "German Sentence Processing" (2000), Kluwer
- Hopp, H., "Ultimate attainment in L2 inflectional morphology: Performance similarities between non-native and native speakers", Lingua, 120, pp.901~931 (2010)
- Huettig, F., "Using the visual world paradigm to study language processing: A review and critical evaluation", Acta Psychologica, 137(2), pp.151~171 (2011)
- Jackson, C. N., "The effects of L2 proficiency level on the processing of wh-questions among Dutch second language speakers of English", International Review of Applied Linguistics in Language Teaching, 49(3), pp.195~219 (2011)
- Juffs, A., "Psycholinguistically oriented second language research", Annual Review of Applied Linguistics, 21, pp.207~220 (2001)
- Just, M. A., "A theory of reading: From eye fixations to comprehension", Psychological Review, 87(4), pp.329~354 (1980)
- Kaiser, E., "Effects of Anaphoric Dependencies and Semantic Representations on Pronoun Interpretation", Anaphora Processing and Applications, pp.121~130 (2009), Springer
- Kang, J. Y., "Referencing in a Second Language: Korean EFL Learners'Cohesive Use of References in Written Narrative Discourse", Discourse Processes, 46, pp.439~466 (2009)
- Koch, E., "Morphosyntax in predictive L2 sentence processing: A Visual World eye-tracking study of German verb morphology", Second Language Research Forum 2019 (2019)
- Kreysa, H., "Distinct effects of different visual cues on sentence comprehension and later recall: The case of speaker gaze versus depicted actions", Acta Psychologica, 188, pp.220~229 (2018)
- MacDonald, M. C., "The lexical nature of syntactic ambiguity resolution", Psychological Review, 101, pp.483~506 (1994)
- Mady, C., "The suitability of core French for recently arrived adolescent immigrants to Canada", Canadian Journal of Applied Linguistics, 10(2), pp.177~196 (2007)
- Marinis, T., "Gaps in second language sentence processing", Studies in Second Language Acquisition, 27, pp.53~78 (2005)
- Marslen-Wilson, W., "The limited compatibility of linguistic and perceptual explanation", Proceedings of Chicago Linguistic Society 11, Papers from the Parasession on Functionalism, pp.409~420 (1975), CLS
- Mayer, R. E., "The Cambridge handbook of multimedia learning", pp.31~48 (2005), Cambridge University Press
- MURPHY ODO DENNIS, "The Effects of Dual-language L2 Texts on English Reading Comprehension and Text-type Preferences", 외국어교육연구, 33(4), pp.449~475 (2019), 외국어교육연구소[DOI]
- Papadopoulou, D., "Parsing strategies in L1 and L2sentence processing: A study of relative clause attachment in Greek", Studies in Second Language Acquisition, 25, pp.501~528 (2003)
- Phillips, C., "ERP effects of the processing of syntactic long-distance dependencies", Cognitive Brain Research, 22, pp.407~428 (2005)
- Pozzan, L., "Second language processing and revision of garden-path sentences: avisual word study", Bilingualism : Language and Cognition, 1(3), pp.1~8 (2016)
- Pulvermüller, F., "Understanding in an instant : neurophysiological evidence for mechanistic language circuits in the brain", Brain and Language, 110, pp.81~94 (2009)
- Rickheit, R., "Psycholinguistik" (2004), Stauffenburg Verlag
- Roberts, L., "Using eye-tracking to investigate topics in L2 acquisition and L2processing", Studies in Second Language Acquisition, 35(2), pp.213~235 (2013)
- Rodrigo, L., "The Interplay of Relational and Non-relational Processes in Sentence Production: The Case of Relative Clause Planning in Japanese and Spanish", Frontiers in Psychology, 9, pp.1~15 (2018)
- Saryazdi, R., "Mapping language to visual referents:Does the degree of image realism matter?", Acta Psychologica, 182, pp.91~99 (2018)
- 신혜원, "The Development of EFL Learners' Grammatical Accuracy with Repeated Written Corrective Feedback", 외국어교육연구, 32(1), pp.291~312 (2018), 외국어교육연구소[DOI]
- Sichelschmidt, L., "Der Augenblick des Verstehens: Motorische Indikatoren unmittelbarer semantischer Verarbeitung", ZiF-Mitteilungen, 4 (1995), Universität Bielefeld
- Taguchi, N., "Pragmatic performance in comprehension and production of English as a second language", 박사, pp.2003
- Tanenhaus, M. K., "Integration of visual and linguistic information in spoken language comprehension", Science, 268, pp.1632~1634 (1995)
- Traxler, M. J., "Introduction to psycholinguistics: understanding language science" (2012), Wiley-Blackwell
- Trueswell, J. C., "The kindergarten-path effect: studying on-line sentence processing in young children", Cognition, 73, pp.89~134 (1999)
- von Studnitz, R. E., "Interlingual homograph interference in German-English bilinguals: Its modulation and locus of control", Bilingualism : Language and Cognition, 5(1), pp.1~23 (2002)
- Yoo, D. G., "Using Contextual Information In Leaners’ Spoken Language Communication: An Eye-Tracking Study", GEMA Online Journal of Language Studies, 13, pp.5~20 (2013)