한국어 다면자료 구축 연구
Construction of a Korean multimodal corpus
0
조회
0
다운로드
9
인용
0
참고문헌
키워드
초록
People use various gestures while thy speak and listen ; they smile, nod,shake hands, look somewhere and change positions. However humans also communicate without gestures: they precisely understand written texts and dialogues on the radio or on the phone without any forms of gestures at all. This study attempts to respond to a question whether gestures are essential components of linguistic behaviour or they are just supplemental components that can be safely disregarded in verbal communication. We also examine a more aggressive assertions that gestures are critical part of human communication and should be interpreted with verbal language in an holistic and integrated perspective as proposed by Bavelas et al. (2000),Clark (1996) and McNeill (1992, 2005). To tackle the research questions, we built a Korean multimodal corpus composed of audio-visual recordings of verbal communication with rich multi-level linguistic annotations. Using this corpus, we compared verbal backchannels with and without gestures. Through an analysis of these different forms of verbal communications, we explore how gestures influence the interpretation of verbal backchannels. (Yonsei University, Institute of Language and Information Studies)
Abstract
People use various gestures while thy speak and listen ; they smile, nod,shake hands, look somewhere and change positions. However humans also communicate without gestures: they precisely understand written texts and dialogues on the radio or on the phone without any forms of gestures at all. This study attempts to respond to a question whether gestures are essential components of linguistic behaviour or they are just supplemental components that can be safely disregarded in verbal communication. We also examine a more aggressive assertions that gestures are critical part of human communication and should be interpreted with verbal language in an holistic and integrated perspective as proposed by Bavelas et al. (2000),Clark (1996) and McNeill (1992, 2005). To tackle the research questions, we built a Korean multimodal corpus composed of audio-visual recordings of verbal communication with rich multi-level linguistic annotations. Using this corpus, we compared verbal backchannels with and without gestures. Through an analysis of these different forms of verbal communications, we explore how gestures influence the interpretation of verbal backchannels. (Yonsei University, Institute of Language and Information Studies)
참고문헌 (38)
- Allwood, Jens, "Multimodal corpora, In Corpus Linguistics" (2008), Walter de Gruyter
- "The MUMIN Multimodal coding Scheme" (2005)[URL]
- Allwood, Jens., "The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena", Lang Resources & Evaluation, 41, pp.273~287 (2007)
- Allwood, Jens., "The analysis of embodied communicative feedback in multimodal corpora: a prerequisite for behavior simulation", Lang Resources & Evaluation, 41, pp.255~272 (2007)
- Bavelas, Janet., "Visible acts of meaning : An integrated message model of language in face-to-face dialogue", Journal of Language and Social Psychology, 19(2), pp.163~194 (2000)
- Bavelas, Janet., "Nonverbal and Verbal Communication-Hand Gestures and Facial Displays as Part of Language Use inFace-to-Face Dialogue, In The Sage Handbook of nonverbal communication", pp.97~115 (2006), Sagepublidations Inc
- Bavelas, Janet., "Listener responses as a collaborative process: the role of gaze", Journal of Communication, 52, pp.225~236 (2002)
- Bertrand, Roxane, "Le CID-Corpus of Interactional Data", Traitement Automatique des Langues, 49(3), pp.105~134 (2008)
- Bertrand, Roxane., "Représentation, édition et exploitation de données multimodales: le cas des backchannels du corpus CID", Cahiers de Linguistique, 33(2), pp.183~212 (2009)
- Clark, Herbert, "Using language" (1996), Cambridge University Press
- Clark, Herbert., "Speaking while monitoring addressees for understanding", Journal of Memory and Language, 50, pp.62~81 (2004)
- Darwin, Charles., "The Expression of the Emotions in Man and Animals" (1998), Oxford University Press
- Engle, Randi, "Not Channels But Composite Signals", Proceedings of the Twentieth Annual Conference of the Cognitive Science, pp.321~326 (1998)
- Ekman, Paul., "Measuring Facial Movement", Environmental Psychology and Nonverbal Behavior, 1(1), pp.56~75 (1976)
- Ekman, Paul., "What the face reveals" (2005), Oxford University Press
- Ekman, Paul., "Facial affect scoring technique : A forst validity study", Semiotica, 3, pp.37~58 (1971)
- Ferré, Gaëlle, "Timing Relationships between Speech and Co-Verbal Gestures in Spontaneous French", Multimodal Corpora Proceedings, pp.86~91 (2010)
- Foster, Mary., "Corpus-based generation of head and eyebrow motion for an embodied conversation agent", Language Resources & Evaluation, 41, pp.305~323 (2007)
- Hadar, Uri., "Head Movements During Listening, Turns in Conversation", Journal of Nonverbal Behavior, 9(4), pp.214~228 (1985)
- Heylen, Dirk, "Listening Heads. Modeling Communication, Lecture Notes in Artificial Intelligence 4930", pp.241~259 (2008), Springer-Verlag
- Heylen, Dirk., "Searching for Prototypical Facial Feedback Signals, In IVA2007, Lecture Notes in Artificial Intelligence 4722", pp.147~153 (2007), Springer-Verlag
- Kipp, Michael., "An annotation scheme for conversational gestures: how economically capture timing and form", Language and Resources and Evaluation, 41, pp.1367~1370 (2007)
- Knight, Dawn, "A Multi-modal corpus approach to the analysis of backchanneling behaviour", pp.2009
- Knight, Dawn., "HeadTalk, HandTalk and the corpus: towards a framework for multi-modal, multi-media corpus develment", Corpus, 4(1), pp.1~32 (2011)
- Lund, Kristine, "Importance of gaze and gesture in interactive multimodal explanation", Lang Resources & Evaluation, 41, pp.289~303 (2007)
- Martell, Craig, "FORM: an Extensible, Kinematically-Based Gesture Annotation Scheme", 7th International Conference on Spoken Language Processing(ICSLP-2002), pp.353~356 (2002)
- Maynard, Senko, "Japanese Conversation: Self-Contextualization through Structure and Interactional Management" (1989), Ablex Publishing
- Mclave, Evelyn, "Linguistic functions of head movements in the context of speech", Journal of pragmatics, 32, pp.855~878 (2000)
- McGregor, Graham., "Reception and Response" (1990), Rougledge
- McNeill, David, "Hand and Mind" (1992), The University of Chicago Press
- McNeill, David, "Gesture and Thought" (2005), The University of Chicago Press
- McNeill, David., "Communicative effective effets of speech-mismatched gestures", Research on Language and Social Interaction, 27, pp.223~237 (1994)
- Meillon, Brigitte., "DYNEMO: A Corpus of dynamic and spontaneous emotional facial expressions", Multimodal Corpora Proceedings, pp.31~36 (2010)
- Pitch, Kaola., "Linking Conversation Analysis and Motion Capturing: How to rebustly track multiple participants?", Multimodal Corpora Proceedings, pp.63~69 (2010)
- Pizzuto, Elena, "Gestures and Language Acquisition: Key Issues Concernig Data Interpretation, Selection, Notation", Speech and Face to Face Communication Workshop in memory of Christian Benoît, pp.62~64 (2008)
- Richmond, Virginia., "Nonverbal Behavior in Interpersonal Relations" (2008), Pearson Education Inc
- Tan, Ning., "Multi-level Annotations of Nonverbal Behaviors in French Spotaneous Conversations", Multimodal Corpora Proceedings, pp.74~79 (2010)
- Wittenburg, Peter, "Preprocessing multimodal corpora, In Corpus Linguistics", pp.664~684 (2008), Walter de Gruyter