본문 바로가기
KERD
논문KCI 등재

언어 사용에서 다중감각 신호의 역할 탐구: ELAN 을 활 용한 비디오 자료 분석의 예

Investigating the role of multimodal cues in language use: A tutorial on video data analysis using ELAN

정종민|김수한|고언숙
한국음운론학회 (The Phonology-Morphology Circle of Korea)|2024.04|음성음운형태론연구 (Studies in Phonetics, Phonology, and Morphology)|제30권 제1호|pp. 3-29|DOI 10.17959/sppm.2024.30.1.3

0

조회

0

다운로드

0

인용

0

참고문헌

키워드

communicationvideoELANMultimodal signals
주제분류언어학

초록

Multimodal signals, including audio, visual, and tactile cues, play a crucial role in speech recognition and language learning, with even infants showing sensitivity to their effects. In this tutorial, we explore various methodologies for recording human interactions auditorily and visually to capture these multimodal signals. We introduce open databases that can be used even when multimodal data collection is challenging for individual researchers. Once the dataset is prepared, researchers can utilize software to code target signals accompanying speech units. We focus on ELAN, a free program that enables the alignment of multiple tiers of annotation to code simultaneous multimodal cues. Real-world examples of coding paradigms and analyses are provided. We discuss the potential of how knowledge of multimodal signals interacting with speech can be extended to improve mechanisms for machine learning to efficiently mimic human learning.

Abstract

Multimodal signals, including audio, visual, and tactile cues, play a crucial role in speech recognition and language learning, with even infants showing sensitivity to their effects. In this tutorial, we explore various methodologies for recording human interactions auditorily and visually to capture these multimodal signals. We introduce open databases that can be used even when multimodal data collection is challenging for individual researchers. Once the dataset is prepared, researchers can utilize software to code target signals accompanying speech units. We focus on ELAN, a free program that enables the alignment of multiple tiers of annotation to code simultaneous multimodal cues. Real-world examples of coding paradigms and analyses are provided. We discuss the potential of how knowledge of multimodal signals interacting with speech can be extended to improve mechanisms for machine learning to efficiently mimic human learning.

참고문헌 (48)

  1. 강미영, "한국수어 말뭉치 구축 모형 연구", 건지인문학, 16, pp.5~27 (2016)
  2. 윤석민, "한국수어(KSL)말뭉치 연구 및 구축의 현황과 전망－2018년 사업을 중심으로－", 국어국문학, 189, pp.93~119 (2019)
  3. "Is maternal touch used referentially? (Publication No. 466)" (2014)[URL]
  4. ABU-ZHAYA, RANA., "Multimodal infant-directed communication : How caregivers combine tactile and linguistic cues", Journal of Child Language, 44(5), pp.1088~1116 (2017)[DOI]
  5. ASLIN, RICHARD N., "How infants view natural scenes gathered from a head-mounted camera", Optometry and Vision Science, 86(6), pp.561~565 (2009)[DOI]
  6. BATES, DOUGLAS, "Package lme4", Journal of Statistical Software, 67(1), pp.1~91 (2015)
  7. BERGELSON, ELIKA, "Day by day, hour by hour: Naturalistic language input to infants", Developmental Science, 115(3), pp.e12715~ (2018)[DOI]
  8. ". Version 6.3" (2022)[URL]
  9. BRUGMAN, HENNIE, "Annotating Multimedia", Multi-modal resources with ELAN, 2004, ~
  10. CASEY, KENNEDY, "Moving beyond “nouns in the lab”: Using naturalistic data to understand why infants’ first words include uh-oh and hi", Developmental Psychology, 59(11), pp.2162~2173 (2021)[DOI]
  11. CHEN-HSIN, CHEN, "Parent-child joint behaviors in novel object play create high-quality data for word learning", Child Development (2021)[DOI]
  12. CUSTODE, STEPHANIE A., "Cracking the code: Social and contextual cues to language input in the home environment", Infancy, 25(6), pp.809~826 (2020)[DOI]
  13. , 2014
  14. (2022)[URL]
  15. FLEISS, JOSEPH L., "Measuring nominal scale agreement among many raters", Psychological Bulletin, 76(5), pp.378~382 (1971)[DOI]
  16. FUSARO, MARIA, "Beside the point : Mothers and head nodding and shaking gestures during parent-child play", Infant Behavior and Development, 37(2), pp.235~247 (2014)[DOI]
  17. GICK, BRYAN, "Aero-tactile integration in speech perception", Nature, 462, pp.502~504 (2009)[DOI]
  18. GILMORE, RICK O, "Curating identifiable data for sharing: The Databrary Project", New York Scientific Data Summit (NYSDS), pp.1~6 (2016)[DOI]
  19. GOGATE, LAKSHMI J, "A study of multimodal motherese : The role of temporal synchrony between verbal labels and gestures", Child Development, 71(4), pp.878~894 (2000)[DOI]
  20. GORMAN, KYLE, "Prosodylab-aligner : A tool for forced alignment of laboratory speech", Canadian Acoustics, 39, pp.192~ (2011)
  21. HOLLER, JUDITH, "Multimodal language processing in human communication", Trends in Cognitive Sciences, 23(8), pp.639~652 (2019)[DOI]
  22. KO, EON-SUK, "Mothers’ use of touch across infants’ development and its implications for word learning : Evidence from Korean dyadic interactions", Infancy, 28(3), pp.597~618 (2023)[DOI]
  23. KUZNETSOVA, ALEXANDRA, "lmerTest package : Tests in linear mixed effects models", Journal of Statistical Software, 82(13), pp.1~26 (2017)[DOI]
  24. LANDIS, RICHARD J., "The measurement of observer agreement for categorical data", Biometrics, 33, pp.159~ (1977)[DOI]
  25. LAUSBERG, HEDDA, "Coding gestural behavior with the NEUROGES-ELAN system", Behavior Research Methods, 41(3), pp.841~849 (2009)[DOI]
  26. LEWKOWICZ, DAVID J., "Multisensory Perception", 89~112 (2020)
  27. LEWKOWICZ, DAVID J., "Recognition of amodal language identity emerges in infancy", International Journal of Behavioral Development, 37(2), pp.90~94 (2013)[DOI]
  28. LEWKOWICZ, DAVID J., "Cross-modal equivalence in early infancy: Auditory-visual intensity matching", Developmental Psychology, 16(6), pp.597~607 (1980)[DOI]
  29. LEW-WILLIAMS, CASEY, "Social touch interacts with infants & learning of auditory patterns", Developmental Cognitive Neuroscience, 35, pp.66~74 (2019)[DOI]
  30. LONG, BRIA, "The BabyView Camera: Designing a new head-mounted camera to capture children’s early social and visual environments", Behavior Research (2023)[DOI]
  31. MACWHINNEY, BRIAN, "The CHILDES Project: Tools for Analyzing Talk" (2000), Lawrence Erlbaum Associates
  32. MACWHINNEY, BRIAN, "The CHILDES Project: Tools for Analyzing Talk, Volume II: The Database" (2014), Psychology Press
  33. MCAULIFFE, MICHAEL, "Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi", Interspeech 2017, pp.498~502 (2017)
  34. MCGURK, HARR, "Hearing lips and seeing voices", Nature, 264(5588), pp.746~748 (1976)[DOI]
  35. NORTHRUP, JESSIE B, "Multimodal coordination of vocal and gaze behavior in mother-infant dyads across the first year of life", Infancy, 25(6), pp.952~972 (2020)[DOI]
  36. POVEY, D, "The Kaldi speech recognition toolkit", Proceedings of ASRU 2011, pp.1~4 (2011), IEEE Signal Processing Society
  37. "R: A language and environment for statistical computing" (2021)[URL]
  38. REGENBOGEN, CHRISTINA, "Multimodal human communication-Targeting facial expressions, speech content and prosody", NeuroImage, 60(4), pp.2346~2356 (2012)[DOI]
  39. ROSENBLUM, LAWRENCE D, "The McGurk effect in infants", Perception & Psychophysics, 59(3), pp.347~357 (1997)[DOI]
  40. SLOETJES, HAN, "Annotation by category - ELAN and ISO DCR", Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008) (2008)
  41. SLONE, LAUREN K, "Gaze in action: Head-mounted eye tracking of children’s dynamic visual attention during naturalistic behavior", Journal of Visualized Experiments, 141, pp.e58496~ (2018)[DOI]
  42. STEEN, FRANCIS F, "Toward an infrastructure for data-driven multimodal communication research", Linguistics Vanguard, 4(1), pp.20170041~ (2018)[DOI]
  43. SUANDA, SUMARGA H, "The signal in the noise: The visual ecology of parents & object naming", Infancy, 23, pp.1387~1322 (2018)[DOI]
  44. SUAREZ-RIVERA, CATALINA, "Multimodal parent behaviors within joint attention support sustained attention in infants", Developmental Psychology, 55(1), pp.96~109 (2019)[DOI]
  45. SULLIVAN, JESSICA, "SAYCam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective", Open Mind: Discoveries in Cognitive Science, 5, pp.20~29 (2021)[DOI]
  46. YOSHIDA, HANAKO, "What’s in view for toddlers? Using a head camera to study visual experience", Infancy, 13(3), pp.229~248 (2008)[DOI]
  47. YU, CHEN, "Joint attention without gaze following: Human infants and their parents coordinate visual attention to objects through eyehand coordination", PLOS ONE, 8(11), pp.e79659~ (2013)[DOI]
  48. YUAN, JIAHONG, "Speaker identification on the SCOTUS corpus", The Journal of the Acoustical Society of America, 123(5), pp.3878~3878 (2008)[DOI]

관련 논문AI추천

다면성과 다면 말뭉치에 대하여 - 다면 말뭉치 개발을 위한 기초 연구 -봉미경언어정보연구원 · 언어사실과 관점2011 · 제28권 담화 연구를 위한 자료 구축의 문제 －비언어 요소의 담화 기능을 중심으로－김현강, 송재영언어정보연구원 · 언어사실과 관점2011 · 제27권 한국수어 말뭉치 구축 모형 연구강미영, 변강석, 원성옥, 윤병천, 이봉원, 이현화, 홍성은인문학연구소 · 건지인문학2016 · 제16호코퍼스에 기반을 둔 일본어 언어행동 연구방법론 비교 고찰 -멀티모달 코퍼스와 멀티미디어 코퍼스를 중심으로-손영석대한일어일문학회 · 일어일문학2018 · 제79호몸짓 분석을 위한 프랑스어 다면자료 구축 연구손현정프랑스학회 · 프랑스학연구2011 · 제56호프랑스어 MPA 다면자료 구축 연구손현정한국불어불문학회 · 불어불문학연구2011 · 제88호

이 간행물의 다른 논문 (1건)

미얀마인 한국어 고급 화자의 단모음 산출과 지각 연구이주희pp. 31-53
원문 보기