Multimodal Attention-Based Fusion Model for Context- Aware Emotion Recognition
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Human Emotion Recognition is an exciting topic that has been attracting many researchers for a lengthy time. In recent years, there has been an increasing interest in exploiting contextual information on emotion recognition. Some previous explorations in psychology show that emotional perception is impacted by facial expressions, as well as contextual information from the scene, such as human activities, interactions, and body poses. Those explorations initialize a trend in computer vision in exploring the critical role of contexts, by considering them as modalities to infer predicted emotion along with facial expressions. However, the contextual information has not been fully exploited. The scene emotion created by the surrounding environment, can shape how people perceive emotion. Besides, additive fusion in multimodal training fashion is not practical, because the contributions of each modality are not equal to the final prediction. The purpose of this paper was to contribute to this growing area of research, by exploring the effectiveness of the emotional scene gist in the input image, to infer the emotional state of the primary target. The emotional scene gist includes emotion, emotional feelings, and actions or events that directly trigger emotional reactions in the input image. We also present an attention-based fusion network, to combine multimodal features based on their impacts on the target emotional state. We demonstrate the effectiveness of the method, through a significant improvement on the EMOTIC dataset.
Abstract
Human Emotion Recognition is an exciting topic that has been attracting many researchers for a lengthy time. In recent years, there has been an increasing interest in exploiting contextual information on emotion recognition. Some previous explorations in psychology show that emotional perception is impacted by facial expressions, as well as contextual information from the scene, such as human activities, interactions, and body poses. Those explorations initialize a trend in computer vision in exploring the critical role of contexts, by considering them as modalities to infer predicted emotion along with facial expressions. However, the contextual information has not been fully exploited. The scene emotion created by the surrounding environment, can shape how people perceive emotion. Besides, additive fusion in multimodal training fashion is not practical, because the contributions of each modality are not equal to the final prediction. The purpose of this paper was to contribute to this growing area of research, by exploring the effectiveness of the emotional scene gist in the input image, to infer the emotional state of the primary target. The emotional scene gist includes emotion, emotional feelings, and actions or events that directly trigger emotional reactions in the input image. We also present an attention-based fusion network, to combine multimodal features based on their impacts on the target emotional state. We demonstrate the effectiveness of the method, through a significant improvement on the EMOTIC dataset.
참고문헌 (38)
- L. F. Barrett, "Context in emotion perception", Current Directions in Psychological Science, 20(5), pp.286~290 (2011)[DOI]
- L. F. Barrett, "How emotions are made: The secret life of the brain", Houghton Mifflin Harcourt (2017)[DOI]
- A. M. Martinez, "Context may reveal how you feel", Proceedings of the National Academy of Sciences, 116(15), pp.7169~7171 (2019)[DOI]
- B. Mesquita, "Emotions in context: A sociodynamic model of emotions", Emotion Review, 6(4), pp.298~302 (2014)[DOI]
- A. Aldao, "The future of emotion regulation research: Capturing context", Perspectives on Psychological Science, 8(2), pp.155~172 (2013)[DOI]
- K. H. Greenaway, "Context is everything (in emotion research)", Social and Personality Psychology Compass, 12(6), pp.e12393~ (2018)[DOI]
- R. Kosti, "Context based emotion recognition using emotic dataset", IEEE transactions on pattern analysis and machine intelligence, 42(11), pp.2755~2766 (2019)[DOI]
- J. Y. Lee, "Context-aware emotion recognition networks", Proceedings of the IEEE, 2019, ~
- K. Wang, "Cascade attention networks for group emotion recognition with face, body and image cues", Proceedings of the 20th ACM International Conference on Multimodal Interaction (2018)
- M. Zhang, "Context-aware affective graph reasoning for emotion recognition", 2019 IEEE International Conference on Multimedia and Expo (ICME) IEEE (2019)
- De Gelder, Beatrice, "Towards the neurobiology of emotional body language", Nature Reviews Neuroscience, 7(3), pp.242~249 (2006)[DOI]
- Grezes, Julie, "Perceiving fear in dynamic body expressions", Neuroimage, 35(2), pp.959~967 (2007)[DOI]
- Peelen, Marius V., "The neural basis of visual body perception", Nature reviews neuroscience, 8(8), pp.636~648 (2007)[DOI]
- Yamamoto, Kyoko, "The effects of social interaction and personal relationships on facial expressions", Journal of Nonverbal Behavior, 30(4), pp.167~179 (2006)[DOI]
- E. Jakobs, "Social context effects on facial activity in a negative emotional setting", Emotion, 1(1), pp.51~ (2001)[DOI]
- A. Vaswani, "Attention is all you need", ~ (2017)
- K. Xu, "Show, attend and tell:Neural image caption generation with visual attention", International conference on machine learning, PMLR (2015)
- H. Hu, "Relation networks for object detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
- J. Fu, "Dual attention network for scene segmentation", Proceedings of the IEEE, 2019, ~
- K. Sikka, "Multiple kernel learning for emotion recognition in the wild", Proceedings of the 15th ACM on International conference on multimodal interaction (2013)
- H. Gunes, "Bi-modal emotion recognition from expressive face and body gestures", Journal of Network and Computer Applications, 30(4), pp.1334~1345 (2007)[DOI]
- S. H. Yoon, "Speech emotion recognition using multi-hop attention mechanism", ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019), IEEE
- Y. L. Kim, "Deep learning for robust feature generation in audiovisual emotion recognition", 2013 IEEE international conference on acoustics, speech and signal processing (2013), IEEE
- C. W. Lee, "Convolutional attention networks for multimodal emotion recognition from speech and text data", ACL (2018)
- S. Li, "Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild", Proceedings of the IEEE conference on computer vision and pattern recognition (2017)
- K. Liu, "Learn to combine modalities in multimodal deep learning", ~ (2018)
- W. Zijun, "Learning visual emotion representations from web data", Proceedings of the IEEE, 2020, ~
- B. Gelder, "Towards the neurobiology of emotional body language", Nature Reviews Neuroscience, 7(3), pp.242~249 (2006)[DOI]
- K. M. Meeren, "Rapid perceptual integration of facial expression and emotional body language", Proceedings of the National Academy of Sciences, 102(45), pp.16518~16523 (2005)[DOI]
- A. Kleinsmith, "Automatic recognition of non-acted affective postures", IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 41(4), pp.1027~1038 (2011)[DOI]
- S. Konrad, "Recognizing emotions expressed by body pose: A biologically inspired neural model", Neural networks, 21(9), pp.1238~1246 (2008)[DOI]
- J. Deng, "Imagenet: A large-scale hierarchical image database", 2009 IEEE conference on computer vision and pattern recognition (2009), IEEE
- B. Zhou, "Places: An image database for deep scene understanding", ~ (2016)
- L. Tsung-Yi, "Microsoft coco: Common objects in context", European conference on computer vision (2014), Springer, Cham
- B. Zhou, "Semantic understanding of scenes through the ade20k dataset", International Journal of Computer Vision, 127(3), pp.302~321 (2019)[DOI]
- Paszke, Adam, "Automatic differentiation in pytorch", NIPS, ~ (2017)
- H. Kaiming, "Deep residual learning for image recognition", Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
- Selvaraju, "Grad-cam:Visual explanations from deep networks via gradient-based localization", Proceedings of the IEEE international conference on computer vision (2017)