Gesture-Based Emotion Recognition by 3D-CNN and LSTM with Keyframes Selection
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In recent years, emotion recognition has been an interesting and challenging topic. Compared to facial expressions and speech modality, gesture-based emotion recognition has not received much attention with only a few efforts using traditional hand-crafted methods. These approaches require major computational costs and do not offer many opportunities for improvement as most of the science community is conducting their research based on the deep learning technique. In this paper, we propose an end-to-end deep learning approach for classifying emotions based on bodily gestures. In particular, the informative keyframes are first extracted from raw videos as input for the 3D-CNN deep network. The 3D-CNN exploits the short-term spatiotemporal information of gesture features from selected keyframes, and the convolutional LSTM networks learn the long-term feature from the features results of 3D-CNN. The experimental results on the FABO dataset exceed most of the traditional methods results and achieve state-of-the-art results for the deep learning-based technique for gesture-based emotion recognition.
Abstract
In recent years, emotion recognition has been an interesting and challenging topic. Compared to facial expressions and speech modality, gesture-based emotion recognition has not received much attention with only a few efforts using traditional hand-crafted methods. These approaches require major computational costs and do not offer many opportunities for improvement as most of the science community is conducting their research based on the deep learning technique. In this paper, we propose an end-to-end deep learning approach for classifying emotions based on bodily gestures. In particular, the informative keyframes are first extracted from raw videos as input for the 3D-CNN deep network. The 3D-CNN exploits the short-term spatiotemporal information of gesture features from selected keyframes, and the convolutional LSTM networks learn the long-term feature from the features results of 3D-CNN. The experimental results on the FABO dataset exceed most of the traditional methods results and achieve state-of-the-art results for the deep learning-based technique for gesture-based emotion recognition.
참고문헌 (28)
- Andrea Kleinsmith, "Affective Body Expression Perception and Recognition: A Survey", IEEE Transactions on Affective Computing, 4(1), pp.15~33 (2013), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- Michelle Karg, "Body Movements for Affective Expression: A Survey of Automatic Recognition and Generation", IEEE Transactions on Affective Computing, 4(4), pp.341~359 (2013), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- D. Glowinski, "Toward a Minimal Representation of Affective Gestures", IEEE Transactions on Affective Computing, 2(2), pp.106~118 (2011), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- D. Glowinski, "Technique for Automatic Emotion Recognition by Body Gesture Analysis", Proc. CVPR (2008)
- S. Piana, "Real-time Automatic Emotion Recognition from Body Gestures", ~ (2014)
- Stefano Piana, "Adaptive Body Gesture Representation for Automatic Emotion Recognition", ACM Transactions on Interactive Intelligent Systems, 6(1), pp.6~ (2016), Association for Computing Machinery (ACM)[DOI]
- H. Gunes, "A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior", Proc. ICPR, pp.1148~1153 (2006)
- H. Gunes, "Automatic Temporal Segment Detection and Affect Recognition From Face and Body Display", IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(1), pp.64~84 (2009), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- S. Chen, "Recognizing Expression from Face and Body Gesture by Temporal Normalized Motion and Appearance Features", Jounal Image and Vision Computing, 31(2), pp.175~185 (1880)[DOI]
- Weiyi Wang, "Adaptive Real-Time Emotion Recognition from Body Movements", ACM Transactions on Interactive Intelligent Systems, 5(4), pp.18~ (2016), Association for Computing Machinery (ACM)[DOI]
- P. M. Muller, "Emotion Recognition from Embedded Bodily Expression and Speech during Dyadic Interactions", Proc. Affective Computing and Intelligent Interaction (ACII), pp.663~669 (2016)
- Y. Lecun, "Gradient-based learning applied to document recognition", Proc. of the IEEE, 86(11), pp.2278~2324 (1998), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- Pablo Barros, "Multimodal emotional state recognition using sequence-dependent deep hierarchical features", Neural Networks, 72, pp.140~151 (2015), Elsevier BV[DOI]
- Shuiwang Ji, "3D Convolutional Neural Networks for Human Action Recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), pp.221~231 (2013), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- H. Ranganathan, "Multimodal Emotion Recognition using Deep Learning Architectures", Proc. IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1~9 (2016)
- S. T. Ly, "3D Convolution and Convolution LSTM for Gesture-based Emotion Recognition", Int. Symposium on Information Technology Convergence (ISITC) (2018)
- A. Torralba, "80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11), pp.1958~1970 (2008), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- Ruslan Salakhutdinov, "Semantic hashing", International Journal of Approximate Reasoning, 50(7), pp.969~978 (2009), Elsevier BV[DOI]
- Alexandr Andoni, "Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions", IEEE Symposium on Foundations of Computer Science (FOCS'06), 51(1), pp.117~122 (2008), Association for Computing Machinery (ACM)
- Y. Weiss, "Spectral Hashing", Proc. NIPS, pp.1753~1760 (2009)
- M. Raginsky, "Locality Sensitive Binary Codes from Shift-Invariant Kernels", Proc. NIPS, pp.1509~1517 (2009)
- Yunchao Gong, "Iterative Quantization: A Procrustean Approach to Learning Binary Codes for Large-Scale Image Retrieval", IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12), pp.2916~2929 (2013), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- K. Simonyan, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ~ (2014)
- D. Tran, "Learning spatiotemporal features with 3D convolutional networks", Proc. ICCV, pp.4489~4497 (2015)
- R. Hou, "Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos", Proc. ICCV, pp.5822~5831 (2017)
- X. Shi, "Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting", International Conference on Neural Information Processing Systems, 1, pp.802~810
- Jeff Donahue, "Long-Term Recurrent Convolutional Networks for Visual Recognition and Description", IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4), pp.677~691 (2017), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- German I. Parisi, "A Deep Neural Model for Emotion-Driven Multimodal Attention", The AAAI Spring Symposium on Interactive Multisensory Object Perception for Embodied Agents (2017)