Audio and Video Bimodal Emotion Recognition in Social Networks Based on Improved AlexNet Network and Attention Mechanism
2
조회
0
다운로드
2
인용
0
참고문헌
키워드
초록
In the task of continuous dimension emotion recognition, the parts that highlight the emotional expression arenot the same in each mode, and the influences of different modes on the emotional state is also different. Therefore, this paper studies the fusion of the two most important modes in emotional recognition (voice andvisual expression), and proposes a two-mode dual-modal emotion recognition method combined with theattention mechanism of the improved AlexNet network. After a simple preprocessing of the audio signal andthe video signal, respectively, the first step is to use the prior knowledge to realize the extraction of audiocharacteristics. Then, facial expression features are extracted by the improved AlexNet network. Finally, themultimodal attention mechanism is used to fuse facial expression features and audio features, and the improvedloss function is used to optimize the modal missing problem, so as to improve the robustness of the model andthe performance of emotion recognition. The experimental results show that the concordance coefficient of theproposed model in the two dimensions of arousal and valence (concordance correlation coefficient) were 0.729and 0.718, respectively, which are superior to several comparative algorithms.
Abstract
In the task of continuous dimension emotion recognition, the parts that highlight the emotional expression arenot the same in each mode, and the influences of different modes on the emotional state is also different. Therefore, this paper studies the fusion of the two most important modes in emotional recognition (voice andvisual expression), and proposes a two-mode dual-modal emotion recognition method combined with theattention mechanism of the improved AlexNet network. After a simple preprocessing of the audio signal andthe video signal, respectively, the first step is to use the prior knowledge to realize the extraction of audiocharacteristics. Then, facial expression features are extracted by the improved AlexNet network. Finally, themultimodal attention mechanism is used to fuse facial expression features and audio features, and the improvedloss function is used to optimize the modal missing problem, so as to improve the robustness of the model andthe performance of emotion recognition. The experimental results show that the concordance coefficient of theproposed model in the two dimensions of arousal and valence (concordance correlation coefficient) were 0.729and 0.718, respectively, which are superior to several comparative algorithms.
참고문헌 (27)
- R. Jamwal, "The use of electronic assistive technology for social networking by people with disability living in shared supported accommodation", Disability and Rehabilitation : Assistive Technology, 15(1), pp.101~108 (2020)
- S. Sharma, "Use of social networking sites by SMEs to engage with their customers : a developing country perspective", Journal of Internet Commerce, 19(1), pp.62~81 (2020)
- D. Tiwari, "Ensemble approach for twitter sentiment analysis", International Journal of Information Technology and Computer Science, 11(8), pp.20~26 (2019)
- R. Bhargava, "Neural network-based architecture for sentiment analysis in Indian languages", Journal of Intelligent Systems, 28(3), pp.361~375 (2019)
- J. McDonald, "Advancing text-analysis to tap into the student voice : a proof-of-concept study", Assessment & Evaluation in Higher Education, 45(1), pp.154~164 (2020)
- F. R. Sullivan, "Exploring the potential of natural language processing to support microgenetic analysis of collaborative learning discussions", British Journal of Educational Technology, 50(6), pp.3047~3063 (2019)
- A. L. Afzal, "Deep multiple multilayer kernel learning in core vector machines", Expert Systems with Applications, 96, pp.149~156 (2018)
- G. Manogaran, "Hybrid recommendation system for heart disease diagnosis based on multiple kernel learning with adaptive neuro-fuzzy inference system", Multimedia Tools and Applications, 77(4), pp.4379~4399 (2018)
- Z. Dong, "BMF-CNN : an object detection method based on multi-scale feature fusion in VHR remote sensing images", Remote Sensing Letters, 11(3), pp.215~224 (2020)
- A. Moussavi-Khalkhali, "Feature fusion models for deep autoencoders : application to traffic flow prediction", Applied Artificial Intelligence, 33(13), pp.1179~1198 (2019)
- M. Wollmer, "Youtube movie reviews : sentiment analysis in an audio-visual context", IEEE Intelligent Systems, 28(3), pp.46~53 (2013)
- S. Zhang, "Multimodal emotion recognition integrating affective speech with facial expression", WSEAS Transactions on Signal Processing, 10, pp.526~537 (2014)
- S. Dobrisek, "Towards efficient multi-modal emotion recognition", International Journal of Advanced Robotic Systems, 10 (2013)
- Y. Wang, "Multimodal emotion recognition algorithm based on edge network emotion element compensation and data fusion", Personal and Ubiquitous Computing, 23(3), pp.383~392 (2019)
- S. Sahoo, "Emotion recognition from audio-visual data using rule based decision level fusion", Proceedings of 2016 IEEE Students’ Technology Symposium (TechSym), pp.7~12 (2016)
- M. Glodek, "Multiple Classifier Systems", pp.85~94 (2013), Springer
- P. Tzirakis, "End-to-end multimodal emotion recognition using deep neural networks", IEEE Journal of Selected Topics in Signal Processing, 11(8), pp.1301~1309 (2017)
- J. Huang, "Multimodal emotion recognition with transfer learning of deep neural network", ZTE Communications, 15(S2), pp.23~29 (2017)
- S. Zhang, "Learning affective features with a hybrid deep model for audio-visual emotion recognition", IEEE Transactions on Circuits and Systems for Video Technology, 28(10), pp.3030~3043 (2018)
- W. Li, "Design of a hierarchy modular neural network and its application in multimodal emotion recognition", Soft Computing, 23(22), pp.11817~11828 (2019)
- M. Valstar, "AVEC 2014:3D dimensional affect and depression recognition challenge", Proceedings of the 4th International Workshop on Audio, 2014, 3~10
- Y. Wang, "Kernel cross-modal factor analysis for information fusion with application to bimodal emotion recognition", IEEE Transactions on Multimedia, 14(3), pp.597~607 (2012)
- S. Poria, "Fusing audio, visual and textual clues for sentiment analysis from multimodal content", Neurocomputing, 174, pp.50~59 (2016)
- N. Rasiwasia, "A new approach to cross-modal multimedia retrieval", Proceedings of the 18th ACM International Conference on Multimedia, pp.251~260 (2010)
- J. Ma, "Multimodal emotion recognition for the fusion of speech and EEG signals", Xi'an Dianzi Keji Daxue Xuebao, 2019, pp.46(1), 143~150
- B. Knyazev, "Leveraging large face recognition data for emotion classification", Proceedings of 2018 13th IEEE International Conference on Automatic Face &Gesture Recognition (FG2018), pp.692~696 (2018)
- J. Friedman, "Regularization paths for generalized linear models via coordinate descent", Journal of Statistical Software, 33(1), pp.1~22 (2010)