A Video Expression Recognition Method Based on Multi-mode Convolution Neural Network and Multiplicative Feature Fusion
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The existing video expression recognition methods mainly focus on the spatial feature extraction of videoexpression images, but tend to ignore the dynamic features of video sequences. To solve this problem, a multimode convolution neural network method is proposed to effectively improve the performance of facialexpression recognition in video. Firstly, OpenFace 2.0 is used to detect face images in video, and two deepconvolution neural networks are used to extract spatiotemporal expression features. Furthermore, spatialconvolution neural network is used to extract the spatial information features of each static expression image,and the dynamic information feature is extracted from the optical flow information of multiple expressionimages based on temporal convolution neural network. Then, the spatiotemporal features learned by the twodeep convolution neural networks are fused by multiplication. Finally, the fused features are input into supportvector machine to realize the facial expression classification. Experimental results show that the recognitionaccuracy of the proposed method can reach 64.57% and 60.89%, respectively on RML and Baum-ls datasets. It is better than that of other contrast methods.
Abstract
The existing video expression recognition methods mainly focus on the spatial feature extraction of videoexpression images, but tend to ignore the dynamic features of video sequences. To solve this problem, a multimode convolution neural network method is proposed to effectively improve the performance of facialexpression recognition in video. Firstly, OpenFace 2.0 is used to detect face images in video, and two deepconvolution neural networks are used to extract spatiotemporal expression features. Furthermore, spatialconvolution neural network is used to extract the spatial information features of each static expression image,and the dynamic information feature is extracted from the optical flow information of multiple expressionimages based on temporal convolution neural network. Then, the spatiotemporal features learned by the twodeep convolution neural networks are fused by multiplication. Finally, the fused features are input into supportvector machine to realize the facial expression classification. Experimental results show that the recognitionaccuracy of the proposed method can reach 64.57% and 60.89%, respectively on RML and Baum-ls datasets. It is better than that of other contrast methods.
참고문헌 (30)
- Sayan Maity, "Multimodal Biometrics Recognition from Facial Video with Missing Modalities Using Deep Learning", JIPS(Journal of Information Processing Systems), 16(1), pp.6~29 (2020), 한국정보처리학회[DOI]
- O. Agbolade, "3-Dimensional facial expression recognition in human using multi-points warping", BMC Bioinformatics, 20(1) (2019)[DOI]
- K. Talele, "Facial expression recognition using digital signature feature descriptor", Signal, Image and Video Processing, 14, pp.701~709 (2020)[DOI]
- H. Sadeghi, "Human vision inspired feature extraction for facial expression recognition", Multimedia Tools and Applications, 78(21), pp.30335~30353 (2019)
- M. Sajjad, "Human behavior understanding in big multimedia data using CNN based facial expression recognition", Mobile Networks and Applications, 25, pp.1611~1621 (2020)[DOI]
- S. Nestler, "Safety-critical human computer interaction", it-Information Technology, 61(1), pp.67~70 (2019)
- P. Loslever, "Using fuzzy coding with qualitative data : example with subjective data in human-computer interaction", Theoretical Issues in Ergonomics Science, 20(4), pp.459~488 (2019)
- U. A. Shaikh, "Dynamic scene multi-exposure image fusion", IETE Journal of Education, 59(2), pp.53~61 (2018)
- Y. Jiang, "A novel distributed multitask fuzzy clustering algorithm for automatic MR brain image segmentation", Journal of Medical Systems, 43 (2019)[DOI]
- F. Ramdani, "Analysis of the application of an advanced classifier algorithm to ultra-high resolution unmanned aerial aircraft imagery : a neural network approach", International Journal of Remote Sensing, 41(9), pp.3266~3286 (2020)
- N. Zikiou, "Hyperspectral image classification using graph-based wavelet transform", International Journal of Remote Sensing, 41(7), pp.2624~2643 (2020)
- F. An, "Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM", The Visual Computer, 36(3), pp.483~498 (2020)
- R. Ramya, "3D facial expression recognition using multi-channel deep learning framework", Circuits, Systems, and Signal Processing, 39(2), pp.789~804 (2020)
- C. Xu, "Person-independent facial expression recognition method based on improved Wasserstein generative adversarial networks in combination with identity aware", Multimedia Systems, 26(1), pp.53~61 (2020)
- K. Li, "Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy", The Visual Computer, 36(2), pp.391~404 (2020)
- S. Kumar, "Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition", The Visual Computer, 37(1), pp.143~159 (2021)
- H. D. Nguyen, "Facial emotion recognition using an ensemble of multi-level convolutional neural networksm", International Journal of Pattern Recognition and Artificial Intelligence, 33(11) (2019)[DOI]
- X. Liu, "Emotion recognition based on multi-composition deep forest and transferred convolutional neural network", Journal of Advanced Computational Intelligence and Intelligent Informatics, 23(5), pp.883~890 (2019)
- N. Jain, "Effective approach for facial expression recognition using hybrid squarebased diagonal pattern geometric model", Multimedia Tools and Applications, 78(20), pp.29555~29571 (2019)
- F. Kong, "Facial expression recognition method based on deep convolutional neural network combined with improved LBP features", Personal and Ubiquitous Computing, 23(3), pp.531~539 (2019)
- F. Z. Salmam, "Fusing multi-stream deep neural networks for facial expression recognition", Signal, Image and Video Processing, 13(3), pp.609~616 (2019)
- X. Zhu, "Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network", The Visual Computer, 36, pp.743~755 (2020)
- Y. Luo, "Facial expression recognition based on improved completed local ternary patterns", Optoelectronics Letters, 15(3), pp.224~230 (2019)
- S. Nigam, "Efficient facial expression recognition using histogram of oriented gradients in wavelet domain", Multimedia Tools and Applications, 77(21), pp.28725~28747 (2018)
- M. S. Zia, "A novel spontaneous facial expression recognition using dynamically weighted majority voting based ensemble classifier", Multimedia Tools and Applications, 77(19), pp.25537~25567 (2018)
- H. Wang, "Facial expression recognition using iterative fusion of MO-HOG and deep features", The Journal of Supercomputing, 76(5), pp.3211~3221 (2020)
- T. Baltrusaitis, "Openface 2.0: Facial behavior analysis toolkit", Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp.59~66 (2018)
- Y. Zhou, "The LAP under facility disruptions during early post-earthquake rescue using PSOGA hybrid algorithm", Fresenius Environmental Bulletin, 28(12A), pp.9906~9914 (2019)
- J. Jian, "A multi-objective optimization model for green supply chain considering environmental benefits", Sustainability, 11(21) (2019)[DOI]
- Y. Ren, "Deep spatio-temporal residual neural networks for road-network-based data modeling", International Journal of Geographical Information Science, 33(9), pp.1894~1912 (2019)