Video Expression Recognition Method Based on Spatiotemporal Recurrent Neural Network and Feature Fusion
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Automatically recognizing facial expressions in video sequences is a challenging task because there is littledirect correlation between facial features and subjective emotions in video. To overcome the problem, a videofacial expression recognition method using spatiotemporal recurrent neural network and feature fusion isproposed. Firstly, the video is preprocessed. Then, the double-layer cascade structure is used to detect a face ina video image. In addition, two deep convolutional neural networks are used to extract the time-domain andairspace facial features in the video. The spatial convolutional neural network is used to extract the spatialinformation features from each frame of the static expression images in the video. The temporal convolutionalneural network is used to extract the dynamic information features from the optical flow information frommultiple frames of expression images in the video. A multiplication fusion is performed with the spatiotemporalfeatures learned by the two deep convolutional neural networks. Finally, the fused features are input to thesupport vector machine to realize the facial expression classification task. The experimental results oncNTERFACE, RML, and AFEW6.0 datasets show that the recognition rates obtained by the proposed methodare as high as 88.67%, 70.32%, and 63.84%, respectively. Comparative experiments show that the proposedmethod obtains higher recognition accuracy than other recently reported methods.
Abstract
Automatically recognizing facial expressions in video sequences is a challenging task because there is littledirect correlation between facial features and subjective emotions in video. To overcome the problem, a videofacial expression recognition method using spatiotemporal recurrent neural network and feature fusion isproposed. Firstly, the video is preprocessed. Then, the double-layer cascade structure is used to detect a face ina video image. In addition, two deep convolutional neural networks are used to extract the time-domain andairspace facial features in the video. The spatial convolutional neural network is used to extract the spatialinformation features from each frame of the static expression images in the video. The temporal convolutionalneural network is used to extract the dynamic information features from the optical flow information frommultiple frames of expression images in the video. A multiplication fusion is performed with the spatiotemporalfeatures learned by the two deep convolutional neural networks. Finally, the fused features are input to thesupport vector machine to realize the facial expression classification task. The experimental results oncNTERFACE, RML, and AFEW6.0 datasets show that the recognition rates obtained by the proposed methodare as high as 88.67%, 70.32%, and 63.84%, respectively. Comparative experiments show that the proposedmethod obtains higher recognition accuracy than other recently reported methods.
참고문헌 (30)
- J. Li, "CNN-based facial expression recognition from annotated rgb-d images for human–robot interaction", International Journal of Humanoid Robotics, 16(4) (2019)[DOI]
- M. U. Nagaral, "Hybrid approach for facial expression recognition using HJDLBP and LBP histogram in video sequences", International Journal of Image, Graphics and Signal Processing, 10(2), pp.1~9 (2018)[DOI]
- X. Fan, "A discriminative dynamic framework for facial expression recognition in video sequences", Journal of Visual Communication and Image Representation, 56, pp.182~187 (2018)
- F. Ahmed, "Facial expression recognition under difficult conditions : a comprehensive study on edge directional texture patterns", International Journal of Applied Mathematics and Computer Science, 28(2), pp.399~409 (2018)[DOI]
- H. Yan, "Collaborative discriminative multi-metric learning for facial expression recognition in video", Pattern Recognition, 75, pp.33~40 (2018)
- J. Zhao, "Learning deep facial expression features from image and optical flow sequences using 3D CNN", The Visual Computer, 34(10), pp.1461~1475 (2018)
- A. M. Shabat, "Angled local directional pattern for texture analysis with an application to facial expression recognition", IET Computer Vision, 12(5), pp.603~608 (2018)
- Z. Gong, "Sequential data classification by dynamic state warping", Knowledge and Information Systems, 57(3), pp.545~570 (2018)
- O. Yi, "Dynamic games with asymmetric information : common information based perfect Bayesian equilibria and sequential decomposition", IEEE Transactions on Automatic Control, 62(1), pp.222~237 (2016)
- L. H. Nguyen, "Real-time anomaly detection with Bayesian dynamic linear models", Structural Control and Health Monitoring, 26(9) (2019)[DOI]
- E. Zangeneh, "Facial expression recognition by using differential geometric features", The Imaging Science Journal, 66(8), pp.463~470 (2018)[DOI]
- Z. Sun, "Combining the kernel collaboration representation and deep subspace learning for facial expression recognition", Journal of Circuits, Systems and Computers, 27(8) (2018)[DOI]
- A. Moeini, "Facial expression recognition using dual dictionary learning", Journal of Visual Communication and Image Representation, 45, pp.20~33 (2017)
- E. Owusu, "Face detection based on multilayer feed-forward neural network and Haar features", Software : Practice and Experience, 49(1), pp.120~129 (2019)[DOI]
- N. Jain, "Hybrid deep neural networks for face emotion recognition", Pattern Recognition Letters, 115, pp.101~106 (2018)
- N. P. Gopalan, "Pattern averaging technique for facial expression recognition using support vector machines", IJ Image, Graphics and Signal Processing, 9, pp.27~33 (2018)[DOI]
- M. S. Hossain, "Real time facial expression recognition for nonverbal communication", International Arab Journal of Information Technology, 15(2), pp.278~288 (2018)
- S. Yuan, "Exponential elastic preserving projections for facial expression recognition", Neurocomputing, 275, pp.711~724 (2018)
- Y. Chen, "Robust and energy-efficient expression recognition based on improved deep ResNets", Biomedical Engineering, 2019, pp.64(5), 519~528
- "Facial expression recognition using facial landmark detection and feature extraction via neural networks" (2018)[URL]
- Y. Huang, "Expression-targeted feature learning for effective facial expression recognition", Journal of Visual Communication and Image Representation, 55, pp.677~687 (2018)
- X. Liu, "Adaptive metric learning with deep neural networks for video-based facial expression recognition", Journal of Electronic Imaging, 27(1) (2008)[DOI]
- H. Li, "Multimodal 2D+ 3D facial expression recognition with deep fusion convolutional neural network", IEEE Transactions on Multimedia, 19(12), pp.2816~2831 (2017)
- Z. Yu, "Deeper cascaded peak-piloted network for weak expression recognition", The Visual Computer, 34(12), pp.1691~1699 (2018)
- H. Boughrara, "MLP neural network using modified constructive training algorithm : application to face recognition", International Journal of Intelligent Systems Technologies and Applications, 16(1), pp.53~79 (2017)
- Y. Zhou, "The LAP under facility disruptions during early post-earthquake rescue using PSOGA hybrid algorithm", Fresenius Environmental Bulletin, 28(12 A), pp.9906~9914 (2019)
- J. Jian, "A multi-objective optimization model for green supply chain considering environmental benefits", Sustainability, 11(21) (2019)[DOI]
- N. Wang, "Parsimonious extreme learning machine using recursive orthogonal least squares", IEEE Transactions on Neural Networks and Learning Systems, 25(10), pp.1828~1841 (2014)
- M. Li, "Epidemic forest : a spatiotemporal model for communicable diseases", Annals of the American Association of Geographers, 109(3), pp.812~836 (2019)[DOI]
- S. Yu, "Single image dehazing using multiple transmission layer fusion", Journal of Modern Optics, 63(6), pp.519~535 (2016)[DOI]