A Framework for Facial Expression Recognition Combining Contextual Information and Attention Mechanism
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Facial expressions (FEs) serve as fundamental components for human emotion assessment and human–computer interaction. Traditional convolutional neural networks tend to overlook valuable information duringthe FE feature extraction, resulting in suboptimal recognition rates. To address this problem, we propose a deeplearning framework that incorporates hierarchical feature fusion, contextual data, and an attention mechanismfor precise FE recognition. In our approach, we leveraged an enhanced VGGNet16 as the backbone networkand introduced an improved group convolutional channel attention (GCCA) module in each block to emphasizethe crucial expression features. A partial decoder was added at the end of the backbone network to facilitate thefusion of multilevel features for a comprehensive feature map. A reverse attention mechanism guides the modelto refine details layer-by-layer while introducing contextual information and extracting richer expressionfeatures. To enhance feature distinguishability, we employed islanding loss in combination with softmax loss,creating a joint loss function. Using two open datasets, our experimental results demonstrated the effectivenessof our framework. Our framework achieved an average accuracy rate of 74.08% on the FER2013 dataset and98.66% on the CK+ dataset, outperforming advanced methods in both recognition accuracy and stability.
Abstract
Facial expressions (FEs) serve as fundamental components for human emotion assessment and human–computer interaction. Traditional convolutional neural networks tend to overlook valuable information duringthe FE feature extraction, resulting in suboptimal recognition rates. To address this problem, we propose a deeplearning framework that incorporates hierarchical feature fusion, contextual data, and an attention mechanismfor precise FE recognition. In our approach, we leveraged an enhanced VGGNet16 as the backbone networkand introduced an improved group convolutional channel attention (GCCA) module in each block to emphasizethe crucial expression features. A partial decoder was added at the end of the backbone network to facilitate thefusion of multilevel features for a comprehensive feature map. A reverse attention mechanism guides the modelto refine details layer-by-layer while introducing contextual information and extracting richer expressionfeatures. To enhance feature distinguishability, we employed islanding loss in combination with softmax loss,creating a joint loss function. Using two open datasets, our experimental results demonstrated the effectivenessof our framework. Our framework achieved an average accuracy rate of 74.08% on the FER2013 dataset and98.66% on the CK+ dataset, outperforming advanced methods in both recognition accuracy and stability.
참고문헌 (28)
- N. Samadiani, "A review on automatic facial expression recognition systems assisted by multimodal sensor data", Sensors, 19(8) (2019)[DOI]
- S. Li, "Deep facial expression recognition : a survey", IEEE Transactions on Affective Computing, 13(3), pp.1195~1215 (2022)[DOI]
- A. John, "Real-time facial emotion recognition system with improved preprocessing and feature extraction", Proceedings of 2020 3rd International Conference on Smart Systems and Inventive Technology (ICSSIT), pp.1328~1333 (2020)
- S. Saeed, "Automated facial expression recognition framework using deep learning", Journal of Healthcare Engineering, 2022 (2022)[DOI]
- M. S. Ejaz, "Implementation of principal component analysis on masked and non-masked face recognition", Proceedings of 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), pp.1~5 (2019)
- A. S. Rubel, "Facial expression recognition using adaptive robust local complete pattern", Proceedings of 2019 IEEE International Conference on Image Processing (ICIP), pp.41~45 (2019)
- S. Ahmed, "Optimum feature selection with particle swarm optimization to face recognition system using Gabor wavelet transform and deep learning", BioMed Research International, 2021 (2021)[DOI]
- A. Barman, "Facial expression recognition using distance and shape signature features", Pattern Recognition Letters, 145, pp.254~261 (2021)[DOI]
- S. Agarwal, "Facial expression recognition through adaptive learning of local motion descriptor", Multimedia Tools and Applications, 76, pp.1073~1099 (2017)[DOI]
- W. Mellouk, "Facial emotion recognition using deep learning : review and insights", Procedia Computer Science, 175, pp.689~694 (2020)[DOI]
- A. Mollahosseini, "Going deeper in facial expression recognition using deep neural networks", Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1~10 (2016)
- H. Ding, "FaceNet2ExpNet : regularizing a deep face recognition net for expression recognition", Proceedings of 2017 12th IEEE International Conference on Automatic Face &Gesture Recognition (FG 2017), pp.118~126 (2017)
- H. W. Ng, "Deep learning for emotion recognition on small datasets using transfer learning", Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp.443~449 (2015)
- M. Verma, "Facial expression recognition with skip-connection to leverage low-level features", Proceedings of 2019 IEEE International Conference on Image Processing (ICIP), pp.51~55 (2019)
- Y. Liu, "Dynamic multi-channel metric network for joint pose-aware and identity-invariant facial expression recognition", Information Sciences, 578, pp.195~213 (2021)[DOI]
- Z. Niu, "A review on the attention mechanism of deep learning", Neurocomputing, 452, pp.48~62 (2021)[DOI]
- J. Hu, "Squeeze-and-excitation networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.7132~7141 (2018)
- S. Woo, "CBAM: convolutional block attention module", Computer Vision - ECCV 2018, pp.3~19 (2018), Springer
- S. Rathi, "Deep learning and globally guided image filtering technique based image dehazing and enhancement", International Journal of Technical Research & Science, 2020(Special), pp.60~68 (2020)[DOI]
- T. Jia, "Semi-supervised single-image dehazing network via disentangled metaknowledge", IEEE Transactions on Multimedia, 26, pp.2634~2647 (2023)[DOI]
- X. Zhang, "MFFE : multi-scale feature fusion enhanced net for image dehazing", Signal Processing: Image Communication, 105 (2022)[DOI]
- T. Zhang, "Interleaved group convolutions", Proceedings of the IEEE International Conference on Computer Vision, pp.4373~4382 (2017)
- Z. Wu, "Cascaded partial decoder for fast and accurate salient object detection", Proceedings of the IEEE, 2019, 3907~3916
- S. Chen, "Reverse attention for salient object detection", Computer Vision -ECCV 2018, pp.234~250 (2018)
- J. Cai, "Island loss for learning discriminative features in facial expression recognition", Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp.302~309 (2018)
- P. Lucey, "The extended Cohn-Kanade dataset(CK+) : a complete dataset for action unit and emotion-specified expression", Proceedings of 2010IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp.94~101 (2010)
- I. J. Goodfellow, "Challenges in representation learning : a report on three machine learning contests", Neural Information Processing, pp.117~124 (2013), Springer
- P. Jiang, "Accurate and reliable facial expression recognition using advanced softmax loss with fixed weights", IEEE Signal Processing Letters, 27, pp.725~729 (2020)[DOI]