얼굴영상과 음성을 이용한 멀티모달 감정인식
Multimodal Emotion Recognition using Face Image and Speech
1
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
A challenging research issue that has been one of growing importance to those working in human-computer interaction are to endow a machine with an emotional intelligence. Thus,emotion recognition technology plays an important role in the research area of human-computer interaction, and it allows a more natural and more human-like communication between human and computer. In this paper, we propose the multimodal emotion recognition system using face and speech to improve recognition performance. The distance measurement of the face-based emotion recognition is calculated by 2D-PCA of MCS-LBP image and nearest neighbor classifier, and also the likelihood measurement is obtained by Gaussian mixture model algorithm based on pitch and mel-frequency cepstral coefficient features in speech-based emotion recognition. The individual matching scores obtained from face and speech are combined using a weighted-summation operation, and the fused-score is utilized to classify the human emotion. Through experimental results, the proposed method exhibits improved recognition accuracy of about 11.25% to 19.75% when compared to the most uni-modal approach. From these results, we confirmed that the proposed approach achieved a significant performance improvement and the proposed method was very effective.
Abstract
A challenging research issue that has been one of growing importance to those working in human-computer interaction are to endow a machine with an emotional intelligence. Thus,emotion recognition technology plays an important role in the research area of human-computer interaction, and it allows a more natural and more human-like communication between human and computer. In this paper, we propose the multimodal emotion recognition system using face and speech to improve recognition performance. The distance measurement of the face-based emotion recognition is calculated by 2D-PCA of MCS-LBP image and nearest neighbor classifier, and also the likelihood measurement is obtained by Gaussian mixture model algorithm based on pitch and mel-frequency cepstral coefficient features in speech-based emotion recognition. The individual matching scores obtained from face and speech are combined using a weighted-summation operation, and the fused-score is utilized to classify the human emotion. Through experimental results, the proposed method exhibits improved recognition accuracy of about 11.25% to 19.75% when compared to the most uni-modal approach. From these results, we confirmed that the proposed approach achieved a significant performance improvement and the proposed method was very effective.
참고문헌 (33)
- S. I. Stephen, "Using a live-In laboratory for ubiquitous computing research", Lecture Notes in Computer Science, 3968, pp.349~365 (2006)
- K. Partridge, "On using existing time-use study data for ubiquitous computing applications", Proceedings of the 10th international conference on Ubiquitous computing, 344, pp.144~153 (2008)
- Y. S. Shin, "The effect of facial expression recognition based on the dimensions of emotion using PCA representation and neural networks", Lecture Notes in Computer Science, 3656, pp.1133~1140 (2005)
- P. Penev, "Local feature analysis: a general statistical theory for object representation", Network : Computation in Neural Systems, 7, pp.477~500 (1996)
- P. Belhumeur, "Eigenfaces vs. fisherfaces: Recognition using class specific linear projection", IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), pp.711~720 (1997)
- M. S. Bartlett, "Independent of face images: a representation for face recognition", Proceedings of the Fourth Annual Joint Symposium on Neural Computation (1997)
- C. Padgett, "Representing face images for emotion classification", Advances in Neural Information Processing Systems, 9 (1997)
- Z. Zhang, "Comparison between geometry based and Gaborwavelets-based facial expression recognition using multi-layer perceptron", Third IEEE International Conference on Automatic Face and Gesture Recognition, pp.454~459 (1998)
- J. Nicholson, "Emotion recognition in speech using neural networks", Neural Computing & Application, 9, pp.290~296 (2000)
- A. A. Razak, "Comparison between fuzzy and NN method for speech emotion recognition", Proceedings of the Third International Conference on Information Technology and Applications (2005)
- C. M. Lee, "Classifying emotions in human-machine spoken dialogs", ICME’02, 1, pp.737~740 (2002)
- Hyoun-Joo Go, "A multimodal emotion recognition using the facial image and speech signal", International Journal of Fuzzy Logic and Intelligent systems, 5(1), pp.1~6 (2005), 한국지능시스템학회
- M. Song, "A robust multimodal approach for emotion recognition", Neurocomputing, 71, pp.1913~1920 (2008)
- C. Shan, "Facial expression recognition based on local binary pattern : A comprehensive study", Image and Vision Computing, 27, pp.803~816 (2009)
- T. Ahonen, "Face recognition with local binary patterns", ECCV, pp.469~481 (2004)
- G. Zhang, "Boosting local binary pattern-based face recognition", in Proc. Advances in Biometric Person Authentication, 3338, pp.179~186 (2004)
- X. Fu, "Centralized binary patterns embedded with image euclidean distance for facial expression recognition", Fourth International Conference on Natural Computation, 4, pp.115~199 (2008)
- K. Meena, "Local binary patterns and its variants for face recognition", International Conference on Recent Trends in Information Technology, pp.782~786 (2011)
- M. Kirby, "Application of the Karhunen-Loeve procedure for the characterization of human faces", IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(1), pp.103~108 (1990)
- M. Turk, "Eigenfaces for recognition", Journal of Cognitive Neuroscience, 3(1), pp.71~86 (1991)
- Y. Jian, "Two-dimensional PCA : A new approach to appearance-based face representation and recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), pp.131~137 (2004)
- A. P. Dempster, "Maximum likelihood from incomplete data via the EM algorithm", Journal of the Royal Statistical Society B, pp.1~38 (1977)
- A. Ross, "Information fusion in biometrics", Pattern Recognition. Letter, 24(13), pp.2115~2125 (2003)
- C. Sanderson, "Identity verification using speech and face information", Digital Signal Processing, 14(5), pp.449~480 (2004)
- J. L. Michael, "Coding facial expressions with gabor wavelets", Proceedings, Third IEEE International Conference on Automatic Face and Gesture Recognition, pp.200~205 (1998)
- M. Minear, "A lifespan database of adult facial stimuli", Behavior Research Methods, Instruments and Computers, 36, pp.630~633 (2004)
- K. H. Hyun, "Improvement of emotion recognition by Bayesian classifier using non-zero-pitch concept", IEEE International Workshop on Robot and Human Interactive Communication, pp.312~316 (2005)
- 박창현, "음성 신호를 이용한 감정인식에서의 패턴인식 방법", 한국퍼지 및 지능시스템학회 논문지, 12(3), pp.284~288 (2006)
- 김정철, "모바일 환경에서 Haar-Like Features와 PCA를 이용한 실시간 얼굴 인증 시스템", (사)디지털산업정보학회 논문지, 6(2), pp.199~207 (2010), (사)디지털산업정보학회
- 이현구, "차세대 PC를 위한 음성 인터페이스에 관한 연구", (사)디지털산업정보학회 논문지, 2(3), pp.59~66 (2006), (사)디지털산업정보학회
- 강면구, "음성 신호를 사용한 GMM 기반의 감정인식", 한국음향학회지, 23(3), pp.235~241 (2004), 한국음향학회
- 고현주, "얼굴표정과 음성을 이용한 감정인식", 정보과학회논문지 : 소프트웨어 및 응용, 31(6), pp.799~807 (2004), 한국정보과학회
- 이현구, "HCI를 위한 오감정보처리에 관한 연구", (사)디지털산업정보학회 논문지, 5(2), pp.77~85 (2009), (사)디지털산업정보학회