Multimodal Biometrics Recognition from Facial Video with Missing Modalities Using Deep Learning
2
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
Biometrics identification using multiple modalities has attracted the attention of many researchers as it producesmore robust and trustworthy results than single modality biometrics. In this paper, we present a novel multimodalrecognition system that trains a deep learning network to automatically learn features after extracting multiplebiometric modalities from a single data source, i.e., facial video clips. Utilizing different modalities, i.e., leftear, left profile face, frontal face, right profile face, and right ear, present in the facial video clips, we trainsupervised denoising auto-encoders to automatically extract robust and non-redundant features. The automaticallylearned features are then used to train modality specific sparse classifiers to perform the multimodalrecognition. Moreover, the proposed technique has proven robust when some of the above modalities weremissing during the testing. The proposed system has three main components that are responsible for detection,which consists of modality specific detectors to automatically detect images of different modalities present infacial video clips; feature selection, which uses supervised denoising sparse auto-encoders network to capturediscriminative representations that are robust to the illumination and pose variations; and classification, whichconsists of a set of modality specific sparse representation classifiers for unimodal recognition, followed byscore level fusion of the recognition results of the available modalities. Experiments conducted on theconstrained facial video dataset (WVU) and the unconstrained facial video dataset (HONDA/UCSD), resultedin a 99.17% and 97.14% Rank-1 recognition rates, respectively. The multimodal recognition accuracydemonstrates the superiority and robustness of the proposed approach irrespective of the illumination, nonplanarmovement, and pose variations present in the video clips even in the situation of missing modalities.
Abstract
Biometrics identification using multiple modalities has attracted the attention of many researchers as it producesmore robust and trustworthy results than single modality biometrics. In this paper, we present a novel multimodalrecognition system that trains a deep learning network to automatically learn features after extracting multiplebiometric modalities from a single data source, i.e., facial video clips. Utilizing different modalities, i.e., leftear, left profile face, frontal face, right profile face, and right ear, present in the facial video clips, we trainsupervised denoising auto-encoders to automatically extract robust and non-redundant features. The automaticallylearned features are then used to train modality specific sparse classifiers to perform the multimodalrecognition. Moreover, the proposed technique has proven robust when some of the above modalities weremissing during the testing. The proposed system has three main components that are responsible for detection,which consists of modality specific detectors to automatically detect images of different modalities present infacial video clips; feature selection, which uses supervised denoising sparse auto-encoders network to capturediscriminative representations that are robust to the illumination and pose variations; and classification, whichconsists of a set of modality specific sparse representation classifiers for unimodal recognition, followed byscore level fusion of the recognition results of the available modalities. Experiments conducted on theconstrained facial video dataset (WVU) and the unconstrained facial video dataset (HONDA/UCSD), resultedin a 99.17% and 97.14% Rank-1 recognition rates, respectively. The multimodal recognition accuracydemonstrates the superiority and robustness of the proposed approach irrespective of the illumination, nonplanarmovement, and pose variations present in the video clips even in the situation of missing modalities.
참고문헌 (52)
- F. Karray, "Multi modal biometric systems : a state of the rt survey", Proceedings of the 4th International Conference on Computational Intelligence (2007), Robotics and Autonomous Systems (CIRAS)
- S. Cadavid, "Multi-modal biometric modeling and recognition of the human face and ear", Proceedings of 2009 IEEE International Workshop on Safety, pp.1~6 (2009), Security & Rescue Robotics (SSRR)
- M. H. Mahoor, "A multimodal approach for face modeling and recognition", IEEE Transactions on Information Forensics and Security, 3(3), pp.431~440 (2008)
- A. Ross, "Multimodal biometrics : an overview", Proceedings of 2004 12th European Signal Processing Conference, pp.1221~1224 (2004)
- P. Viola, "Rapid object detection using a boosted cascade of simple features", Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR), pp.511~518 (2001)
- Z. Huang, "A robust face and ear based multimodal biometric system using sparse representation", Pattern Recognition, 46(8), pp.2156~2168 (2013)
- G. Fahmy, "The effect of lighting direction", condition on the performance of face recognition algorithms, 2006, ~[DOI]
- K. C. Lee, "Visual tracking and recognition using probabilistic appearance manifolds", Computer Vision and Image Understanding, 99(3), pp.303~331 (2005)
- A. M. Bronstein, "Three-dimensional face recognition", International Journal of Computer Vision, 64(1), pp.5~30 (2005)
- K. Etemad, "Discriminant analysis for recognition of human face images", Journal of the Optical Society of America A, 14(8), pp.1724~1733 (1997)
- C. Lu, "Surpassing human-level face verification performance on LFW with GaussianFace", Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp.3811~3819 (2015)
- Y. Taigman, "Deepface : closing the gap to human-level performance in face verification", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1701~1708 (2014)
- M. Turk, "Eigenfaces for recognition", Journal of Cognitive Neuroscience, 3(1), pp.71~86 (1991)
- P. P. Sarangi, "Multimodal biometric recognition using human ear and profile face", Proceedings of 2018 4th International Conference on Recent Advances in Information Technology(RAIT), pp.1~6 (2018)
- Y. C. Chen, "Dictionary-based face recognition from video", Computer Vision – ECCV 2012, pp.766~779 (2012), Springer
- K. C. Lee, "Video-based face recognition using probabilistic appearance manifolds", Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.313~320 (2003)
- U. Park, "3D model-based face recognition in video", Advances in Biometrics, pp.1085~1094 (2007), Springer
- U. Park, "Face recognition in video : adaptive fusion of multiple matchers", Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp.1~8 (2007)
- A. Lumini, "Overview of the combination of biometric matchers", Information Fusion, 33, pp.71~85 (2017)
- A. Abaza, "A survey on ear biometrics", ACM Computing Surveys, 45(2), pp.1~35 (2013)
- N. B. Boodoo, "Robust multi biometric recognition using face and ear images", International Journal of Computer Science and Information Security, 6(2), pp.164~169 (2009)
- "Multibiometrics belief fusion" (2010)[URL]
- X. Pan, "Ear and face based multimodal recognition based on KFDA", Proceedings of 2008 International Conference on Audio, pp.965~969 (2008)
- S. El-Naggar, "Surveillance in Action", pp.77~104 (2018), Springer
- J. Ngiam, "Multimodal deep learning", Proceedings of the 28th International Conference on Machine Learning, pp.689~696 (2011)
- A. Krizhevsky, "Imagenet classification with deep convolutional neural networks", Advances in Neural Information Processing Systems, 25, pp.1097~1105 (2012)
- S. Chopra, "Learning a similarity metric discriminatively, with application to face verification", Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR), pp.539~546 (2005)
- G. Goswami, "MDLFace : memorability augmented deep learning for video face recognition", Proceedings of IEEE International Joint Conference on Biometrics, pp.1~7 (2014)
- D. Menotti, "Deep representations for iris, face, and fingerprint spoofing detection", IEEE Transactions on Information Forensics and Security, 10(4), pp.864~879 (2015)
- S. Gao, "Single sample face recognition via learning deep supervised autoencoders", IEEE Transactions on Information Forensics and Security, 10(10), pp.2108~2118 (2015)
- P. J. Phillips, "The FERET evaluation methodology for face-recognition algorithms", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10), pp.1090~1104 (2000)
- K. Chang, "Comparison and combination of ear and face images in appearance-based biometrics", IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), pp.1160~1165 (2003)
- C. Liu, "Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition", IEEE Transactions on Image processing, 11(4), pp.467~476 (2002)
- R. Khorsandi, "Ear recognition via sparse representation and Gabor filters", Proceedings of 2012 IEEE 5th International Conference on Biometrics : Theory, pp.278~282 (2012)
- S. Urolagin, "Rotation invariant object recognition using Gabor filters", Proceedings of 2010 5th International Conference on Industrial and Information Systems, pp.404~407 (2010)
- M. Yang, "Gabor feature based sparse representation for face recognition with Gabor occlusion dictionary", Computer Vision – ECCV 2010, pp.448~461 (2010), Springer
- Y. Bengio, "Learning deep architectures for AI", Foundations and Trends in Machine Learning, 2(1), pp.1~127 (2009)
- D. E. Rumelhart, "MacClelland, Parallel Distributed Processing, Explorations in the Microstructure of Cognition" (1986), MIT Press
- I. Jolliffe, "Encyclopedia of Statistics in Behavioral Science" (2005), John Wiley & Sons
- S. Rifai, "Contractive auto-encoders : explicit invariance during feature extraction", Proceedings of the 28th International Conference on Machine Learning, pp.833~840 (2011)
- P. Lennie, "The cost of cortical computation", Current Biology, 13(6), pp.493~497 (2003)
- P. Vincent, "Extracting and composing robust features with denoising autoencoders", Proceedings of the 25th International Conference on Machine Learning, pp.1096~1103 (2008)
- G. E. Hinton, "Reducing the dimensionality of data with neural networks", Science, 313(5786), pp.504~507 (2006)
- Y. Bengio, "Large-Scale Kernel Machines", pp.321~360 (2007), MIT Press
- M. Ranzato, "Efficient learning of sparse representations with an energy-based model", Advances in Neural Information Processing Systems, 19, pp.1137~1144 (2006)
- Y. Bengio, "Greedy layer-wise training of deep networks", Advances in Neural Information Processing Systems, 19, pp.153~160 (2006)
- R. Lengelle, "Training MLPs layer by layer using an objective function for internal representations", Neural Networks, 9(1), pp.83~97 (1996)
- G. E. Hinton, "A fast learning algorithm for deep belief nets", Neural Computation, 18(7), pp.1527~1554 (2006)
- H. Mohimani, "A fast approach for overcomplete sparse decomposition based on smoothed l0 norm", IEEE Transactions on Signal Processing, 57(1), pp.289~301 (2008)
- A. A. Ross, "Handbook of Multibiometrics" (2006), Springer
- F. R. Hampel, "Robust Statistics: The Approach Based on Influence Functions" (2011), John Wiley & Sons
- M. Aharon, "K-SVD : an algorithm for designing overcomplete dictionaries for sparse representation", IEEE Transactions on Signal Processing, 54(11), pp.4311~4322 (2006)