수화 인식 성능 향상을 위한 입 모양 인식 방법
Approaches of Lip Reading to Assist Sign Language Understanding
2
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
With the rapid development of capable machine learning models, comes the possibility for the creation tools and services to benefit those with disabilities. There are currently an estimated 70 million individuals who suffer from hearing impairments. To help overcome this issue, we will be researching the application of using lip-reading machine learning models to help understand and translate those who communicate through sign language (signers). This paper will present the current state of the art lip reading models along with the current limitations. We show that the real-world accuracy of existing models significantly underperforms results reported in controlled testing environments. We propose solutions to these problems as well as our own design approach to create a lipreading model to help understand sign language. This research is preliminary research to use lip reading to enhance the performance of sign understanding, which is a novel idea, and will contribute to sign understanding with lip reading as a future work.
Abstract
With the rapid development of capable machine learning models, comes the possibility for the creation tools and services to benefit those with disabilities. There are currently an estimated 70 million individuals who suffer from hearing impairments. To help overcome this issue, we will be researching the application of using lip-reading machine learning models to help understand and translate those who communicate through sign language (signers). This paper will present the current state of the art lip reading models along with the current limitations. We show that the real-world accuracy of existing models significantly underperforms results reported in controlled testing environments. We propose solutions to these problems as well as our own design approach to create a lipreading model to help understand sign language. This research is preliminary research to use lip reading to enhance the performance of sign understanding, which is a novel idea, and will contribute to sign understanding with lip reading as a future work.
참고문헌 (30)
- D. Mamo, "The Indigenous World 2020", IWGIA (2020)
- S. Lai, "Lip reading in the wild", Computer Vision – ACCV 2016, pp.87~103 (2017)
- M. Cooke, "An audiovisual corpus for speech perception and automatic speech recognition", The Journal of the Acoustical Society of America, 120(5), pp.2421~2424 (2006)[DOI]
- B. Martinez, "Lipreading using temporal convolutional networks", ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.6319~6323 (2020)
- "Avicar: audio-visual speech corpus in a car environment" (2004)[URL]
- I. Matthews, "Extraction of visual features for lipreading", IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(2), pp.198~213 (2002)[DOI]
- E. K. Patterson, "Cuave : A new audio-visual database for multimodal human-computer interface research", 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, pp.II-2002~II-2020 (2002)
- G. Zhoa, "Lipreading with local spatiotemporal descriptors", IEEE Transactions on Multimedia, 11(7), pp.1254~1265 (2009)[DOI]
- I. Anina, "Ouluvs2 : A Multiview audiovisual database for non-rigid mouth motion analysis", 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp.1~5 (2015)
- J. S. Chung, "Lip reading in profile", British Machine Vision Conference (2017)
- J. S. Chung, "Lip reading sentences in the wild", 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.3444~3453 (2017)
- P. Ma, "Towards practical lipreading with distilled and efficient models", ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.7608~7612 (2021)
- Y. Fu, "Classification and feature extraction by simplexization", IEEE Transactions on Information Forensics and Security, 3(1), pp.91~100 (2008)[DOI]
- T. Ozcan, "Lip reading using convolutional neural networks with and without pre-trained models", Balkan Journal of Electrical & Computer Engineering, 7(2), pp.195~201 (2019)[DOI]
- A. Krizhevsky, "Imagenet classification with deep convolutional neural networks", Advances in Neural Information Processing Systems 25(NIPS 2012), 25, pp.1106~1114 (2012)
- C. Szegedy, "Going deeper with convolutions", 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1~9 (2014)
- G. Papandreou, "Adaptive multimodal fusion by uncertainty compensation with application to audiovisual speech recognition", IEEE Transactions on Audio, Speech, and Language Processing, 17(3), pp.423~435 (2009)[DOI]
- Y. Lan, "Improving visual features for lip-reading", International Conference on Auditory-Visual Speech Processing (2010)
- Y. Pei, "Unsupervised random forest manifold alignment for lipreading", 2013 IEEE International Conference on Computer Vision, pp.129~136 (2013)
- Z. Zhou, "A review of recent advances in visual speech decoding", Image and Vision Computing, 32(9), pp.590~605 (2014)[DOI]
- S. Petridis, "Deep complementary bottleneck features for visual speech recognition", 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.2304~2308 (2016)
- T. Saitoh, "Concatenated frame image-based CNN for visual speech recognition", Computer Vision – ACCV 2016 Workshop, pp.277~289 (2016)
- D. Lee, "Multi-view automatic lip-reading using neural network", Computer Vision – ACCV 2016 Workshops, pp.290~302 (2016)
- F. Tracey, "Human Motion Mimicking on Pepper Social Robot", The 2023 Australasian Conference on Robotics and Automation (ACRA 2023) (2023)
- H. S. Ahn, "Appropriate Emotions for Facial Expressions of 33-DOFs Android Head EveR-4 H33", 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp.1115~1120 (2012)
- H. S. Ahn, "Designing of Android Head System by Applying Facial Muscle Mechanism of Humans", 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012), pp.799~804 (2012)
- H. S. Ahn, "Uses of Facial Expressions of Android Head System according to Gender and Age", 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp.2300~2305 (2012)