Lip and Voice Synchronization Using Visual Attention
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study explores lip-sync detection, focusing on the synchronization between lip movements and voices in videos. Typically, lip-syncdetection techniques involve cropping the facial area of a given video, utilizing the lower half of the cropped box as input for the visualencoder to extract visual features. To enhance the emphasis on the articulatory region of lips for more accurate lip-sync detection, wepropose utilizing a pre-trained visual attention-based encoder. The Visual Transformer Pooling (VTP) module is employed as the visualencoder, originally designed for the lip-reading task, predicting the script based solely on visual information without audio. Ourexperimental results demonstrate that, despite having fewer learning parameters, our proposed method outperforms the latest model,VocaList, on the LRS2 dataset, achieving a lip-sync detection accuracy of 94.5% based on five context frames. Moreover, our approachexhibits an approximately 8% superiority over VocaList in lip-sync detection accuracy, even on an untrained dataset, Acappella.
Abstract
This study explores lip-sync detection, focusing on the synchronization between lip movements and voices in videos. Typically, lip-syncdetection techniques involve cropping the facial area of a given video, utilizing the lower half of the cropped box as input for the visualencoder to extract visual features. To enhance the emphasis on the articulatory region of lips for more accurate lip-sync detection, wepropose utilizing a pre-trained visual attention-based encoder. The Visual Transformer Pooling (VTP) module is employed as the visualencoder, originally designed for the lip-reading task, predicting the script based solely on visual information without audio. Ourexperimental results demonstrate that, despite having fewer learning parameters, our proposed method outperforms the latest model,VocaList, on the LRS2 dataset, achieving a lip-sync detection accuracy of 94.5% based on five context frames. Moreover, our approachexhibits an approximately 8% superiority over VocaList in lip-sync detection accuracy, even on an untrained dataset, Acappella.
참고문헌 (21)
- Y. Shalev, "End to end lip synchronization with a temporal autoencoder", Proceedings of the IEEE, 2020, ~
- K. R. Prajwal, "A lip sync expert is all you need for speech to lip generation in the wild", Proceedings of the 28th ACM International Conference on Multimedia, pp.484~492 (2020)
- P. Ma, "End-to-end audio-visual speech recognition with conformers", ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.7613~7617 (2021), IEEE
- T. Makino, "Recurrent neural network transducer for audio-visual speech recognition", 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.905~912 (2019), IEEE
- A. Katharopoulos, "Transformers are rnns : Fas autoregressive transformers with linear attention", International Conference on Machine Learning, pp.5156~5165 (2020), PMLR
- V. S. Kadandale, "VocaLiST : An audio-visual synchronisation model for lips and voices", Interspeech, pp.3128~3132 (2022)
- J. S. Chung, "Lip Reading Sentences in the Wild", IEEE Conference on Computer Vision and Pattern Recognition (2017)
- S. Chopra, "Learning a similarity metric discriminatively, with application to face verification", 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 1, pp.539~546 (2005), IEEE
- B. V. Mahavidyalaya, "Phoneme and viseme based approach for lip synchronization", International Journal of Signal Processing, Image Processing and Pattern Recognition, 7(3), pp.385~394 (2014)
- J. S. Chung, "Out of time : automated lip sync in the wild", Workshop on Multi-view Lipreading (2016), ACCV
- Y. J. Kim, "Endto-end lip synchronisation based onpattern classification", 2021 IEEE Spoken Language Technology Workshop (SLT), pp.598~605 (2021), IEEE
- A. Bulat, "How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)", International Conference on Computer Vision (2017)
- Y. Zhang, "Can we read speech beyond the lips? rethinking roi selection for deep visual speech recognition", 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp.356~363 (2020), IEEE
- K. R. Prajwal, "Sub-word level lip reading with visual attention", Proceedings of the IEEE, 2022, 5162~5172
- A. Katharopoulos, "Transformers are rnns : Fast autoregressive transformers with linear attention", International Conference on Machine Learning, pp.5156~5165 (2020), PMLR
- Y. -H. H. Tsai, "Multimodal transformer for unaligned multimodal language sequences", Proceedings of the Conference Association for Computational Linguistics, Meeting, NIH Public Access, pp.6558~ (2019)
- J. F. Montesinos, "Acappella : audio-visual singing voice separation", 32nd British Machine Vision Conference, BMVC 2021 (2021)
- S. -W. Chung, "Perfect match : Improved cross-modal embeddings for audio-visual synchronisation", ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.3965~3969 (2019), IEEE
- H. Gupta, "Perceptual synchronization scoring of dubbed content using phoneme-viseme agreement", Proceedings of the IEEE, 2024, ~