Korean Text to Gloss: Self-Supervised Learning approach
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Natural Language Processing (NLP) has grown tremendously in recent years. Typically, bilingual, and multilingual translation models have been deployed widely in machine translation and gained vast attention from the research community. On the contrary, few studies have focused on translating between spoken and sign languages, especially non-English languages. Prior works on Sign Language Translation (SLT) have shown that a mid-level sign gloss representation enhances translation performance. Therefore, this study presents a new large-scale Korean sign language dataset, the Museum-Commentary Korean Sign Gloss (MCKSG) dataset, including 3828 pairs of Korean sentences and their corresponding sign glosses used in Museum-Commentary contexts. In addition, we propose a translation framework based on self-supervised learning, where the pretext task is a text-to-text from a Korean sentence to its back-translation versions, then the pre-trained network will be fine-tuned on the MCKSG dataset. Using self-supervised learning help to overcome the drawback of a shortage of sign language data. Through experimental results, our proposed model outperforms a baseline BERT model by 6.22%.
Abstract
Natural Language Processing (NLP) has grown tremendously in recent years. Typically, bilingual, and multilingual translation models have been deployed widely in machine translation and gained vast attention from the research community. On the contrary, few studies have focused on translating between spoken and sign languages, especially non-English languages. Prior works on Sign Language Translation (SLT) have shown that a mid-level sign gloss representation enhances translation performance. Therefore, this study presents a new large-scale Korean sign language dataset, the Museum-Commentary Korean Sign Gloss (MCKSG) dataset, including 3828 pairs of Korean sentences and their corresponding sign glosses used in Museum-Commentary contexts. In addition, we propose a translation framework based on self-supervised learning, where the pretext task is a text-to-text from a Korean sentence to its back-translation versions, then the pre-trained network will be fine-tuned on the MCKSG dataset. Using self-supervised learning help to overcome the drawback of a shortage of sign language data. Through experimental results, our proposed model outperforms a baseline BERT model by 6.22%.
참고문헌 (31)
- C. Won, "Open Korean Corpora: A Practical Report", Proceedings of Second Workshop for NLP Open Source Software (2020)
- C. Necati Cihan, "Neural Sign Language Translation", IEEE, 2018, ~
- L. Dongxu, "Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison", Proceedings of the IEEE, 2020, ~
- S. OZGE MERCANOGLU, "AUTSL : A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods", IEEE Access, 8, pp.181340~181355 (2020)
- K. Sang-Ki, "Neural Sign Language Translation Based on Human Keypoint Estimation", Applied sciences, 9(13), pp.2683~ (2019)
- R. Razieh, "Sign Language Production: A Review", Proceedings of the IEEE, 2021, ~
- S. Z. Gurbuz, "American sign language recognition using rf sensing", IEEE Sensors Journal, 21(3), pp.3763~3775 (2020)
- R. Razieh, "Sign Language Recognition : A Deep Survey", Expert Systems With Applications, 164, pp.113794~ (2021)
- C. C. Necati, "Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation", Proceedings of the IEEE, 2020, ~
- Z. Hao, "Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition", Proceedings of the AAAI Conference on Artificial Intelligence (2020)
- K. Yin, "Better Sign Language Translation with STMC-Transformer", Proceedings of the 28th International Conference on Computational Linguistics (2020)
- B. Saunders, "Progressive transformers for end-to-end sign language production", European Conference on Computer Visio (2020)
- L. Ventura, "Can everybody sign now? Exploring sign language video generation from 2D poses" (2020)
- S. Stoll, "Text2Sign : towards sign language production using neural machine translation and generative adversarial networks", International Journal of Computer Vision, 128(4), pp.891~908 (2020)
- J. Zelinka, "Neural sign language synthesis: Words are our glosses", Proceedings of the IEEE, 2020, ~
- C. Chan, "Everybody dance now", Proceedings of the IEEE, 2019, ~
- Z. Cao, "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields", IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)
- E. Park, "KoNLPy: Korean natural language processing in Python", Proceedings of the 26th Annual Conference on Human & Cognitive Language Technology (2014)
- T. Kudo, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" (2018)
- A. Vaswani, "Attention is all you need", Advances in neural information processing systems (2017)
- J. Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding" (2018)
- Y. Liu, "Roberta: A robustly optimized bert pretraining approach" (2019)
- X. Song, "Fast wordpiece tokenization" (2020)
- L. Sangah, "KR-BERT: A Small-Scale Korean-Specific Language Model" (2020)
- L. Hyunjae, "KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding"
- S. Edunov, "Understanding back-translation at scale" (2018)
- D. T. Vu, "Text Data Augmentation for the Korean Language", Applied Sciences, 12(7), pp.3425~ (2022)
- Q. Xie, "Unsupervised data augmentation for consistency training", Advances in Neural Information Processing Systems, 33, pp.6256~6268 (2020)
- M. Johnson, "Transactions of the Association for Computational Linguistics" (2017)
- N. Reimers, "Making monolingual sentence embeddings multilingual using knowledge distillation" (2020)
- B. Ban, "A Survey on Awesome Korean NLP Datasets" (2021)