Analysis of Research Trends in Deep Learning-Based Video Captioning
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Video captioning technology, as a significant outcome of the integration between computer vision and natural language processing,has emerged as a key research direction in the field of artificial intelligence. This technology aims to achieve automatic understandingand language expression of video content, enabling computers to transform visual information in videos into textual form. This paperprovides an initial analysis of the research trends in deep learning-based video captioning and categorizes them into four main groups:CNN-RNN-based Model, RNN-RNN-based Model, Multimodal-based Model, and Transformer-based Model, and explain the concept ofeach video captioning model. The features, pros and cons were discussed. This paper lists commonly used datasets and performanceevaluation methods in the video captioning field. The dataset encompasses diverse domains and scenarios, offering extensive resourcesfor the training and validation of video captioning models. The model performance evaluation method mentions major evaluation indicatorsand provides practical references for researchers to evaluate model performance from various angles. Finally, as future research tasksfor video captioning, there are major challenges that need to be continuously improved, such as maintaining temporal consistency andaccurate description of dynamic scenes, which increase the complexity in real-world applications, and new tasks that need to be studiedare presented such as temporal relationship modeling and multimodal data integration.
Abstract
Video captioning technology, as a significant outcome of the integration between computer vision and natural language processing,has emerged as a key research direction in the field of artificial intelligence. This technology aims to achieve automatic understandingand language expression of video content, enabling computers to transform visual information in videos into textual form. This paperprovides an initial analysis of the research trends in deep learning-based video captioning and categorizes them into four main groups:CNN-RNN-based Model, RNN-RNN-based Model, Multimodal-based Model, and Transformer-based Model, and explain the concept ofeach video captioning model. The features, pros and cons were discussed. This paper lists commonly used datasets and performanceevaluation methods in the video captioning field. The dataset encompasses diverse domains and scenarios, offering extensive resourcesfor the training and validation of video captioning models. The model performance evaluation method mentions major evaluation indicatorsand provides practical references for researchers to evaluate model performance from various angles. Finally, as future research tasksfor video captioning, there are major challenges that need to be continuously improved, such as maintaining temporal consistency andaccurate description of dynamic scenes, which increase the complexity in real-world applications, and new tasks that need to be studiedare presented such as temporal relationship modeling and multimodal data integration.
참고문헌 (62)
- J. Zhang, "Video captioning with objectaware spatio-temporal correlation and aggregation", IEEE Transactions on Image Processing, 29, pp.6209~6222 (2020)
- B. Pan, "Spatio-temporal graph for video captioning with knowledge distillation", Proceedings of the IEEE, 2020, 10870~10879
- L. Li, "Long short-term relation transformer with global gating for video captioning", IEEE Transactions on Image Processing, 31, pp.2726~2738 (2022)
- L. Yao, "Describing videos by exploiting temporal structure", Proceedings of the IEEE International Conference on Computer Vision, pp.4507~4515 (2015)
- Z. Gan, "Semantic compositional networks for visual captioning", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5630~5639 (2017)
- H. Chen, "A semantics"assisted video captioning model trained with scheduled sampling", Frontiers in Robotics and AI, 7, pp.475767~ (2020)
- B. Wang, "Reconstruction network for video captioning", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.7622~7631 (2018)
- H. Ye, "Hierarchical modular network for video captioning", Proceedings of the IEEE, 2022, 7939~17948
- S. Venugopalan, "Sequence to sequence-video to text", Proceedings of the IEEE International Conference on Computer Vision, pp.4534~4542 (2015)
- C. Szegedy, "Going deeper with convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1~9 (2015)
- K. He, "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- D. Tran, "Learning spatiotemporal features with 3d convolutional networks", Proceedings of the IEEE international Conference on Computer Vision, pp.4489~4497 (2015)
- S. Hochreiter, "Long short-term memory", Neural Computation, 9(8), pp.1735~1780 (1997)
- S. Xie, "Aggregated residual transformations for deep neural networks", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1492~1500 (2017)
- M. Zolfaghari, "Eco : Efficient convolutional network for online video understanding", Proceedings of the European Conference on Computer Vision ECCV, pp.695~712 (2018)
- J. P. Martin, "Improving video captioning with temporal composition of a visualsyntactic embedding", Proceedings of the IEEE, 2021, 3039~3049
- B. K. Horn, "Determining optical flow", Artificial Intelligence, 17(1-3), pp.185~203 (1981)
- J. Donahue, "Long-term recurrent convolutional networks for visual recognition and description", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.2625~2634 (2015)
- P. Pan, "Hierarchical recurrent neural encoder for video representation with application to captioning", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1029~1038 (2016)
- L. Gao, "Video captioning with attention-based lstm and semantic con"sistency", IEEE Transactions on Multimedia, 19(9), pp.2045~2055 (2017)
- C. Szegedy, "Rethinking the inception architecture for computer vision", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.2818~2826 (2016)
- H. Jégou, "Aggregating local descriptors into a compact image representation", 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.3304~3311 (2010), IEEE
- L. Zhou, "End-to-end dense video captioning with masked transformer", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.8739~8748 (2018)
- R. Girshick, "Fast r-cnn", Proceedings of the IEEE International Conference on Computer Vision, pp.1440~1448 (2015)
- T. Wang, "End-to-end dense video captioning with parallel decoding", Proceedings of the IEEE, 2021, 6847~6857
- V. Ramanishka, "Multimodal video description", Proceedings of the 24th ACM International Conference on Multimedia, pp.1092~1096 (2016)
- B. Logan, "Mel frequency cepstral coefficients for music modeling", Ismir. Plymouth, 270, pp.11~ (2000)
- Y. Xu, "Semantic-filtered soft-splitaware video captioning with audio-augmented feature", Neurocomputing, 357, pp.24~35 (2019)
- C. Sun, "Videobert : A joint model for video and language representation learning", Proceedings of the IEEE, 2019, 7464~7473
- S. Xie, "Rethinking spatiotemporal feature learning : Speed-accuracy tradeoffs in video classification", Proceedings of the European Conference on Computer Vision ECCV, pp.305~321 (2018)
- S. Ging, "Coot : Cooperative hierarchical transformer for video-text representation learning", Advances in Neural Information Processing Systems, 33, pp.22605~22618 (2020)
- V. Iashin, "Multi-modal dense video captioning", Proceedings of the IEEE, 2020, 958~959
- A. Vaswani, "Attention is all you need", Advances in Neural Information Processing Systems, 30 (2017)
- J. Carreira, "Quo vadis, action recognition? a new model and the kinetics dataset", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.6299~6308 (2017)
- S. Hershey, "Cnn architectures for large-scale audio classification", 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.131~135 (2017), IEEE
- H. Luo, "Clip4clip : An empirical study of clip for end to end video clip retrieval and captioning", Neuro computing, 508, pp.293~304 (2022)
- A. Radford, "Learning transferable visual models from natural language supervision", International Conference on Machine Learning, PMLR, pp.8748~8763 (2021)
- P. H. Seo, "End-toend generative pretraining for multimodal video captioning", Proceedings of the IEEE, 2022, 17959~17968
- A. Arnab, "Proceedings of the IEEE", CVF International Conference on Computer Vision(ICCV), 2021, 6836~6846
- D. Chen, "Collecting highly parallel data for paraphrase evaluation", Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp.190~200 (2011)
- J. Xu, "Msr-vtt : A large video description dataset for bridging video and language", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.5288~5296 (2016)
- R. Krishna, "Dense-captioning events in videos", Proceedings of the IEEE International Conference on Computer Vision, pp.706~715 (2017)
- G. A. Sigurdsson, "Hollywood in homes : Crowd sourcing data collection for activity understanding", Computer Vision-ECCV 2016: 14th European Conference, pp.510~526 (2016)
- X. Wang, "Vatex : A large-scale, high-quality multilingual dataset for video-and-language research", Proceedings of the IEEE, 2019, 4581~4591
- L. Zhou, "Towards automatic learning of procedures from web instructional videos", AAAI Conference on Artificial Intelligence, pp.7590~7598 (2018)
- A. Rohrbach, "A dataset for movie description", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.3202~3212 (2015)
- S. Pini, "M-vad names : a dataset for video captioning with naming", Multimedia Tools and Applications, 78, pp.14007~14027 (2019)
- K. Papineni, "Bleu : a method for automatic evaluation of machine translation", Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp.311~318 (2002)
- S. Banerjee, "Meteor : An automatic metric for mt evaluation with improved correlation with human judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and, 2005, 65~72
- R. Vedantam, "Cider : Consensus-based image description evaluation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.4566~4575 (2015)
- C. Lin, "Rouge : A package for automatic evaluation of summaries", Text Summarization Branches Out, pp.74~81 (2004)
- A. Rohrbach, "Coherent multi-sentence video description with variable level of detail", Pattern Recognition:36th German Conference, GCPR 2014, pp.184~195 (2014), Springer
- C. Szegedy, "Inceptionv4, inception-resnet and the impact of residual connections on learning", Proceedings of the AAAI Conference on Artificial Intelligence, 31(1) (2017)