Target-centered context-detection technique using dual R-CNN
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Image captioning, which aims to understand the context of an image, generates natural language using object feature vectors. This results in verbose sentences because they usually contain modifiers and objects. However, unlike human descriptions, the infor-mation needed in natural environments must be simplified. Therefore, we propose a target-centered context-detection model that uses dual R-CNN to generate short sentences about subject-centered attributes to describe the behavior of the target. The proposed model consists of target context detection (TCD), which detects subjects and actions, and activity image caption, which generates sentences centered on actions. The proposed TCD uses two RCNN heads to estimate objects and their properties. In this process, we added target-description region expansion to filter out unnecessary objects and encompass surrounding information. Afterward, the proposed activity image caption combines the feature vector and attributes of each target to generate a short description of the target-attribute pair. The proposed model can effectively convey information using brief rather than long sentences.
Abstract
Image captioning, which aims to understand the context of an image, generates natural language using object feature vectors. This results in verbose sentences because they usually contain modifiers and objects. However, unlike human descriptions, the infor-mation needed in natural environments must be simplified. Therefore, we propose a target-centered context-detection model that uses dual R-CNN to generate short sentences about subject-centered attributes to describe the behavior of the target. The proposed model consists of target context detection (TCD), which detects subjects and actions, and activity image caption, which generates sentences centered on actions. The proposed TCD uses two RCNN heads to estimate objects and their properties. In this process, we added target-description region expansion to filter out unnecessary objects and encompass surrounding information. Afterward, the proposed activity image caption combines the feature vector and attributes of each target to generate a short description of the target-attribute pair. The proposed model can effectively convey information using brief rather than long sentences.
참고문헌 (20)
- O. Vinyals, "Show and tell: A neural image caption generator", Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.3156~3164 (2015)
- J. Johnson, "Densecap: Fully convolutional localization networks for dense captioning", Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp.4565~4574 (2016)
- L. Yang, "Dense captioning with joint inference and visual context", Proceedings of the IEEE conference on Computer Vision and Pattern Recogni-tion, pp.2193~2202 (2017)
- X. Li, "Learning object context for dense captioning", Proceedings of the AAAI conference on Artificial Intelligence, 33(01), pp.8650~8657 (2019)
- 신영재, "Description Region Expansion-based relationship-oriented dense image captioning model", 한국마린엔지니어링학회지, 45(6), pp.434~441 (2021), 한국마린엔지니어링학회[DOI]
- S. Ren, "Faster r-cnn: To-wards real-time object detection with region proposal net-works", Advances in neural information processing sys-tems, 28, pp.91~99 (2015)
- Z. Li, "Deep people counting with faster R-CNN and correla-tion tracking", Proceedings of the International Confer-ence on Internet Multimedia Computing and Service, pp.57~60 (2016)
- F. Cui, "Automatic recogni-tion and tracking of highway layer-interface using Faster R-CNN", Journal of Applied Geophysics, 196, pp.104477~ (2022)
- M. Othmani, "A vehicle detection and tracking method for traffic video based on faster R-CNN", Multimedia Tools and Applications, 81(20), pp.28347~28365 (2022)
- T. Yao, "Exploring visual rela-tionship for image captioning", Proceedings of the Euro-pean Conference on Computer Vision, pp.684~699 (2018)
- X. Yang, "Reformer: The relational transformer for image captioning", Proceedings of the 30th ACM International Conference on Multimedia, pp.5398~5406 (2022)
- J. Yu, "Multimodal transformer with multi-view visual representation for image caption-ing", IEEE Transactions on Circuits and Systems for Video Technology, 30(12), pp.4467~4480 (2019)
- Y. Nagaoka, "Text detection by faster R-CNN with multiple region proposal networks", Proceedings of 2017 14th IAPR International Conference on Document Analysis and Recognition, 6, pp.15~20 (2017)
- J. Yang, "Handwriting text recogni-tion based on faster R-CNN", Proceedings of 2019 Chi-nese Automation Congress, pp.2450~2454 (2019)
- N. P. Ap, "Automatic number plate detection in vehicles using faster R-CNN", Proceedings of 2020 International Conference on System, Computation, Automation and Net-working, pp.1~6 (2020)
- R. Krishna, "Visual genome: Connecting language and vision using crowdsourced dense image annotations", Inter-national Journal of Computer Vision, 123, pp.32~73 (2017)
- K. Papineni, "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th Annual Meeting of the Associa-tion for Computational Linguistics, pp.311~318 (2002)
- S. Banerjee, "METEOR: An automatic metric for MT evaluation with improved correlation with human judgments", Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And, 2005, 65~72
- G. Yin, "Context and attribute grounded dense captioning", Pro-ceedings of the IEEE, 2019, 6241~6250
- S. Chen, "Say as you wish: Fine-grained control of image caption generation with ab-stract scene graphs", Proceedings of the IEEE, 2020, 9959~9968