Description Region Expansion-based relationship-oriented dense image captioning model
2
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
In dense image captioning, multiple sentences are generated by identifying the various contexts present in an image through the use of an object detection network to extract the regions of interest (ROIs) at which objects exist in the image. However, because the size of ROIs obtained from the object detection network is dependent on the size of the target object, each ROI is limited to containing only fragmentary information about the target object. Therefore, in this paper, we propose Description Region Expansion (DRE), in which clusters of ROIs are extracted from the detection network and combined into single ROIs. DRE is located behind the detection network. The expansion of the ROIs allows the relationships between object clusters rather than single objects to be expressed in dense image captioning. To verify the validity of the model, training and evaluation were conducted using the Visual Genome dataset. The BELU-1 score of 0.678 and Meteor score of 0.463 achieved by the model confirms its excellent performance.
Abstract
In dense image captioning, multiple sentences are generated by identifying the various contexts present in an image through the use of an object detection network to extract the regions of interest (ROIs) at which objects exist in the image. However, because the size of ROIs obtained from the object detection network is dependent on the size of the target object, each ROI is limited to containing only fragmentary information about the target object. Therefore, in this paper, we propose Description Region Expansion (DRE), in which clusters of ROIs are extracted from the detection network and combined into single ROIs. DRE is located behind the detection network. The expansion of the ROIs allows the relationships between object clusters rather than single objects to be expressed in dense image captioning. To verify the validity of the model, training and evaluation were conducted using the Visual Genome dataset. The BELU-1 score of 0.678 and Meteor score of 0.463 achieved by the model confirms its excellent performance.
참고문헌 (23)
- S. Antol, "VQA : Visual question answering", Proceedings of the 2015 IEEE international conference on computer vision, pp.2425~2433 (2015)
- J. Johnson, "Densecap : Fully convolutional localization networks for dense captioning", Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, pp.4565~4574 (2016)
- A. Farhadi, "Every picture tells a story : Generating sentences from images", Euro-pean conference on computer vision, pp.15~29 (2010), Springer
- Y. Yang, "Cor-pus-guided sentence generation of natural images", Pro-ceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp.444~454 (2011)
- S. Li, "Com-posing simple image descriptions using web-scale n-grams", Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pp.220~228 (2011)
- O. Vinyals, "Show and tell : A neural image caption generator", Proceedings of the IEEE conference on computer vision and pattern recog-nition, pp.3156~3164 (2015)
- K. Xu, "Show, attend and tell : Neural image caption generation with visual attention", International conference on machine learning, Proceedings of Machine Learning Research (PMLR), pp.2048~2057 (2015)
- Q. You, "Image cap-tioning with semantic attention", Proceedings of the IEEE conference on computer vision and pattern recogni-tion, pp.4651~4659 (2016)
- L. Yang, "Dense captioning with joint inference and visual context", Proceedings of the 2017 IEEE conference on computer vision and pattern recognition, pp.2193~2202 (2017)
- X. Li, "Learning object context for dense captioning", Proceedings of the AAAI conference on artificial intelligence, 33(1), pp.8650~8657 (2019)
- X. He, "Image caption generation with part of speech guidance", Pattern Recognition Letters, 119, pp.229~237 (2019)
- K. Simonyan, "Very deep convolutional networks for large-scale image recognition", arXiv preprint arXiv:1409.1556, ~ (2014)
- K. He, "Deep residual learn-ing for image recognition", Proceedings of the IEEE con-ference on computer vision and pattern recognition, pp.770~778 (2016)
- C. Szegedy, "Rethinking the inception architecture for computer vision", Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, pp.2818~2826 (2016)
- C. Szegedy, "In-ception-v4, inception-resnet and the impact of residual con-nections on learning", Thirty-first AAAI conference on artificial intelligence (2017)
- J. Redmon, "You only look once : Unified, real-time object detection", Pro-ceedings of the IEEE conference on computer vision and pattern recognition, pp.779~788 (2016)
- J. Redmon, "Yolov3 : An incremental im-provement", arXiv preprint arXiv : 1804. 02767, ~ (2018)
- S. Ren, "Faster r-cnn : To-wards real-time object detection with region proposal net-works", Advances in neural information processing sys-tems, 28, pp.91~99 (2015)
- R. Krishna, "Visual genome : Connecting language and vision using crowdsourced dense image annotations", Inter-national journal of computer vision (2016)
- K. Papineni, "BLEU : A method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the Associa-tion for Computational Linguistics, pp.311~318 (2002)
- S. Banerjee, "METEOR : An automatic metric for MT evaluation with improved correlation with human judgments", Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and, 2005, 65~72
- S. Chen, "Say as you wish : Fine-grained control of image caption generation with ab-stract scene graph", Proceedings of the IEEE, 2020, 9962~9971
- G. Yin, "Context and attribute grounded dense captioning", Pro-ceedings of the IEEE, 2019, 6241~6250