Activity-oriented visual relationship detection technique for context recognition
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Context recognition is a technology that acquires information about events that occur based on information on various objects appearing in images. To implement this, dense image captioning, which recognizes all objects in the image, is mainly applied. However, since this approach targets all objects, it provides status information such as location or color, which is relatively less im-portant to humans, and even static object information. To humans, this information is unnecessary because of its low readability. In this paper, to solve this problem, we propose a Context Pair Network that describes only the context of an object based on Visual Relationship Detection (VRD). The proposed model consists of a Pair Object Module (POM) that extracts subjects, objects, and rela-tionships and a Pair Embedding Module (PEM) that creates a subject-predicate-object structure. The proposed POM detects objects corresponding to subjects and objects based on three RCNNs and matches the detected objects with Subject-Object (S-O) pairs. PEM also generates sentences consisting of subject-predicate-object by S-O pairs based on LSTM. Through this, the proposed model is capable of situation awareness that provides only interactions between objects, not conventional captioning that describes all infor-mation indiscriminately.
Abstract
Context recognition is a technology that acquires information about events that occur based on information on various objects appearing in images. To implement this, dense image captioning, which recognizes all objects in the image, is mainly applied. However, since this approach targets all objects, it provides status information such as location or color, which is relatively less im-portant to humans, and even static object information. To humans, this information is unnecessary because of its low readability. In this paper, to solve this problem, we propose a Context Pair Network that describes only the context of an object based on Visual Relationship Detection (VRD). The proposed model consists of a Pair Object Module (POM) that extracts subjects, objects, and rela-tionships and a Pair Embedding Module (PEM) that creates a subject-predicate-object structure. The proposed POM detects objects corresponding to subjects and objects based on three RCNNs and matches the detected objects with Subject-Object (S-O) pairs. PEM also generates sentences consisting of subject-predicate-object by S-O pairs based on LSTM. Through this, the proposed model is capable of situation awareness that provides only interactions between objects, not conventional captioning that describes all infor-mation indiscriminately.
참고문헌 (21)
- J. Johnson, "Densecap: Fully convolutional localization networks for dense captioning", Proceedings of the IEEE conference on computer vision and pattern recognition, pp.4565~4574 (2016)
- A. Farhadi, "Every picture tells a story: Generating sentences from images", European conference on computer vision, pp.15~29 (2010)
- Y. Yang, "Cor-pus-guided sentence generation of natural images", Pro-ceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp.444~454 (2011)
- S. Li, "Com-posing simple image descriptions using web-scale n-grams", Proceedings of the Fifteenth Conference on Com-putational Natural Language Learning, pp.220~228 (2011)
- O. Vinyals, "Show and tell: A neural image caption generator", Proceedings of the IEEE Conference on Computer Vision and Pattern Recog-nition, pp.3156~3164 (2015)
- Q. You, "Image cap-tioning with semantic attention", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.4651~4659 (2016)
- L. Yang, "Dense captioning with joint inference and visual context", Proceedings of the IEEE Conference on Computer Vision and Pattern Recog-nition, pp.1978~1987 (2017)
- X. Li, "Learning object context for dense captioning", Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), pp.8650~8657 (2019)
- K. Simonyan, "Very deep convolutional networks for large-scale image recognition", arXiv preprint arXiv:1409.1556, ~ (2014)
- K. He, "Deep residual learn-ing for image recognition", Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- C. Szegedy, "Rethinking the inception architecture for computer vision", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.2818~2826 (2016)
- C. Szegedy, "In-ception-v4, inception-resnet and the impact of residual con-nections on learning", Thirty-first AAAI Conference on Ar-tificial Intelligence, 31(1) (2017)
- J. Redmon, "You only look once: Unified, real-time object detection", Pro-ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.779~788 (2016)
- J. Redmon, "Yolov3: An incremental im-provement", arXiv preprint arXiv:1804.02767, ~ (2018)
- S. Ren, "Faster r-cnn: To-wards real-time object detection with region proposal net-works", Advances in neural information processing sys-tems, 28, pp.91~99 (2015)
- R. Krishna, "Visual genome: Connecting language and vision using crowdsourced dense image annotations", Inter-national journal of computer vision, 123(1), pp.32~73 (2017)
- C. Lu, "Visual re-lationship detection with language priors", European Con-ference on Computer Vision, pp.852~869 (2016)
- Y. Zhan, "On exploring undeter-mined relationships for visual relationship detection", Pro-ceedings of the IEEE, 2019, 5123~5132
- L. Mi, "Hierarchical graph attention network for visual relationship detection", Proceedings of the IEEE, 2020, 13883~13892
- A. G. Howard, "Mobilenets: Efficient convolu-tional neural networks for mobile vision applications", arXiv preprint arXiv : 1704. 04861, ~ (2017)
- G. Yin, "Context and attribute grounded dense captioning", Pro-ceedings of the IEEE, 2019, 6234~6243