객체 검출 기반 이미지 캡션 알고리즘 연구
A study on image caption algorithm based on object detection
0
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
어떠한 이미지에서 검출된 객체간의 상호 연관관계를 이해하는 기술인 이미지 캡션은 향후 자동 감시 시스템에 필수적이기 때문에 컴퓨터 비전 및 자연언어 처리 분야에서 많은 연구가 진행되고 있다. 이미지 전체를 학습하는 기존의 이미지캡션 방식은 각 이미지 영역의 부분적인 상황 이해가 힘들며 동시에 새로운 이벤트 발생에 취약하다. 본 논문에서는 이를해결하기 위해 검출된 객체의 경계박스 면적과 객체간의 거리를 이용하여 객체 정보를 생성하는 CNN(Convolutional Neural Network)기반의 객체 검출단과 RNN(Recurrent Neural Network) 기반의 문장 생성단을 결합하여 이미지의 세부적인 문장을 생성하는 모델을 제안한다. 제안한 방식은 CNN 기반의 객체 검출단의 느린 처리속도를 향상시키기 위하여 YOLO(You Only Look Once) 네트워크의 Grid 방식을 적용하였으며, 객체 검출단과 문장 생성단으로 나누어진 두 네트워크를 하나의 인공 신경망 모델로 구성하여 객체 검출 기반 이미지 캡션이 가능하다. 제안한 모델은 캡션 데이터세트인 Flickr 8K, Flickr 30K, MS COCO를 이용하여 학습하였고 BLEU(BiLingual Evaluation Understudy) 스코어 방식을 사용하여 캡션 성능을 검증하였다.
Abstract
Image caption, which is a method to understand the interrelationships between objects detected in a certain image, is being actively studied in the fields of computer vision and natural language processing, because it is expected to be an essential part of the future automatic monitoring systems. The conventional image caption method, which learns the whole image, finds it difficult to understand partial situations in different image regions, and is vulnerable in the case of the occurrence of a new event at the same time. In order to solve these problems, this paper presents a new model that generates detailed sentences of images by combining a convolutional neural network (CNN)-based object detection step, which generates object information using the bounding box area of the detected object and the distance between the objects, and a sentence generation step based on recurrent neural networks . The proposed model adopts the grid method of the YOLO (You Only Look Once) network to improve the slow processing speed of the CNN-based object detection step. It is possible by the image caption based on object detection because the two network models involved in the object detection and sentence generation steps are composed of one artificial neural network model. The proposed model learns using the caption data sets Flickr 8K, Flickr 30K, and MS COCO, and the caption performance can be verified using the BLEU (BiLingual Evaluation Understudy) score method.
참고문헌 (15)
- O. Vinyals, "Show and tell: A neural image caption generator", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.3156~3164 (2015)
- 이창기, "Recurrent Neural Network를 이용한 이미지 캡션 생성", 정보과학회논문지, 43(08), pp.878~882 (2016), 한국정보과학회
- A. Karpathy, "Deep visual-semantic alignments for generating image descriptions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.3128~3137 (2015)
- C. Liu, "MAT: A multimodal attentive translator for image captioning", arXiv preprint arXiv:1702.05658, ~ (2017)
- K. Cho, "Learning phrase representations using RNN encoder-decoder for statistical machine translation", arXive preprint arXive:1406.1078, ~ (2014)
- K. Simonyan, "Very deep convolutional networks for large-scale image recognition", arXiv preprint arXiv:1409.1556, ~ (2014)
- C. Szegedy, "Going deeper with convolutions", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1~9 (2015)
- K. He, "Deep residual learning for image recognition", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.770~778 (2016)
- S. H. Song, "A Pedestrian Detection Method using Deep Neural Network", Journal of Korea Information Science Society, 44(1), pp.44~50 (2017)
- J. Redmon, "You only look once: Unified, Real-Time Object Detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.779~788 (2016)
- M. Hodosh, "Framing image description as a ranking task: Data, models and evaluation metrics", Journal of Artificial Intelligence Research, 47, pp.853~899 (2013)
- J. Mao, "Deep captioning with multimodal recurrent neural networks (m-rnn)", arXiv preprint arXiv:141236632, ~ (2014)
- K. Xu, "Show, attend and tell: Neural image caption generation with visual attention", International Conference on Machine Learning, pp.2048~2057 (2015)
- T. Lin, "Microsoft coco: Common objects in context", arXiv:1405.0312v3, ~ (2015)
- K. Papineni, "BLEU: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting on association for computational linguistics, pp.311~318 (2002)