Image-Text Sentiment Analysis Model Based on Visual Aspect Attention
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Social network has become an integral part of our daily life. Sentiment analysis of social media information is helpful to understand people's views, attitudes and emotions on social networking sites. Traditional sentiment analysis mainly relies on text. With the rise of smart phones, information on the network is gradually diversified, including not only text, but also images. It is found that, in many cases, images can enhance the text rather than express emotions independently. We propose a novel image text sentiment analysis model (LSTM-VAA). Specifically, this model does not take the picture information as the direct input, but uses the VGG16 network to extract the image features, and then generates the visual aspect attention, and gives the core sentences in the document a higher weight, and get a document representation based on the visual aspect attention. In addition, we use the LSTM network to extract the text sentiment and get the document representation based on text only. Finally, we fuse the two groups of classification results to obtain the final classification label. On the yelp restaurant reviews data set, our model achieves an accuracy of 62.08%, which is 18.92% higher than BiGRU-m VGG, which verifies the effectiveness of using visual information as aspect attention assisted text for emotion classification; It is 0.32% higher than Vista-Net model, which proves that LSTM model can effectively make up for the defect that images in Vista-Net model cannot completely cover text.
Abstract
Social network has become an integral part of our daily life. Sentiment analysis of social media information is helpful to understand people's views, attitudes and emotions on social networking sites. Traditional sentiment analysis mainly relies on text. With the rise of smart phones, information on the network is gradually diversified, including not only text, but also images. It is found that, in many cases, images can enhance the text rather than express emotions independently. We propose a novel image text sentiment analysis model (LSTM-VAA). Specifically, this model does not take the picture information as the direct input, but uses the VGG16 network to extract the image features, and then generates the visual aspect attention, and gives the core sentences in the document a higher weight, and get a document representation based on the visual aspect attention. In addition, we use the LSTM network to extract the text sentiment and get the document representation based on text only. Finally, we fuse the two groups of classification results to obtain the final classification label. On the yelp restaurant reviews data set, our model achieves an accuracy of 62.08%, which is 18.92% higher than BiGRU-m VGG, which verifies the effectiveness of using visual information as aspect attention assisted text for emotion classification; It is 0.32% higher than Vista-Net model, which proves that LSTM model can effectively make up for the defect that images in Vista-Net model cannot completely cover text.
참고문헌 (29)
- LI X, "News impact on stock price return via sentiment analysis", Proceedings of the Knowledge Based System, 69, pp.14~23 (2014)
- KAGAN V, "Using twitter sentiment to forecast the 2013 Pakistani election and the 2014 Indian election", Proceeding of the IEEE Intelligent Systems, 30(1), pp.2~5 (2014)[DOI]
- YADAV S, "Medical sentiment analysis using social media:towards building a patient assisted system", Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)., pp.2790~2797 (2018)
- TRUONG T Q, "VistaNet: Visual Aspect Attention Network for Multimodal Sentiment analysis", Proceedings of The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-2019)., pp.305~312 (2019)
- YU D, "Multilevel attention networks for visual question answering", Proceedings of the Computer Vision and Pattern Recognition (CVPR)., pp.4187~4195 (2017)
- PORIA S CHATURVEDI I CAMBRIA, "Convolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis", Proceedings of 2016 IEEE 16th International Conference on Data Mining (ICDM), pp.439~448 (2017)
- XU N, "Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis", Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), pp.371~378 (2019)
- CHAN S W K, "Sentiment analysis in financial texts", Proceedings of the Decision Support Systems, 94(2017), pp.53~64 (2017)
- SEVERYN A MOSCHITTI A, "Twitter sentiment analysis with deep convolutional neural networks", Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.959~962 (2015)
- LAI SXU L, "Recurrent convolutional neural networks for text classification", Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 333, pp.2267~2273 (2015)
- NGUYEN T, "A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips", Proceedings of the Neural Networks, 118, pp.208~219 (2019)
- BAHDANAU D, "BENGIO Y-Neural machine translation by jointly learning to align and translate", arXiv preprint arXiv:1409.0473, ~ (2014)
- YANG Z, "Hierarchical attention networks for document classification", Proceedings of North American Chapter of the Association for Computational Linguistics (HLTNAACL)., pp.1480~1489 (2016)
- TANG D, "Aspect level sentiment classification with deep memory network", Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.214~224 (2016)
- TRUONG Q, "Visual sentiment analysis for review images with item-oriented and user-oriented CNN", Proceedings of the 25th ACM international conference on Multimedia, pp.1274~1282 (2017)
- LI M, "Long-tail Hashtag Recommendation for Micro-videos with Graph Convolutional Network", Proceedings of the 28th ACM International Conference on Information and Knowledge Managemen, pp.509~518 (2019)
- SIERSDORFER S, "Analyzing and predicting sentiment of images on the social web", Proceedings of the 18th ACM international conference on Multimedia., pp.715~718 (2010)
- YOU QLUO, "Robust image sentiment analysis using progressively trained and domain transferred deep networks", Proceedings In AAAI, pp.381~388 (2015)
- BORTH D, JI R, "Large-scale visual sentiment ontology and detectors using adjective noun pairs", Proceedings of the 21st ACM international conference on Multimedia, pp.223~232 (2013)
- XU N, "Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis", Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), pp.371~378 (2019)
- YOU Q, "Visual sentiment analysis by attending on local image regions", Proceedings of the 31st AAAI Conference on Artificial Intelligence, pp.231~237 (2017)
- XU N, "Multi-SentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis", Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp.2399~2402 (2017)
- PENNINGTON J, "Glove: Global vectors for word representation", Proceedings of the Conference on Empirical Methods in Natural Language processing, pp.1532~1543 (2014)
- CHO KVAN MERRIENBOER B GULCEHRE C, "Learning phrase representations using rnn encoder-decoder for statistical machine translation", Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2014)., pp.1724~1734 (2014)
- SIMONYAN K, "Very deep convolutional networks for large-scale image recognition", Proceedings of the International Conference on Learning Representations (2015)
- YUE W, "Multi-Modality Image Manipulation Detection", Proceedings of the IEEE, 2019, 9543~9552
- TANG D, "Document modeling with gated recurrent neural network for sentiment classification", Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)., pp.1422~1432 (2015)
- 왕지소, "Adjusting the Difficulty of Running Game with Facial Expression Recognition Technology Using Convolutional Neural Network", 한국컴퓨터게임학회논문지, 31(2), pp.39~46 (2018), (사)한국컴퓨터게임학회[DOI]
- 강호성, "Analysis and Improvement Plans on Visualization Systems for Word Analysis: Focused on Word2Vec", 차세대컨버전스정보서비스기술논문지, 7(2), pp.177~186 (2018), 차세대컨버전스정보서비스학회[DOI]