Korean visual abductive reasoning: AI Language Model’s ability to understand plausibility
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Visual abductive reasoning is the logical process of drawing the most plausible hypothesis based on given observations. This ability is fundamental to artificial intelligence because it enables inference from incomplete information. However, little research has been conducted on Korean visual abductive reasoning. To examine the capability of a multimodal language model’s Korean visual abductive reasoning, we set a simple baseline model and analyzed how it numerically estimated the plausibility for all Korean hypothesis sentences through a multiple-choice task. This task was implemented using a simple dual encoder model and the Korean Story Cloze dataset. After fine-tuning with the binary-choice task discriminating the plausible hypothesis from the implausible one, our baseline model shows an accuracy of 79.81%. In multiple-choice task designed to check for the influence of overfitting or annotation artifacts, the model estimated the plausibilities of four options in the order of Groundtruth≃ Plausible>Implausible≫Random. We also conducted experiments to analyze how the model performed Korean visual abductive reasoning. It was observed that the model made little use of the observation before the hypothesis but demonstrated a similar tendency to humans, struggling with data samples which humans also struggle with when evaluating the plausibility of given sentences. Our study sets a research foundation for numerically analyzing and understanding the language models’ visual abductive reasoning ability in the Korean context. It also shows both the potential and limitations of the language model’s Korean visual abductive reasoning ability and provides clues for future research directions.
Abstract
Visual abductive reasoning is the logical process of drawing the most plausible hypothesis based on given observations. This ability is fundamental to artificial intelligence because it enables inference from incomplete information. However, little research has been conducted on Korean visual abductive reasoning. To examine the capability of a multimodal language model’s Korean visual abductive reasoning, we set a simple baseline model and analyzed how it numerically estimated the plausibility for all Korean hypothesis sentences through a multiple-choice task. This task was implemented using a simple dual encoder model and the Korean Story Cloze dataset. After fine-tuning with the binary-choice task discriminating the plausible hypothesis from the implausible one, our baseline model shows an accuracy of 79.81%. In multiple-choice task designed to check for the influence of overfitting or annotation artifacts, the model estimated the plausibilities of four options in the order of Groundtruth≃ Plausible>Implausible≫Random. We also conducted experiments to analyze how the model performed Korean visual abductive reasoning. It was observed that the model made little use of the observation before the hypothesis but demonstrated a similar tendency to humans, struggling with data samples which humans also struggle with when evaluating the plausibility of given sentences. Our study sets a research foundation for numerically analyzing and understanding the language models’ visual abductive reasoning ability in the Korean context. It also shows both the potential and limitations of the language model’s Korean visual abductive reasoning ability and provides clues for future research directions.
참고문헌 (24)
- Bhagavatula, Chandra, "Abductive commonsense reasoning", Paper presented at 2020 International Conference on Learning Representations (ICLR 2020) (2020)
- Choi, Yejin, "The curious case of commonsense intelligence", Daedalus, 151(2), pp.139~155 (2022)
- Dosovitskiy, Alexey, "An image is worth 16x16 words : Transformers for image recognition at scale", Paper presented at 2021 International Conference on Learning Representations (ICLR 2021) (2021)
- Gururangan, Suchin, "Annotation artifacts in natural language inference data", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2, pp.107~112 (2018), Association for Computational Linguistics
- Hessel, Jack, "The abduction of Sherlock Holmes: A dataset for visual abductive reasoning", European Conference on Computer Vision, pp.558~575 (2022), Springer Nature Switzerland
- Lee, Jaewook, "Ko-ATOMIC : Korean commonsense knowledge graph", Annual Conference on Human and Language Technology, pp.412~417 (2022), Human and Language Technology
- Liang, Chen, "Visual abductive reasoning", Proceedings of the IEEE, 2022, 15565~15575[DOI]
- Lin, Bill Yuchen, "CommonGen : A constrained text generation challenge for generative commonsense reasoning", Findings of the Association for Computational Linguistics: EMNLP 2020, pp.1823~1840 (2019), Association for Computational Linguistics
- Liu, Jiacheng, "Rainier : Reinforced knowledge introspector for commonsense question answering", Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.8938~8958 (2022)
- Mostafazadeh, Nasrin, "A corpus and evaluation framework for deeper understanding of commonsense stories", Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, pp.839~849 (2016), Association for Computational Linguistics
- Ovchinnikova, Ekaterina, "Abductive reasoning with a large knowledge base for discourse processing", Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011), pp.225~234 (2011), Association for Computational Linguistics
- Park, Sungjoon, "KLUE: Korean language understanding evaluation", Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. NeurlPS, 1 (2021)
- Peirce, Charles S., "Philosophical writings of Peirce" (1955), Dover Publications
- Rajani, Nazneen Fatema, "Explain yourself! Leveraging language models for commonsense reasoning", Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.4932~4942 (2019), Association for Computational Linguistics
- Sap, Maarten, "Atomic : An atlas of machine commonsense for if-then reasoning", Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), pp.3027~3035 (2019), The Association for the Advancement of Artificial Intelligence
- Seo, Jaehyung, "Kommongen : A dataset for Korean generative commonsense reasoning evaluation", Annual Conference on Human and Language Technology, pp.55~60 (2021), Human and Language Technology
- Seo, Jaehyung, "A dog is passing over the jet? A text-generation dataset for Korean commonsense reasoning and evaluation", Findings of the Association for Computational Linguistics: NAACL 2022, pp.2233~2249 (2022), Association for Computational Linguistics
- Song, Sanghoun, "How AI language models judge common sense", Journal of Korean Linguistics, 105(0), pp.375~400 (2023)
- Talmor, Alon, "CommonsenseQA: A question answering challenge targeting commonsense knowledge", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, pp.4149~4158 (2019), Association for Computational Linguistics
- Viola, Tullio, "Pragmatism and embodied cognitive science: From bodily intersubjectivity to symbolic articulation", pp.251~268 (2016), De Gruyter
- Wang, Alex, "GLUE : A multi-task benchmark and analysis platform for natural language understanding", Proceedings of the 2018 EMNLP Workshop BlackboxNLP : Analyzing and Interpreting Neural Networks for NLP, pp.353~355 (2018), Association for Computational Linguistics
- Zellers, Rowan, "SWAG : A large-scale adversarial dataset for grounded commonsense inference", Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.93~104 (2018), Association for Computational Linguistics