Reasoning Ability of Deep Learning Models on Tautological Expressions in Korean
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
The purpose of this paper is to evaluate the abductive reasoning capability of deep learning-based language models, with a specific focus on their ability to infer the meaning of tautological expressions. To achieve this, we conducted two experiments that required language models to understand the meaning of tautological expressions in different contexts. The first experiment was a binary classification task in which the models were presented with preceding sentences as contexts along with a tautological expression, and evaluated the adequacy of the tautological sentence. We employed three BERT-based models and assessed their capability based on the results of human evaluation. The second experiment involved having ChatGPT generate a coherent sentence that followed the preceding sentence and tautological expression. We categorized certain types of errors from the generated sentences that were deemed inappropriate. This study is significant in revealing the limitations of contemporary deep learning-based language models in terms of abductive reasoning and suggests a new task for standardized model evaluation.
Abstract
The purpose of this paper is to evaluate the abductive reasoning capability of deep learning-based language models, with a specific focus on their ability to infer the meaning of tautological expressions. To achieve this, we conducted two experiments that required language models to understand the meaning of tautological expressions in different contexts. The first experiment was a binary classification task in which the models were presented with preceding sentences as contexts along with a tautological expression, and evaluated the adequacy of the tautological sentence. We employed three BERT-based models and assessed their capability based on the results of human evaluation. The second experiment involved having ChatGPT generate a coherent sentence that followed the preceding sentence and tautological expression. We categorized certain types of errors from the generated sentences that were deemed inappropriate. This study is significant in revealing the limitations of contemporary deep learning-based language models in terms of abductive reasoning and suggests a new task for standardized model evaluation.
참고문헌 (26)
- Bang, Y., "A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity", ~ (2023)
- Bulhof, J., "Deep tautologies", Pragmatics & Cognition, 9(2), pp.279~291 (2001)
- Choi, C. -L., "Interpretation of the Meaning of Tautology", HUMANITIES RESEARCH, 1, pp.139~150 (1992)
- Gao, Y., "Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System", ~ (2023)
- 홍승혜, "항진적 표현과 은유적 표현의 서술어 기제에 관한 기호학적 접근 - ‘지각적 판단’과 ‘논리적 해석체’를 중심으로", 기호학 연구, 72, pp.169~197 (2022), 한국기호학회
- Ippolito, D., "Comparison of Diverse Decoding Methods from Conditional Language Models", Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.3752~3762 (2019), Association for Computational Linguistics
- 전혜영, "구어 담화에 나타나는 ‘X는 X다’ 표현의 화용 양상", 國語學, (64), pp.273~299 (2012), 국어학회[DOI]
- Jeretic, P., "Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.8690~8705 (2020), Association for Computational Linguistics
- Kwon, I., "A tautology is a tautology: specificity and categorization in nominal tautological constructions", Berkeley Linguistics Society, 35, pp.1~36 (2009)
- 이정애, "국어 항진명제에 대한 의미 연구", 한국어 의미학, 33, pp.179~202 (2010), 한국어의미학회
- 이미순, "항진 명제 발화의 해석에 대한 재검토", 담화와 인지, 16(2), pp.123~147 (2009), 담화·인지언어학회[DOI]
- Lee, S., "KR-BERT: A small-scale Korean-specific language model", ~ (2020)
- Levinson, S. C., "Pragmatics. Cambridge" (1983), Cambridge University Press
- "NIKL Written Corpus(v.1.2)" (2020)[URL]
- "OpenAI API"[URL]
- "Introducing ChatGPT" (2022)[URL]
- Peirce, C. S, "Collected Papers of Charles S. Peirce 8 vols" (1958), Harvard University Press
- Peng, K., "Towards making the most of chatgpt for machine translation", ~ (2023)
- Polak, M. P., "Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering -Example of ChatGPT", ~ (2023)
- Reiss, M. V., "Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark", ~ (2023)
- Srivastava, A., "Beyond the imitation game: Quantifying and extrapolating the capabilities of language model", ~ (2022)
- Viola, Tullio, "Pragmatism and Embodied Cognitive Science: From Bodily Intersubjectivity to Symbolic Articulation", pp.251~268 (2016), De Gruyter
- Wang, J., "Cross-lingual summarization via chatgpt", ~ (2023)
- Warstadt, A., "Neural network acceptability judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
- Wei, J., "Finetuned language models are zero-shot learners", ~ (2021)
- Wolf, T., "Transformers: State-of-the-art natural language processing", Proceedings of the 2020 conference on Empirical Methods in Natural Language Processing: System Demonstrations (2020)