Prompting Strategies of Generative AI for Korean Pragmatic Inference

Abstract

This study explores the pragmatic inference abilities of large language models (LLMs) in the context of Korean, focusing on their ability to understand conversational implicature as defined by Grice’s four maxims (Grice 1975): Quantity, Quality, Relation, and Manner. Though generative AI, partially based on LLMs, has revolutionized natural language processing (NLP), their effects in interpreting implied meanings in human conversation remain significant challenges. One approach to improving this aspect is through chain-of-thought (CoT) prompting, which enables the models to solve tasks in a step-by-step manner by incorporating intermediate reasoning steps in prompts. CoT prompting has established remarkable accuracy on many logical tasks. Therefore, this study evaluated pragmatic inference abilities of different LLMs such as GPT, Gemini, Llama, and HyperClova-X, assessing their performance using various prompting strategies—zero-shot-CoT (Koima et al. 2022), CoT (Wei et al. 2022b), and contrastive-CoT (Chia et al. 2023) — in a Korean pragmatic test set (Park et al. 2024). The study revealed that CoT prompting generally enhanced the models’ accuracy across most maxims, while zero-shot-CoT and contrastive-CoT prompting showed mixed results. The findings suggest that while various prompting strategies have been beneficial for logical reasoning tasks, its application in pragmatic inference requires careful consideration of context and the nature of conversational implicature. This study contributes to the understanding of how different prompting strategies impact LLMs’ performance in pragmatic tasks, providing insights that can help bridge the gap between advanced AI abilities and human-like comprehension of language.