언어 인공지능의 상식추론과 평가 체계 현황
Commonsense Reasoning of Neural Language Models and Trends in Evaluation Benchmarks
0
조회
0
다운로드
5
인용
0
참고문헌
키워드
초록
최근 인공지능의 성능이 고도로 향상됨에 따라, 인공지능이 인간의 언어 구사 능력에 가까워졌다는 주장이 제기되었다. 예컨대, 인공지능 GPT-3는 인간의 작문 능력과 구별되지 않는 성능을 보이는 것처럼 알려졌다. 그러나, 구체적인 평가의 영역에 따라 인공지능과 인간이 큰 격차를 보인다. 대표적인 것이 상식추론이다. 예를 들어, 영희가 책가방을 메고 학교에 가는지, 아니면 나이트클럽을 가는지는 논리가 아닌 상식에 비추어 자명하다. 특히, 상식추론은 경험세계에 대한 광범위한 지식이 필요하다는 점에서, 문자열의 분포적 정보로부터 사실적인 지식을 이끌어내야 하는 인공지능에게 매우 도전적인 과제이다. 이 점에 착안하여 최근 인공지능이 상식추론을 학습하였는지 평가하기 위한 정량적 평가 체계 또는 벤치마크가 공개되고 있다. 튜링 테스트에서 출발한 벤치마크는 일종의 수만 건의 문제은행으로서,정확도와 유사도를 기반으로 인공지능의 상식추론을 정량적으로 검증한다. 이에 본고는 인공지능 상식추론과 평가 체계의 현황을 폭넓게 검토하고, 인문사회학적 관점에서 비판적인 이해를 시도한다. 구체적으로, 자연어처리 분야의 신경망 언어 모형 또는 워드 임베딩이 어떻게 문자열을 학습하는지 개념적으로 이해한다. 이와 함께, 인공지능이 학습한 추론 지식을 검증하는 평가 체계 또는 자연어처리 벤치마크의 구축 방법론과 예시 문장을 분석한다. 이를 위하여 최근 공개한 한국어 인공지능 벤치마크인 KLUE를 사례로 분석을 제시한다. 또한, 대표적인 벤치마크인 SWAG, CosmosQA, 그리고 CommonGen을 분석한다. 이와 함께, 최근의 대규모 인공지능의 개발이 내포하는 환경적, 경제적, 윤리적 우려가 커지고 있음을 지적하고, 언어학적 튜링 테스트를 중심으로 정량적 평가 체계의 본질적인 한계를 논의한다.
Abstract
Recent advances in artificial intelligence (AI) showed that language model, i.e., probability estimator of word occurrences in contexts, may capture the human reasoning ability. This surprising finding is built upon the previous research on the empirical evaluation of the reasoning ability of AIs. The empirical evaluation measures the performance gap between the AIs and human speakers on “well-curated” datasets, often called as NLP benchmarks. Recently, many researchers propose new NLP benchmarks testing the commonsense reasoning, e.g, what should you do if you encounter a grizzly bear chasing the baby?. The question is tricky because the decision depends on situations, where baby may refer to the human baby or simply the baby cow. Thus, it has been noted that benchmarking the commonsense reasoning is critical for AI’s human-like performance in intelligent tasks. In this paper, we review some types of commonsense reasoning and newly released evaluation benchmarks, suggesting that the reliability of dataset is hinged upon the data curation method. We first briefly introduce how language model using neural networks learns or predicts the word probability. We then proceed to review how workers or annotators curate the NLP benchmarks, focusing on the collection of human intuitions regarding the text examples. Although these curation methods empirically show the diverse reasoning ability of AI, there are concerns about the negative social impacts of extremely large AIs. Importantly, NLP benchmarks are sometimes misleading because AI simply captures the shallow surface structure of language, which denotes that AI successfully mapped texts to texts. Overall, we suggest that constructing the colossal AIs is not a silver bullet to commonsense reasoning since AIs are not free from data bias.
참고문헌 (35)
- 한지윤, "언어 추론 모델 개발을 위한 말뭉치 구축 방법론 연구", 언어사실과 관점, 48, pp.351~384 (2019), 언어정보연구원[DOI]
- 변순용, "인공지능 윤리하다", pp.137~157 (2020), 어문학사
- Bender, E. M., "Climbing towards NLU: On meaning, form, and understanding in the age of data", Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.5185~5198 (2020)
- Bender, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?", Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp.610~623 (2021)
- Bowman, "A large annotated corpus for learning natural language inference", arXiv preprint, arXiv:1508.05326, ~ (2015)
- Brown, "Language models are few-shot learners", Advances in neural information processing systems, 33, pp.1877~1901 (2020)
- Burke, K., "A grammar of motives" (1969), Univ of California Press
- Dagan, "The pascal recognising textual entailment challenge", Machine learning challenges workshop, pp.177~190 (2005)
- Davis, E., "Commonsense reasoning and commonsense knowledge in artificial intelligence", Communications of the ACM, 58(9), pp.92~103 (2015)
- Devlin, "Bert: Pre-training of deep bidirectional transformers for language understanding", arXiv preprint, arXiv:1810.04805, ~ (2018)
- Fodor, J. A., "Connectionism and cognitive architecture: A critical analysis", Cognition, 28(1), pp.3~71 (1988)
- Forbus, K. D., "Qualitative process theory", Artificial intelligence, 24(1-3), pp.85~168 (1984)
- Gehrmann, "The gem benchmark: Natural language generation, its evaluation and metrics", arXiv preprint, arXiv:2102.01672, ~ (2021)
- Goodfellow, "Deep learning" (2016), MIT press
- Gururangan, "Annotation artifacts in natural language inference data", Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pp.107~112 (2018)
- Huang, "Cosmos QA: Machine reading comprehension with contextual commonsense reasoning", arXiv preprint, arXiv:1909.00277, ~ (2019)
- LeCun, "Deep learning", nature, 521, pp.436~444
- Lin, "CommonGen: A constrained text generation challenge for generative commonsense reasoning", arXiv preprint, arXiv:1911.03705, ~ (2019)
- Linzen, "Assessing the ability of LSTMs to learn syntax-sensitive dependencies", Transactions of the Association for Computational Linguistics, 4, pp.521~535 (2016)
- Liu, "KG-BART: Knowledge graph-augmented BART for generative commonsense reasoning", Proceedings of the AAAI Conference on Artificial Intelligence, 35(7), pp.6418~6425 (2021)
- Melamud, "context2vec: Learning generic context embedding with bidirectional lstm", Proceedings of the 20th SIGNLL conference on computational natural language learning, pp.51~61 (2016)
- Papineni, "Bleu: a method for automatic evaluation of machine translation", Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.311~318 (2002)
- Park, "Klue: Korean language understanding evaluation", NeurIPS 2021Datasets Benchmarks Track (2021)
- Sap, "Socialiqa: Commonsense reasoning about social interactions", arXiv preprint, arXiv:1904.09728, ~ (2019)
- Searle, J., "The Philosophy of Artificial Intelligence, Computing machinery and intelligence" (1990)
- Seo, "KommonGen: A Dataset for Korean Generative Commonsense Reasoning Evaluation", Annual Conference on Human and Language Technology, pp.55~60 (2021)
- Srivastava, "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models", arXiv preprint, arXiv:2206.04615, ~ (2022)
- Storks, "Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches", arXiv preprint arXiv:1904.01172, 1~60 (2019)
- Strubell, "Energy and policy considerations for deep learning in NLP", Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.3645~3650 (2019)
- Turing, A. M.m, "Computing Machinery and Intelligence", Mind, 49, pp.433~460 (1950)
- Vig, "Investigating gender bias in language models using causal mediation analysis", Advances in Neural Information Processing Systems, 33, pp.12388~12401 (2020)
- Wang, "GLUE: A multi-task benchmark and analysis platform for natural language understanding", International Conference on Learning Representations (2018)
- Warstadt, "Neural network acceptability judgments", Transactions of the Association for Computational Linguistics, 7, pp.625~641 (2019)
- Williams, "A broad-coverage challenge corpus for sentence understanding through inference", arXiv preprint, arXiv:1704.05426, ~ (2017)
- Zellers, "Swag: A large-scale adversarial dataset for grounded commonsense inference", arXiv preprint, arXiv:1808.05326, ~ (2018)