혐오와 대항: 혐오표현 탐지 모델 평가를 위한 대항표현 데이터셋 구축
Countering the hatred: The counter-speech dataset in Korean for evaluating hate speech detection models
2
조회
0
다운로드
4
인용
0
참고문헌
키워드
초록
This study argues for the necessity of a Korean counter-speech dataset for ethical and effective hate speech detection research. Counter-speech is a response to online hate in order to stop the spread of hate speech and is considered an alternative approach to deleting and blocking. However, since counter-speech often employs offensive language or linguistic structures similar to hate speech, even the state-of-the-art hate speech detection models usually classify it as hate speech. This false positive bias risks silencing the language of minorities and their allies. However, the evaluation of Korean hate speech detection models remains untouched due to the absence of a Korean counter-speech dataset. Thus, we introduce the first Korean counter-speech dataset with annotations about target groups. We then tested a Korean hate speech detection model with our dataset, revealing a significant drop in the model’s accuracy from 97.9% to 42.7%.
Abstract
This study argues for the necessity of a Korean counter-speech dataset for ethical and effective hate speech detection research. Counter-speech is a response to online hate in order to stop the spread of hate speech and is considered an alternative approach to deleting and blocking. However, since counter-speech often employs offensive language or linguistic structures similar to hate speech, even the state-of-the-art hate speech detection models usually classify it as hate speech. This false positive bias risks silencing the language of minorities and their allies. However, the evaluation of Korean hate speech detection models remains untouched due to the absence of a Korean counter-speech dataset. Thus, we introduce the first Korean counter-speech dataset with annotations about target groups. We then tested a Korean hate speech detection model with our dataset, revealing a significant drop in the model’s accuracy from 97.9% to 42.7%.
참고문헌 (26)
- 이수진, "‘혐오’의 공기 경향성과 의미적 운율 - 뉴스 기사문 말뭉치를 중심으로 -", 담화와 인지, 29(1), pp.21~44 (2022), 담화·인지언어학회
- 지인영, "대형 사전학습 언어모델 연구에 대한 고찰", 담화와 인지, 28(4), pp.129~148 (2021), 담화·인지언어학회
- 홍성수, "세션1 : 사이버상 혐오표현의 법적 쟁점과 규제 방안", 언론중재, 140, pp.44~57 (2016)
- 홍성수, "말이 칼이 될 때: 혐오표현은 무엇이고 왜 문제인가?" (2018), 어크로스
- 홍성수, "혐오표현의 해악과 개입의 정당성: 금지와 방치를 넘어서", 법철학연구, 22(3), pp.27~64 (2019), 한국법철학회[DOI]
- Benesch, Susan, "Countering dangerous speech: New ideas for genocide prevention", United States Holocaust Memorial Museum, ~ (2014)
- Chung, Yi-Ling, "CONAN--COunter NArratives through Nichesourcing: a multilingual dataset of responses to fight online hate speech", arXiv:1910.03270, ~ (2019)
- Davidson, Thomas, "Automated hate speech detection and the problem of offensive language", Proceedings of the International AAAI Conference on Web and Social Media (2017)
- Devlin, Jacob, "BERT : Pre-training of deep bidirectional transformers for language understanding", arXiv:1810.04805, ~ (2019)
- Dixon, Lucas, "Measuring and mitigating unintended bias in text classification", Proceedings of the 2018 AAAI, 2018, ~
- Gagliardone, Iginio, "Countering online hate speech" (2015), UNESCO Publishing
- Garland, Joshua, "Countering hate on social media: Large scale classification of hate and counter speech", arXiv:2006.01974, ~ (2020)
- Habermas, Jürgen, "Between facts and norms: Contributions to a discourse theory of law and democracy" (2015), John Wiley & Sons
- Kurrek, Jana, "Towards a comprehensive taxonomy and large-scale annotated corpus for online slur usage", Proceedings of the Fourth Workshop on Online Abuse and Harms (2020)
- Mathew, Binny, "Analyzing the hate and counter speech accounts on twitter", arXiv:1812.02712, ~ (2018)
- Moon, Jihyung, "BEEP! Korean corpus of online news comments for toxic speech detection", arXiv:2005.12503, ~ (2020)
- Palmer, Alexis, "COLD : Annotation scheme and evaluation data set for complex offensive language in English", Journal for Language Technology and Computational Linguistics, 34(1), pp.1~28 (2020)
- Park, Ji Ho, "Reducing gender bias in abusive language detection", arXiv:1808.07231, ~ (2018)
- Parrish, Alicia, "Does putting a linguist in the loop improve NLU data collection?", arXiv:2104.07179, ~ (2021)
- Qian, Jing, "A benchmark dataset for learning to intervene in online hate speech", arXiv:1909.04251, ~ (2019)
- Röttger, Paul, "HateCheck: functional tests for hate speech detection models", arXiv:2012.15606, ~ (2021)
- Schieb, Carla, "Governing hate speech by means of counterspeech on Facebook", Paper presented at the 66th ICA Annual Conference (2016)
- The impact of counter-narratives, "Silverman, Tanya", Institute for Strategic Dialogue, 1~54 (2016)
- Waseem, Zeerak, "Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter", Proceedings of the NAACL Student Research Workshop, pp.88~93 (2016)
- Waseem, Zeerak, "Online harassment" (2018)
- Ziegele, Marc, "Journalistic counter-voices in comment sections: Patterns, determinants, and potential consequences of interactive moderation of uncivil user comments", SCM Studies in Communication and Media, 7(4), pp.525~554 (2018)[DOI]