Evaluation of BERT-Based Models’ Ability to Linguistically Distinguish Between Human-Authored and Machine-Generated Texts
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This research evaluates the effectiveness of bidirectional language models, particularly those based on the BERT architecture, in distinguishing between human-authored text and machine-generated text in the Korean language. Through an extensive empirical analysis using a newly established benchmark, KoMGTDetect-Bench, we explore the linguistic attributes and complexities that differentiate machine-generated text from human-authored text, such as vocabulary usage, sentence structure, and syntax. Our findings reveal that BERT-based models excel in detecting subtle nuances and irregularities in text, attributing their success to the model's deep semantic and syntactic understanding enabled by bidirectional context-aware training strategy. This study also examines the performance of these models across various linguistic phenomena specific to the Korean language, highlighting their strengths and limitations.
Abstract
This research evaluates the effectiveness of bidirectional language models, particularly those based on the BERT architecture, in distinguishing between human-authored text and machine-generated text in the Korean language. Through an extensive empirical analysis using a newly established benchmark, KoMGTDetect-Bench, we explore the linguistic attributes and complexities that differentiate machine-generated text from human-authored text, such as vocabulary usage, sentence structure, and syntax. Our findings reveal that BERT-based models excel in detecting subtle nuances and irregularities in text, attributing their success to the model's deep semantic and syntactic understanding enabled by bidirectional context-aware training strategy. This study also examines the performance of these models across various linguistic phenomena specific to the Korean language, highlighting their strengths and limitations.
참고문헌 (31)
- Devlin, Jacob, "BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding", Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp.4171~4186 (2019)
- Lee, Bruce W., "LFTK: Handcrafted Features in Computational Linguistics", Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, pp.1~19 (2023)
- Lee, Iksop, "The Korean Language" (1997), Singu Munhwasa
- "Kcelectra: Korean Comments Electra" (2021)[URL]
- Macko, Dominik, "MULTITuDE: Large-scale Multilingual Machine-generated Text Detection Benchmark", Houda Bouamor, Juan Pino, and Kalika Bali, eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.9960~9987 (2023)
- Mok, Jeongsoo, "Linguistic Typology, Psychomechanics of Language and Korean Grammar" (2020), Hankook Munhwasa
- Research on 2023 Selection of Korean Basic Vocabulary and Vocabulary Grading, "National Institute of Korean Language", National Institute of Korean Language, ~ (2023)
- "KoELECTRA: Pretrained ELECTRA Model for Korean" (2020)[URL]
- Qazi, Zubair, "GPT-generated Text Detection: Benchmark Dataset and Tensor-based Detection Method", Companion Proceedings of the ACM on Web Conference 2024, pp.842~846 (2024)
- "Improving Language Understanding by Generative Pre-training" (2018)[URL]
- Rafailov, Rafael, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", Proceedings of the 37th International Conference on Neural Information Processing Systems, pp.53728~53741 (2024)
- Tan, Xiaoyu, "Enhancing Task Performance in Continual Instruction Fine-tuning Through Format Uniformity", Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.2384~2389 (2024)
- Wang, Yuxia, "SemEval-2024 Task 8: Multidomain, Multimodel and Multilingual Machine-generated Text Detection", Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pp.2057~2079 (2024)
- Wang, Yuxia, "M4GT-bench: Evaluation Benchmark for Black-box Machine-generated Text Detection", Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp.3964~3992 (2024)