인공지능 기반의 말더듬 자동분류 방법: 합성곱신경망(CNN) 활용
AI-based stuttering automatic classification method: Using a convolutional neural network
0
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
본 연구는 말더듬 화자들의 음성 데이터를 기반으로 하여, 인공지능 기술을 활용한 말더듬 자동 식별 방법을 개발하는 것을 주목적으로 진행되었다. 특히, 한국어를 모국어로 하는 말더듬 화자들을 대상으로 CNN(convolutional neural network) 알고리즘을 활용한 식별기 모델을 개발하고자 하였다. 이를 위해 말더듬 성인 9명과 정상화자 9명을 대상으로 음성 데이터를 수집하고, Google Cloud STT(Speech-To-Text)를 활용하여 어절 단위로 자동 분할한 후유창, 막힘, 연장, 반복 등의 라벨을 부여하였다. 또한 MFCCs(mel frequency cepstral coefficients)를 추출하여 CNN 알고리즘을 기반한 말더듬 자동 식별기 모델을 수립하고자 하였다. 연장의 경우 수집결과가 5건으로 나타나 식별기모델에서 제외하였다. 검증 결과, 정확도는 0.96으로 나타났고, 분류성능인 F1-score는 ‘유창’은 1.00, ‘막힘’은 0.67, ‘반복’은 0.74로 나타났다. CNN 알고리즘을 기반한 말더듬 자동분류 식별기의 효과를 확인하였으나, 막힘 및 반복유형에서는 성능이 미흡한 것으로 나타났다. 향후 말더듬의 유형별 충분한 데이터 수집을 통해 추가적인 성능 검증이 필요함을 확인하였다. 향후 말더듬 화자의 발화 빅데이터 확보를 통해 보다 신뢰성 있는 말더듬 자동 식별 기술의 개발과 함께 이를 통한 좀 더 고도화된 평가 및 중재 관련 서비스가 창출되기를 기대해 본다.
Abstract
This study primarily aimed to develop an automated stuttering identification and classification method using artificial intelligence technology. In particular, this study aimed to develop a deep learning-based identification model utilizing the convolutional neural networks (CNNs) algorithm for Korean speakers who stutter. To this aim, speech data were collected from 9 adults who stutter and 9 normally-fluent speakers. The data were automatically segmented at the phrasal level using Google Cloud speech-to-text (STT), and labels such as ‘fluent’, ‘blockage’, prolongation’, and ‘repetition’ were assigned to them. Mel frequency cepstral coefficients (MFCCs) and the CNN-based classifier were also used for detecting and classifying each type of the stuttered disfluency. However, in the case of prolongation, five results were found and, therefore, excluded from the classifier model. Results showed that the accuracy of the CNN classifier was 0.96, and the F1-score for classification performance was as follows: ‘fluent’ 1.00, ‘blockage’ 0.67, and ‘repetition’ 0.74. Although the effectiveness of the automatic classification identifier was validated using CNNs to detect the stuttered disfluencies, the performance was found to be inadequate especially for the blockage and prolongation types. Consequently, the establishment of a big speech database for collecting data based on the types of stuttered disfluencies was identified as a necessary foundation for improving classification performance.
참고문헌 (28)
- Altinkaya, M., "A dynamic, self supervised, large scale audiovisual dataset for stuttered speech", Proceedings of the 1st International Workshop on Multimodal Conversational AI, pp.9~13 (2020)
- Barrett, L., "Systematic review of machine learning approaches for detecting developmental stuttering", IEEE, 2022, pp.30, 1160~1172
- Bayerl, S. P., "KSoF: The Kassel state of fluency dataset –A therapy centered dataset of stuttering", Proceedings of the 13th Conference on Language Resources and Evaluation, pp.1780~1787 (2022)
- Bhushan, P. S., "Stuttered Speech Recognition using Convolutional Neural Networks", International Journal of Engineering Research & Technology, 9(12), pp.250~254 (2021)
- Das, A., "Multimodal explainable AI predicts upcoming speech behavior in adults who stutter", Frontiers in Neuroscience, 16, pp.912798~ (2022)
- Fang, S. H., "Detection of pathological voice using cepstrum vectors : A deep learning approach", Journal of Voice, 33(5), pp.634~641 (2019)
- Garg, U., "Prediction of emotions from the audio speech signals using MFCC, MEL and Chroma", Proceedings of the 12th International Conference on Computational Intelligence and Communication Networks (CICN) (2020)
- Goodfellow, I., "Deep learning" (2016), MIT Press
- Guitar, B., "Stuttering: An integrated approach to its nature and treatment" (2019), Lippincott Williams
- Hariharan, M., "Classification of speech disfluencies using LPC based parameterization techniques", Journal of Medical Systems, 36(3), pp.1821~1830 (2012)
- Howell, P., "Development of a two-stage procedure for the automatic recognition of dysfluencies in the speech of children who stutter: II. ANN recognition of repetitions and prolongations with supplied word segment markers", Journal of Speech, Language, and Hearing Research, 40(5), pp.1085~1096 (1997)
- 전희숙, "말더듬 성인의 비유창성군 특성", 언어치료연구, 24(1), pp.135~144 (2015), 한국언어치료학회[DOI]
- 조철우, "기계학습에 의한 후두 장애음성 식별기의 성능 비교", 말소리와 음성과학, 14(4), pp.35~43 (2022), 한국음성학회[DOI]
- Kully, D., "An investigation of interclinic agreement in the identification of fluent and stuttered syllables", Journal of Fluency Disorders, 13(5), pp.309~318 (1988)
- Lee, Y. H., "Speech", audio processing based on deep learning, 2017, pp.22(1), 47~58
- Mishra, N., "Optimization of stammering in speech recognition applications", International Journal of Speech Technology, 24(2), pp.679~685 (2021)
- 박진, "배경소음상황에 따른 성인 말더듬화자의 발화 관련 변수 비교", 말소리와 음성과학, 7(1), pp.27~37 (2015), 한국음성학회[DOI]
- Prabhu, Y., "A CNN-based automated stuttering identification system", Proceeding of the 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) (2022)
- Ravikumar, K. M., "Stuttered Speech Using MFCC Features", ICGST International Journal on Digital Signal Processing, 9, pp.19~24 (2009)
- Riley, G. D., "A stuttering severity instrument for children and adults", Journal of Speech and Hearing Disorders, 37(3), pp.314~322 (1972)
- Sheikh, S. A., "Machine learning for stuttering identification : Review, challenges and future directions", Neurocomputing, 514, pp.385~402 (2022)
- Shim, H. S., "Paradise Fluency Assessment-II (P-FA-II)" (2010), Paradise Welfare Foundation
- Shim, H. S., "Fluency disorders: Assessment and treatment" (2022)
- Tichenor, S. E., "A point of view about fluency", Journal of Speech, Language, and Hearing Research, 65(2), pp.645~652 (2022)
- Van Riper, C., "Speech correction: Principles and methods" (1972), Prentice-Hall
- Wiśniewski, M., "Computer recognition systems 2: Advances in soft computing", pp.445~453 (2007), Springer
- Yang, B., "Facial action unit-based deep learning framework for spotting macro- and micro-expressions in long video sequences", Proceedings of the 29th ACM International Conference on Multimedia, pp.4794~4798 (2021)
- Yaruss, S. J., "Utterance timing and childhood stuttering", Journal of Fluency Disorders, 22(4), pp.263~286 (1997)