딥러닝 기반 감성 데이터 분류에 관한 연구

Abstract

Recently, as attempts to analyze and utilize human emotions increase, it is trying to realize this through various data analysis methods. However, most of the existing emotional analysis has limitations in analyzing human emotions using unimodal data. Therefore, this study proposes an emotional classification deep learning model based on multimodal data using image data. The model was constructed to extract the characteristics of data representing people well using a deep learning structure specialized for each data modality. The multimodal image data was divided into two types: image and speech. First, the image data was extracted using a 3D convolutional neural network, a model suitable for 3D data. Here, it was possible to effectively extract features using only the facial part that plays an important role in emotional classification. Secondly, the speech data was converted into a spectrogram image using the Mel-frequency cepstral coefficient (MFCC) technique, which has shown the best performance in the related field. In a recent spectrogram image analysis study, features were extracted by transforming the VGG-M network that shows good performance according to this proposal method. After combining the obtained feature vectors, a structure that predicts emotional classification through a classifier was used. To demonstrate the superiority of the proposed method, we compared the emotional classification performance with a single form of data and confirmed the performance improvement. This study also has a contributing point in that it effectively classified emotions by applying a multimodal structure to emotional analysis data.