Speech Emotion Recognition in People at High Risk of Dementia
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Background and Purpose: The emotions of people at various stages of dementia need to be effectively utilized for prevention, early intervention, and care planning. With technology available for understanding and addressing the emotional needs of people, this study aims to develop speech emotion recognition (SER) technology to classify emotions for people at high risk of dementia. Methods: Speech samples from people at high risk of dementia were categorized into distinct emotions via human auditory assessment, the outcomes of which were annotated for guided deep-learning method. The architecture incorporated convolutional neural network, long short-term memory, attention layers, and Wav2Vec2, a novel feature extractor to develop automated speech-emotion recognition. Results: Twenty-seven kinds of Emotions were found in the speech of the participants. These emotions were grouped into 6 detailed emotions: happiness, interest, sadness, frustration, anger, and neutrality, and further into 3 basic emotions: positive, negative, and neutral. To improve algorithmic performance, multiple learning approaches were applied using different data sources—voice and text—and varying the number of emotions. Ultimately, a 2-stage algorithm—initial text-based classification followed by voice-based analysis—achieved the highest accuracy, reaching 70%. Conclusions: The diverse emotions identified in this study were attributed to the characteristics of the participants and the method of data collection. The speech of people at high risk of dementia to companion robots also explains the relatively low performance of the SER algorithm. Accordingly, this study suggests the systematic and comprehensive construction of a dataset from people with dementia.
Abstract
Background and Purpose: The emotions of people at various stages of dementia need to be effectively utilized for prevention, early intervention, and care planning. With technology available for understanding and addressing the emotional needs of people, this study aims to develop speech emotion recognition (SER) technology to classify emotions for people at high risk of dementia. Methods: Speech samples from people at high risk of dementia were categorized into distinct emotions via human auditory assessment, the outcomes of which were annotated for guided deep-learning method. The architecture incorporated convolutional neural network, long short-term memory, attention layers, and Wav2Vec2, a novel feature extractor to develop automated speech-emotion recognition. Results: Twenty-seven kinds of Emotions were found in the speech of the participants. These emotions were grouped into 6 detailed emotions: happiness, interest, sadness, frustration, anger, and neutrality, and further into 3 basic emotions: positive, negative, and neutral. To improve algorithmic performance, multiple learning approaches were applied using different data sources—voice and text—and varying the number of emotions. Ultimately, a 2-stage algorithm—initial text-based classification followed by voice-based analysis—achieved the highest accuracy, reaching 70%. Conclusions: The diverse emotions identified in this study were attributed to the characteristics of the participants and the method of data collection. The speech of people at high risk of dementia to companion robots also explains the relatively low performance of the SER algorithm. Accordingly, this study suggests the systematic and comprehensive construction of a dataset from people with dementia.
참고문헌 (48)
- Sutin AR, "Psychological well-being and risk of dementia", Int J Geriatr Psychiatry, 33, pp.743~747 (2018)
- Katon W, "Effect of depression and diabetes mellitus on the risk for dementia : a national population-based cohort study", JAMA Psychiatry, 72, pp.612~619 (2015)
- Ownby RL, "Depression and risk for Alzheimer disease : systematic review, meta-analysis, and metaregression analysis", Arch Gen Psychiatry, 63, pp.530~538 (2006)
- da Silva J, "Affective disorders and risk of developing dementia : systematic review", Br J Psychiatry, 202, pp.177~186 (2013)
- Richard E, "Late-life depression, mild cognitive impairment, and dementia", JAMA Neurol, 70, pp.374~382 (2013)
- Mourao RJ, "Depressive symptoms increase the risk of progression to dementia in subjects with mild cognitive impairment : systematic review and meta-analysis", Int J Geriatr Psychiatry, 31, pp.905~911 (2016)
- Weng X, "Association between subjective cognitive decline and social and emotional support in US adults", Am J Alzheimers Dis Other Demen, 35, pp.1533317520922392~ (2020)
- Lawton MP, "Observed affect in nursing home residents with Alzheimer’s disease", J Gerontol B Psychol Sci Soc Sci, 51, pp.3~14 (1996)
- Vogelpohl TS, "Affective responses to behavioral interventions", Semin Clin Neuropsychiatry, 2, pp.102~112 (1997)
- Ekman P, "Emotions Revealed, Second Edition: Recognizing Faces and Feelings to Improve Communication and Emotional Life" (2007), Henry Holt and Company
- Izdebski K, "Emotions in the Human Voice. Volume 1, Foundations" (2008), Plural Publishing, Inc
- Higuchi M, "Effectiveness of a voice-based mental health evaluation system for mobile devices : prospective study", JMIR Form Res, 4, pp.e16455~ (2020)
- Kwon OW, "Emotion recognition by speech signals", Proceedings of the 8th European Conference on Speech Communication and Technology (Eurospeech 2003), pp.125~128 (2003), International Speech Communication Association
- Nogueiras A, "Speech emotion recognition using hidden Markov models", Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech 2001), pp.2679~2682 (2001), International Speech Communication Association
- Hansen JH, "Analysis and compensation of speech under stress and noise for environmental robustness in speech recognition", Speech Commun, 20, pp.151~173 (1996)
- Bou-Ghazale SE, "A comparative study of traditional and newly proposed features for recognition of speech under stress", IEEE Trans Speech Audio Process, 8, pp.429~442 (2000)
- Chuang ZJ, "Multi-modal emotion recognition from speech and text", Int J Comput Linguist Chin Lang Process, 9, pp.45~62 (2004)
- Lu Q, "Sentiment analysis : comprehensive reviews, recent advances, and open challenges", IEEE Trans Neural Netw Learn Syst, PP, pp.1~21 (2023)
- Livingston G, "Dementia prevention, intervention, and care : 2020 report of the Lancet Commission", Lancet, 396, pp.413~446 (2020)
- Dementia, Disability and Frailty in Later Life – Mid-Life Approaches to Delay or Prevent Onset, "National Institute for Health and Care Excellence (NICE)", NICE, ~ (2015)
- Daviglus ML, "Risk factors and preventive interventions for Alzheimer disease : state of the science", Arch Neurol, 68, pp.1185~1190 (2011)
- Saczynski JS, "Depressive symptoms and risk of dementia : the Framingham Heart Study", Neurology, 75, pp.35~41 (2010)
- Barnes DE, "Depressive symptoms, vascular disease, and mild cognitive impairment : findings from the Cardiovascular Health Study", Arch Gen Psychiatry, 63, pp.273~279 (2006)
- Power MC, "Combined neuropathological pathways account for age-related risk of dementia", Ann Neurol, 84, pp.10~22 (2018)
- Escobar-Linero E, "On the feature extraction process in machine learning. An experimental study about guided versus non-guided process in falling detection systems", Eng Appl Artif Intell, 114, pp.105170~ (2022)
- Mondal A, "Mining emotions on Plutchik’s wheel", Proceedings of the 2020 Seventh International Conference on Social Networks Analysis, Management and Security (SNAMS), pp.1~6 (2020), IEEE
- Baevski A, "wav2vec 2. 0 : A framework for self-supervised learning of speech representations", Adv Neural Inf Process Syst, 33, pp.12449~12460 (2020)
- Lecun Y, "Gradient-based learning applied to document recognition", Proc IEEE, 86, pp.2278~2324 (1998)
- Hochreiter S, "Long short-term memory", Neural Comput, 9, pp.1735~1780 (1997)
- Vaswani A, "Attention is all you need", Adv Neural Inf Process Syst, pp.30~ (2017)
- Kumar Y, "Artificial intelligence in disease diagnosis : a systematic literature review, synthesizing framework and future research agenda", J Ambient Intell Humaniz Comput, 14, pp.8459~8486 (2023)
- Bucks RS, "Emotion processing in Alzheimer’s disease", Aging Ment Health, 8, pp.222~232 (2004)
- Mega MS, "The spectrum of behavioral changes in Alzheimer’s disease", Neurology, 46, pp.130~135 (1996)
- Algase DL, "Need-driven dementia-compromised behavior : an alternative view of disruptive behavior", Am J Alzheimer Dis, 11, pp.10~19 (1996)
- Gonçalves-Pereira M, "Neuropsychiatric Symptoms in Cognitive Impairment and Dementia", pp.1~7 (2017), Springer
- Han KH, "Impairment of vocal expression of negative emotions in patients with Alzheimer’s disease", Front Aging Neurosci, 6, pp.101~ (2014)
- Cadieux NL, "Emotion processing in Alzheimer’s disease", J Int Neuropsychol Soc, 3, pp.411~419 (1997)
- Zandi T, "Facial recognition : a cognitive study of elderly dementia patients and normal older adults", Int Psychogeriatr, 4, pp.215~221 (1992)
- Hahn EA, "Daily experiences in stress, memory, and emotion in older adults with mild cognitive impairment", pp.2012
- Shah Fahad M, "A survey of speech emotion recognition in natural environment", Digit Signal Process, 110, pp.102951~ (2021)
- Livingstone SR, "The Ryerson Audio-Visual Database of Emotional Speech and Song(RAVDESS) : a dynamic, multimodal set of facial and vocal expressions in North American English", PLoS One, 13, pp.e0196391~ (2018)
- Burkhardt F, "A database of German emotional speech", Proceedings of the 9th European Conference on Speech Communication and Technology (Interspeech 2005), pp.1517~1520 (2005), International Speech Communication Association
- "Free conversation with emotion tags (adult)" (2022)[URL]
- Al-onazi BB, "Transformer-based multilingual speech emotion recognition using data augmentation and feature fusion", Applied Sciences, 12, pp.9188~ (2022)
- Xue C, "Detection of dementia on voice recordings using deep learning : a Framingham Heart Study", Alzheimers Res Ther, 13, pp.146~ (2021)
- Park CY, "Harnessing the power of voice : a deep neural network model for Alzheimer’s disease detection", Dement Neurocogn Disord, 23, pp.1~10 (2024)