본문 바로가기
KERD
논문KCI 등재

가상 시뮬레이션 학습자의 인지부하와 정서를 분석하기 위한 표정 분석 및 뇌전도의 활용

Use of Facial Expression Analysis and EEG to Analyze Cognitive Load and Emotion of Learners in Virtual Simulation

0

조회

0

다운로드

1

인용

0

참고문헌

키워드

주제분류교육학

초록

이 연구의 목적은 가상 시뮬레이션을 경험하는 학습자의 인지부하와 정서상태를 분석하기 위한 것이다. 가상 시뮬레이션 활용 교육에서 학습자는 지속적인 시청각 자극을 경험하기 때문에 이들의 사용자 경험은 매우 역동적으로 변화한다. 이를 기존의 사후보고식 방법으로는 정확하게 측정하기 어렵다. 따라서 이 연구에서는 가상 시뮬레이션을 경험하는 사용자가 시나리오의 내용 흐름에 따라 어떠한 인지·정서적 변화를 경험하는지 표정 분석과 뇌파를 통해 확인하고자 하였다. 연구 대상자는 대학생 34명(남자=10명, 여자=24명)이었다. 이 연구에서 사용된 가상 시뮬레이션은 학부모와의 상담사례로, 사용자는 가상 학부모와 면담하게 된다. 사용자 경험 중 인지부하를 분석하기 위해 분당 눈깜빡임(blink per min)과 중간 베타파(mid-beta wave, 15-20Hz)를 측정했다. 정서를 분석하기 위해서는 정서가(valence)와 전두엽 비대칭 알파파(frontal alpha asymmetry)를 측정했다. 시나리오를 내용 전개에 따라 4단계로 구분하고, 구간별 측정지표의 평균 변화를 분석하였다. 연구 결과, 1단계인 인사하기 단계에서 인지부하가 상승하고 긍정 정서가 높아졌다. 그러나 2단계 문제행동 언급 단계에서 인지부하와 긍정 정서 모두 급격히 감소하였다. 그 후, 3-4단계를 거치며 점차 재상승하는 패턴을 보였다. 이를 바탕으로 사용자 경험 중 인지부하와 정서가의 발생 패턴이 유사함을 확인할 수 있었다. 이 연구의 의의는 가상시뮬레이션 교육에서 인지·정서 측면의 사용자 경험을 측정하는데 생리 신호 기반의 멀티모달 측정법의 타당성을 확인하였다는 점에 있다.

Abstract

The purpose of this study was to analyze the cognitive load and emotional state of learners who are experiencing virtual simulations. In educational settings that use virtual simulations, learners experience continuous audiovisual stimuli that make their user experience highly dynamic. However, it is difficult to measure user experiences accurately with conventional post-self-report methods. Therefore, this study attempted to determine the cognitive and emotional changes that virtual simulation users experienced as the scenario content progressed using facial expression analysis and brainwave measurements. The participants consisted of 34 college students (10 males and 24 females). The virtual simulation used in this study was a parent-teacher conference scenario, in which the participants interacted with virtual parents. Eye blinks per minute and mid-beta wave (15-20Hz) were measured to analyze cognitive load during their experience, while valence and frontal alpha asymmetry were measured to analyze emotions. The scenario was divided into four stages based upon content progression, and the mean change in the indicators measured was analyzed for each stage. The results showed that cognitive load and positive emotions increased during the first stage, the greeting. However, both decreased sharply during the second stage, discussing problematic behavior. Subsequently, a gradual re-increase pattern was observed during stages 3 and 4. These results confirmed that the patterns of the cognitive load and valence during users’ experience were similar. The significance of this study lies in the fact that it verified the validity of using multimodal measurement methods based upon physiological signals to measure cognitive and emotional aspects of users’ experience in virtual simulation education.

참고문헌 (87)

  1. 김나라, "진로진학상담교사에 대한 학생과 학부모의 인식 및 요구 분석", 실과교육연구, 18(4), pp.249~268 (2012), 한국실과교육연구학회[DOI]
  2. 김성경, "영화클립을 이용한 눈깜빡임과 시각적 주의력과의 상관성 연구", 의공학회지, 35(4), pp.99~104 (2014), 대한의용생체공학회[DOI]
  3. 김은영, "학부모 상담 과정에 관한 초등 학부모의 체험분석", 초등상담연구, 17(2), pp.177~200 (2018), 한국초등상담교육학회[DOI]
  4. 김은희, "진로상담프로그램이 학교폭력 피해중학생의 회복탄력성과 진로결정자기효능감에 미치는 효과", 進路敎育硏究, 29(4), pp.111~133 (2016), 한국진로교육학회
  5. 김한별, "초등학교 저학년 학부모의담임교사 상담경험에 대한 질적 연구", 학습자중심교과교육연구, 18(16), pp.489~508 (2018), 학습자중심교과교육학회[DOI]
  6. 류지헌, "학부모 상담을 위한 온라인 시뮬레이션에서 시나리오 단계에 따른 교사와 예비교사의 주의집중 및 정서변화 대한 차이 분석", 한국교육학연구, 28(2), pp.27~47 (2022), 안암교육학회[DOI]
  7. 류지헌, "가상현실 기반 수업 시뮬레이션의 시나리오 내용이 예비교사의 교사효능감과 가상실재감에 미치는 효과", 교육정보미디어연구, 22(3), pp.661~680 (2016), 한국교육정보미디어학회[DOI]
  8. 방혜진, "중․고등학생의 진로의사결정 단계별주 호소문제 및 상담전략", 進路敎育硏究, 29(2), pp.257~277 (2016), 한국진로교육학회
  9. 송은지, "가상현실 사용자 경험(VR UX)에 대한 연구동향", 정보처리학회지, 29(2), pp.4~13 (2022)
  10. 오인수, "학교폭력의 맥락적 이해에 기초한 효과적인 상담전략", 상담학연구, 17(2), pp.257~279 (2016), 한국상담학회[DOI]
  11. 초등학생용 부모 참여 진로집단상담 프로그램 개발, "오혜영", 한국청소년상담원, ~ (2011)
  12. 우진경, "유아교사의 경험을 통해 본 효과적 학부모 상담을 위한 전제조건과 상담전략", 유아교육학논집, 22(2), pp.259~284 (2018), 한국영유아교원교육학회[DOI]
  13. 우진경, "학부모 상담 실행에서 유아교사의 정서적 경험 탐색", 학습자중심교과교육연구, 22(8), pp.53~71 (2022), 학습자중심교과교육학회[DOI]
  14. 이건형, "유아교육기관 학부모상담 시 전문가상담 도입에 따른 만족도 연구", 뇌교육연구, 24, pp.75~94 (2019)
  15. 이미아, "유아교육기관에서의 상담에 대한 학부모와 유아교사의 인식비교", 석사, pp.2016
  16. 이보미, "중학생의 학교상담요청 결정 과정의 촉진요인 및 방해요인", 청소년상담연구, 23(1), pp.77~96 (2015), 한국청소년상담복지개발원[DOI]
  17. 이슬이, "시각자극 과제에 의한 집중도의 뇌파분석", 석사, 전기공학과, pp.2014
  18. 이주영, "학교폭력 가해자 대상 상담에서의 상담자 경험과 인식에 대한 질적 연구", 한국심리학회지: 상담 및 심리치료, 27(4), pp.849~880 (2015), 한국상담심리학회
  19. 최준섭, "학부모 상담주간의 운영 과정에서 초등학교 교사가 경험하는 정서 변화에 대한 질적 연구", 초등교육연구, 33(1), pp.1~30 (2020), 한국초등교육학회[DOI]
  20. 한형종, "가상현실 기반 교육용 시뮬레이션 설계원리 개발", 교육공학연구, 36(2), pp.221~264 (2020), 한국교육공학회[DOI]
  21. Baceviciute, S., "Remediating learning from non-immersive to immersive media : Using EEG to investigate the effects of environmental embeddedness on reading in Virtual Reality", Computers & Education, 164, pp.104122~ (2021)
  22. Bautista, N. U., "Exploring the impact of TeachME™ lab virtual classroom teaching simulation on early childhood education majors’ self-efficacy beliefs", Journal of Science Teacher Education, 26(3), pp.237~262 (2015)
  23. Blikstein, P., "Multimodal learning analytics and education data mining : Using computational technologies to measure complex learning tasks", Journal of Learning Analytics, 3(2), pp.220~238 (2016)
  24. Choi, T. J., "Determination of the concentrated state using multiple EEG channels", International Journal of Computer, Electrical, Automation, Control and Information Engineering, 8(8), pp.1373~1376 (2014)
  25. Crescenzi-Lanna, L., "Multimodal learning analytics research with young children : A systematic review", British Journal of Educational Technology, 51(5), pp.1485~1504 (2020)
  26. Cukurova, M., "The promise and challenges of multimodal learning analytics", British Journal of Educational Technology, 51(5), pp.1441~1449 (2020)
  27. D’Mello, S., "Dynamics of affective states during complex learning", Learning and Instruction, 22(2), pp.145~157 (2012)
  28. De Coninck, K., "A measurement of student teachers' parent-teacher communication competences : The design of a video-based instrument", Journal of Education for Teaching, 44(3), pp.333~352 (2018)
  29. Di Mitri, D., "Learning pulse: A machine learning approach for predicting performance in selfregulated learning using multimodal data", Proceedings of the 7th International Learning Analytics & Knowledge Conference, pp.188~197 (2017)
  30. Dotger, B. H., "Exploring the emotional geographies of parent–teacher candidate interactions : An emerging signature pedagogy", The Teacher Educator, 46(3), pp.208~230 (2011)
  31. Drachsler, H., "JCAL special issue on multimodal learning analytics", Journal of Computer Assisted Learning, 34(4), pp.335~337 (2018)
  32. Dubovi, I., "Cognitive and emotional engagement while learning with VR : The perspective of multimodal methodology", Computers & Education, 183, pp.104495~ (2022)
  33. Exler, A., "A wearable system for mood assessment considering smartphone features and data from mobile ECGs", Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp.1153~1161 (2016)
  34. Faure, V., "The effects of driving environment complexity and dual tasking on drivers’ mental workload and eye blink behavior", Transportation Research Part F : Traffic Psychology and Behaviour, 40, pp.78~90 (2016)
  35. Fischer, N. L., "Frontal alpha asymmetry and theta oscillations associated with information sharing intention", Frontiers in Behavioral Neuroscience, 12, pp.166~ (2018)
  36. Fortune, E., "Measuring arousal and emotion in healthcare employees using novel devices", Proceedings of the 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pp.835~838 (2020)
  37. Giannakos, M., "Learner-computer interaction", Proceedings of the 10th Nordic Conference on Human-Computer Interaction, pp.968~971 (2018)
  38. Giannakos, M. N., "Multimodal data as a means to understand the learning experience", International Journal of Information Management, 48, pp.108~119 (2019)
  39. Gundel, E., "Self-efficacy in mixed reality simulations : Implications for preservice teacher education", The Teacher Educator, 54(3), pp.244~269 (2019)
  40. Hassib, M., "Brainatwork: Logging cognitive engagement and tasks in the workplace using electroencephalography", Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia, pp.305~310 (2017)
  41. Henry, J. J., "Preparing teacher candidates for successful communication with diverse families using simulations", Teacher Educators' Journal, 15(1), pp.46~76 (2022)
  42. Hökkä, P., "Emotions in learning at work : A literature review", Vocations and Learning, 13, pp.1~25 (2020)
  43. Hsu, W. C., "Using exaggerated feedback in a virtual reality environment to enhance behavior intention of water-conservation", Journal of Educational Technology & Society, 21(4), pp.187~203 (2018)
  44. Huang, J., "FOCUS: enhancing children's engagement in reading by using contextual BCI training sessions", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.1905~1908 (2014)
  45. Huang, Y., "Combining facial expressions and electroencephalography to enhance emotion recognition", Future Internet, 11(5), pp.105~ (2019)
  46. "Eye-Tracking: The complete Pocket Guide" (2017)[URL]
  47. "Facial Expression Analysis: The complete Pocket Guide" (2017)[URL]
  48. Joseph, A. W., "Potential eye tracking metrics and indicators to measure cognitive load in human-computer interaction research", Journal of Scientific Research, 64(1), pp.168~175 (2020)
  49. Kim, S. O., "Comparing concentration levels and emotional states of children using electroencephalography during horticultural and nonhorticultural activities", HortScience, 56(3), pp.324~329 (2021)
  50. Kosmas, P., "Moving bodies to moving minds : A study of the use of motion-based games in special education", TechTrends, 62, pp.594~601 (2018)
  51. Krigolson, O. E., "Objective measurement : Shedding the limitations of self-report in educational research", Learning and Instruction, 53, pp.1~7 (2018)
  52. Kumar, N., "Measurement of cognitive load in HCI systems using EEG power spectrum : an experimental study", Procedia Computer Science, 84, pp.70~78 (2016)
  53. Ledger, S., "Preservice teachers’ confidence and preferred teaching strategies using TeachLivE™ virtual learning environment: A two-step cluster analysis", EURASIA Journal of Mathematics, Science and Technology Education, 15(3), pp.em1674~ (2019)
  54. Ledger, S., "Micro-teaching 2. 0 : Technology as the classroom", Australasian Journal of Educational Technology, 36(1), pp.37~54 (2020)
  55. Lee, D. H., "Reading what the mind thinks from how the eye sees", Psychological Science, 28(4), pp.494~503 (2017)
  56. 이금실, "가상현실 기반 관광체험미디어의 어포던스 특성이 사용자 경험에 미치는 영향", 관광연구저널, 31(3), pp.105~118 (2017), 한국관광연구학회[DOI]
  57. Liu, R., "Learning linkages : Integrating data streams of multiple modalities and timescales", Journal of Computer Assisted Learning, 35(1), pp.99~109 (2019)
  58. Luke, S. E., "Embedding virtual simulation into a course to teach parent-teacher collaboration skills", Intervention in School and Clinic, 57(3), pp.182~188 (2022)
  59. Maffei, A., "Spontaneous blink rate as an index of attention and emotion during film clips viewing", Physiology & Behavior, 204, pp.256~263 (2019)
  60. Medula, C. T., "Simulated apprenticeship for pre-service Filipino teachers", World Journal on Educational Technology : Current Issues, 9(2), pp.89~97 (2017)
  61. Mehlig, L. M., "How is my child doing? : Preparing pre-service teachers to engage parents through assessment", Teaching Education, 24(2), pp.181~194 (2013)
  62. Mills, C., "Put your thinking cap on: detecting cognitive load using EEG during learning", Proceedings of the 7th International Learning Analytics & Knowledge Conference, pp.80~89 (2017)
  63. Mühl, C., "A survey on affective brain-computer interfaces : Psychological, computational, and machine learning perspectives", Journal of Neural Engineering, 17(4), pp.041001~ (2020)
  64. Niforatos, E., "Amplifying human cognition : bridging the cognitive gap between human and machine", Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pp.673~680 (2017)
  65. Ochoa, X., "Augmenting learning analytics with multimodal sensory data?", Journal of Learning Analytics, 3(2), pp.213~219 (2016)
  66. Otamendi, F. J., "The emotional effectiveness of advertisement", Frontiers in Psychology, 11, pp.2088~ (2020)
  67. Peng-Li, D., ""Sound"decisions : The combined role of ambient noise and cognitive regulation on the neurophysiology of food cravings", Frontiers in Neuroscience, 16, pp.827021~ (2022)
  68. Pérez-Fabello, M. J., "How objective measures can improve self-report questionnaires for studying university students’ motivation", International Journal of Environmental Research and Public Health, 17(16), pp.5766~ (2020)
  69. Plass, J. L., "Four ways of considering emotion in cognitive load theory", Educational Psychology Review, 31, pp.339~359 (2019)
  70. Reich, J., "Teaching drills : Advancing practice-based teacher education through short, low-stakes, high-frequency practice", Journal of Technology and Teacher Education, 30(2), pp.217~228 (2022)
  71. Rohrbach, M., "Eye tracking in virtual reality : A comprehensive guide", Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp.1~13 (2020)
  72. Romero, C., "Educational data mining and learning analytics: An updated survey", Wiley Interdisciplinary Reviews : Data Mining and Knowledge Discovery, 10(3), pp.e1355~ (2020)
  73. Prieto, L. P., "Multimodal teaching analytics : Automated extraction of orchestration graphs from wearable sensor data", Journal of Computer Assisted Learning, 34(2), pp.193~203 (2018)
  74. Scarparolo, G., "Mixed-reality simulations as a tool to enhance parent-teacher conferencing in initial teacher education", Australasian Journal of Educational Technology, 38(5), pp.72~86 (2022)
  75. Sharma, K., "Multimodal data capabilities for learning : What can multimodal data tell us about learning?", British Journal of Educational Technology, 51(5), pp.1450~1484 (2020)
  76. Sharma, K., "Building pipelines for educational data using AI and multimodal analytics : A"grey-box"approach", British Journal of Educational Technology, 50(6), pp.3004~3031 (2019)
  77. Sharma, K., "Towards automatic and pervasive physiological sensing of collaborative learning", A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings, 13th International Conference on Computer Supported Collaborative Learning (CSCL), 2, pp.684~687 (2019), International Society of the Learning Sciences
  78. Taub, M., "The impact of contextualized emotions on self-regulated learning and scientific reasoning during learning with a game-based learning environment", International Journal of Artificial Intelligence in Education, 30(1), pp.97~120 (2020)
  79. Teplan, M., "Fundamentals of EEG measurement", Measurement Science Review, 2(2), pp.1~11 (2002)
  80. Thompson, M., "Teacher Moments : A digital simulation for preservice teachers to approximate parent-teacher conversations", Journal of Digital Learning in Teacher Education, 35(3), pp.144~164 (2019)
  81. Tyng, C. M., "The influences of emotion on learning and memory", Frontiers in Psychology, 8, pp.1454~ (2017)
  82. Veneri, D. A., "Electrooculography(EOG)in educational research : Advantages and challenges", Journal of Eye Movement Research, 12(3), pp.1~13 (2019)
  83. Williamson, B., "Brain data : Scanning, scraping and sculpting the plastic learning brain through neurotechnology", Postdigital Science and Education, 1(1), pp.65~86 (2019)
  84. Worsley, M., "A multimodal analysis of making", International Journal of Artificial Intelligence in Education, 28(3), pp.385~419 (2018)
  85. Zaman, B., "Objective and subjective measures of cognitive load : A review", Educational Psychology Review, 32(1), pp.227~251 (2020)
  86. Zheng, W. -L., "Emotionmeter : A multimodal framework for recognizing human emotions", IEEE Transactions on Cybernetics, 99, pp.1~13 (2018)

관련 논문AI추천

이 간행물의 다른 논문 (8건)