뇌파, 시선추적 및 인공지능 기술에 기반한 디지털 도서관 인터페이스 연구: 암묵적 적합성 피드백 활용을 중심으로
Digital Library Interface Research Based on EEG, Eye-Tracking, and Artificial Intelligence Technologies: Focusing on the Utilization of Implicit Relevance Feedback
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 연구는 디지털 도서관의 콘텐츠를 탐색하는 동안 이용자의 암묵적 적합성 피드백을 활용하여 적합성을 판단하기 위해 뇌파 기반 및 시선추적 기반 방법들을 제안하고 평가해 보았다. 이를 위해서 32명을 대상으로 하여 동영상, 이미지, 텍스트 데이터를 활용하여 뇌파/시선추적 실험들을 수행하였다. 제안된 방법들의 유용성을 평가하기 위해서, 딥러닝 기반의 인공지능 방법들을 경쟁 기준으로 사용하였다. 평가 결과, 주제에 적합한 동영상과 이미지(얼굴/감정)를 선택하는 데에는 뇌파 컴포넌트 기반 방법들(av_P600, f_P3b)이 높은 분류 정확도를 나타냈고, 이미지(객체)와 텍스트(신문 기사)를 선택하는 데에는 인공지능 기반 방법 즉, 객체 인식 기반 방법과 자연언어 처리 방법이 각각 높은 분류 정확도를 나타냈다. 끝으로, 뇌파, 시선추적 및 인공지능 기술에 기반한 디지털 도서관 인터페이스를 구현하기 위한 지침 즉, 암묵적 적합성 피드백에 기반한 시스템 모형을 제안하고, 분류 정확도를 향상시키기 위해서 미디어별로 적합한 뇌파 기반, 시선추적 기반 및 인공지능 기반 방법들을 제시하였다.
Abstract
This study proposed and evaluated electroencephalography (EEG)-based and eye-tracking-based methods to determine relevance by utilizing users’ implicit relevance feedback while navigating content in a digital library. For this, EEG/eye-tracking experiments were conducted on 32 participants using video, image, and text data. To assess the usefulness of the proposed methods, deep learning-based artificial intelligence (AI) techniques were used as a competitive benchmark. The evaluation results showed that EEG component-based methods (av_P600 and f_P3b components) demonstrated high classification accuracy in selecting relevant videos and images (faces/emotions). In contrast, AI-based methods, specifically object recognition and natural language processing, showed high classification accuracy for selecting images (objects) and texts (newspaper articles). Finally, guidelines for implementing a digital library interface based on EEG, eye-tracking, and artificial intelligence technologies have been proposed. Specifically, a system model based on implicit relevance feedback has been presented. Moreover, to enhance classification accuracy, methods suitable for each media type have been suggested, including EEG-based, eye-tracking-based, and AI-based approaches.
참고문헌 (25)
- 윤정원, "페이스북 건강정보 게시물 형식이 시각적 주의와 인지결과에 미치는 영향", 한국문헌정보학회지, 55(3), pp.219~237 (2021)[DOI]
- Allegretti, M., "When relevance judgement is happening? An EEG-based study", Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 719-722. https :, doi. org, pp.10. 1145, 276646[DOI]
- Bhattacharya, N., "Relevance prediction from eye-movements using semi-interpretable convolutional neural networks", Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, pp.223~233 (2020)
- Borgalli, M. R. A., "Deep learning for facial emotion recognition using custom CNN architecture", Journal of Physics : Conference Series, 2236(1), pp.012004~ (2022)[DOI]
- Davis, K. M., "Contradicted by the brain : Predicting individual and group preferences via brain-computer interfacing", IEEE Transactions on Affective Computing, 14(4), pp.3094~3105 (2022)[DOI]
- DeLong, K. A., "Predictability, plausibility, and two late ERP positivities during written sentence comprehension", Neuropsychologia, 61, pp.150~162 (2014)[DOI]
- Eugster, M. J., "Natural brain-information interfaces : Recommending information by relevance inferred from human brain signals", Scientific Reports, 6(1), pp.38580~ (2016)[DOI]
- Eugster, M. J., "Predicting term-relevance from brain signals", Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pp.425~434 (2014)
- Evans, W. J., "Olfactory event-related potentials in normal human subjects : Effects of age and gender", Electroencephalography and Clinical Neurophysiology, 95(4), pp.293~301 (1995)[DOI]
- Foley, J. J., "Encyclopedia of Information Science and Technology" (2015), IGI Global
- Golenia, J. E., "Implicit relevance feedback from electroencephalography and eye tracking in image search", Journal of Neural Engineering, 15(2), pp.026002~ (2018)[DOI]
- Gwizdka, J., "Differences in eye-tracking measures between visits and revisits to relevant and irrelevant web pages", Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.811~814 (2015)
- Gwizdka, J., "Temporal dynamics of eye-tracking and EEG during reading and relevance decisions", Journal of the Association for Information Science and Technology, 68(10), pp.2299~2312 (2017)[DOI]
- Jacucci, G., "Integrating neurophysiologic relevance feedback in intent modeling for information retrieval", Journal of the Association for Information Science and Technology, 70(9), pp.917~930 (2019)[DOI]
- Kim, H. H., "Video summarization using event-related potential responses to shot boundaries in real-time video watching", Journal of the Association for Information Science and Technology, 70(2), pp.164~175 (2019)[DOI]
- Kim, H. H., "ERP", MMR algorithm for classifying topic-relevant and topicirrelevant visual shots of documentary videos, 2019, pp.70(9), 931~941
- Mayer, R. E., "The Cambridge Handbook of Multimedia Learning", pp.134~146 (2005), Cambridge University Press
- Põld, T., "EEG spectral asymmetry is dependent on education level of men", World Congress on Medical Physics and Biomedical Engineering 2018, 2, pp.405~408 (2018)
- Schindler, S., "Attention and emotion : An integrative review of emotional face processing as a function of attention", Cortex, 130, pp.362~382 (2020)[DOI]
- Shi, Z. F., "Attention evaluation with eye tracking glasses for EEG-based emotion recognition", 8th International IEEE, pp.doi. org, 10. 1109[DOI]
- Udurume, M., "Real-time multimodal emotion recognition based on multithreaded weighted average fusion", Journal of the Ergonomics Society of Korea, 42(5), pp.417~433 (2023)[DOI]
- Ul Haq, H. B., "An effective video summarization framework based on the object of interest using deep learning", Mathematical Problems in Engineering, 2022, pp.7453744~ (2022)[DOI]
- Wang, W., "Learning unsupervised video object segmentation through visual attention", Proceedings of the IEEE, 2019, 3064~3074
- Ye, Z., "Towards a better understanding of human reading comprehension with brain signals", Proceedings of the ACM Web Conference, pp.380~391 (2022)
- Zheng, W. L., "Emotionmeter : a multimodal framework for recognizing human emotions", IEEE Transactions on Cybernetics, 49(3), pp.1110~1122 (2018)[DOI]