MediaPipe 모델을 이용한 손-얼굴 혼성 제스처 인터페이스에 관한 연구
A Study on Hand-Face Hybrid Gesture Interface Using MediaPipe Models
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
본 논문은 MediaPipe Hands 모델과 MediaPipe의 Face Mesh 모델을 이용해 얼굴 제스처와 손 제스처를인식한 후, 이들을 결합한 MediaPipe 기반의 혼성 제스처 인터페이스에 관한 것이다. 먼저, 카메라 입력 프레임들에서개별적으로 MediaPipe Hands 모델의 3D 손 랜드마크들과 MediaPipe Face Mesh 모델의 3D 얼굴 랜드마크들을검출해 손과 얼굴의 유무를 판별한 후, 얼굴 커서 위치와 얼굴 제스처, 손 커서 위치와 손 제스처를 인식한다. 이후이들을 사용자 친화적으로 혼용해 손-얼굴 혼성 제스처 기반의 사용자 인터페이스를 구현한다. 제안된 MediaPipe 기반의 손-얼굴 혼성 제스처 인터페이스는 손과 얼굴 중 어느 하나로 제스처 모드가 설정되지만, 모드 전환을 위한 추가제스처 없이 얼굴과 손을 자유롭게 사용해 인터페이스를 제어할 수 있는 것이 장점이다. 또한 윈도우즈 환경에서 소프트웨어 조작 실험을 통해 제안된 손-얼굴 혼성 제스처 인터페이스의 실용성과 유용성을 확인할 수 있었다.
Abstract
This paper describes a hybrid gesture interface based on MediaPipe that recognizes facial gestures and hand gestures using the MediaPipe Hands model and MediaPipe Face Mesh model, and then combines them. First, the presence of hands and faces is determined by individually detecting 3D hand landmarks of MediaPipe Hands model and 3D face landmarks of MediaPipe Face Mesh model from camera input frames, and then the face cursor position and face gestures, as well as the hand cursor position and hand gestures are recognized. Then, these are mixed in a user-friendly way to implement a user interface based on hand-face hybrid gestures. The proposed hand-face hybrid gesture interface based on MediaPipe has the advantage that the gesture mode is set to either the hand or the face, but the interface can be controlled freely using the face and hands without additional gestures for mode switching. In addition, the practicality and usefulness of the proposed hand-face hybrid gesture interface were confirmed through software operation experiments in Windows environment.
참고문헌 (26)
- B. Kumar, "Handbook of Research on AI-Based Technologies and Applications in the Era of the Metaverse", pp.542~358 (2023)
- T. H. Tsai, "Design of Hand Gesture Recognition System for Human-computer Interaction", Multimedia Tools and Applications, 79(9-10), pp.5989~6007 (2020)
- G. Kim, "Real-Time Hand Gesture Recognition Based on Deep Learning", Journal of Korea Multimedia Society, 22(4), pp.424~431 (2019)
- Q. Gao, "Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-robot Interaction", IEEE Sensors Journal, pp.17421~17430 (2021)
- H. Kaur, "A Review : Study of Various Techniques of Hand Gesture Recognition", Proceedings of 2016 IEEE 1st International Conference on Power Electronics, Intelligent Control and Energy Systems, pp.1~5 (2016)
- C. A. Cruz, "Sequential Eyelid Gestures for User Interfaces in VR", Proceedings of 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (2023)
- A. Shimada, "Hand Gesture Based TV Control System—Towards Both User-Machine-friendly Gesture Applications", Proceedings of The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp.121~126 (2013)
- H. Stern, "A Real-Time Hand Gesture Interface for Medical Visualization Applications", Applications of Soft Computing, 36, pp.153~162 (2006)
- G. Pala, "Machine Learning-based Hand Sign Recognition", Proceedings of 2021 International Conference on Artificial Intelligence and Smart Systems, pp.356~363 (2021)
- M. Iskandar, "Artificial Intelligence-based Human Gesture Tracking Control Techniques of Tello EDU Quadrotor Drone", Proceedings of International Conference on Green Energy, Computing and Intelligent Technology (2023)
- N. Qian, "HTML : A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization", Proceedings of the European Conference on Computer Vision, pp.54~71 (2020)
- S. An, "Fast Hand : Fast Monocular Hand Pose Estimation on Embedded Systems", Journal of Systems Architecture, 122 (2022)
- Z. Fan, "Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.1~10 (2021)
- M. Jang, "Implementation of User Gesture Recognition System for Manipulating a Floating Hologram Character", The Journal of the Institute of Internet, Broadcasting and Communication, 19(2), pp.143~149 (2019)
- L. Chen, "MVHM : A Large-Scale multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation", Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition, pp.836~845 (2021)
- "Google MediaPipe"[URL]
- K. Heo, "Hierarchical Hand Pose Model for Hand Expression Recognition", Journal of the Korea Institute of Information and Communication Engineering, 25(10), pp.1323~1329 (2021)
- K. Heo, "Hand Expression Recognition for Virtual Blackboard", Journal of the Korea Institute of Information and Communication Engineering, 25(12), pp.1770~1776 (2021)
- R. Song, "User Interface Using Hand Gesture Recognition Based on MediaPipe Hands Model", Journal of Korea Multimedia Society, 26(2), pp.101~113 (2023)
- J. Prameela, "Mouse Handling Using Facial Gesture", International Research Journal of Modernization in Engineering Technology and Science, 04(5), pp.468~475 (2022)
- S. Sreeni, "Mouse Cursor Control Using Facial Movements-An HCI Application", International Journal of Techno-Engineering, pp.270~274 (2023)
- Z. Sharifisoraki, "A Novel Face Recognition Using Specific Values from Deep Neural Network-based Landmarks", Proceedings of 2023 IEEE International Conference on Consumer Electronics (2023)
- S. Thino, "Developing a Program to Detect Face Direction and the State of Partially Closed Eyes", Master, pp.2023
- J. Mok, "Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh", Journal of Internet of Things and Convergence, 9(6), pp.118~125 (2024)
- J. Park, "Hybrid Gesture based User Interface Using MediaPipe", Proceedings of 2023Fall Annual Conference of Korea Digital Contents Society, pp.34~40 (2023)
- T. Kim, "User Interface Using Face Verification and Hand-Face Hybrid Gesture Recognition", Proceedings of 2024 Winter Annual Conference of Korean Institute of Communication Sciences, 19B(8) (2024)