본문 바로가기
KERD
논문KCI 등재

A Survey of Multimodal Systems and Techniques for Motor Learning

Ramin Tadayon|Troy McDaniel|Sethuraman Panchanathan
한국정보처리학회|2017.02|JIPS(Journal of Information Processing Systems) (Journal of Information Processing Systems)|제13권 제1호|pp. 8-25|DOI https://doi.org/10.3745/JIPS.02.0051

1

조회

0

다운로드

1

인용

0

참고문헌

키워드

Augmented Motor Learning and TrainingMultimodal Systems and FeedbackRehabilitative Technologies

초록

This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.

Abstract

This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.

참고문헌 (106)

  1. H. M. Huang, "Investigating learners attitudes toward virtual reality learning environments: based on a constructivist approach", Computers & Education, 55(3), pp.1171~1182 (2010)
  2. M. K. Holden, "Virtual environments for motor rehabilitation : review", CyberPsychology & Behavior, 8(3), pp.187~211 (2005)
  3. B. Dalgarno, "What are the learning affordances of 3-D virtual environments?", British Journal of Educational Technology, 41(1), pp.10~32 (2010)
  4. G. Saposnik, "Virtual reality in stroke rehabilitation a meta-analysis and implications for clinicians", Stroke, 42(5), pp.1380~1386 (2011)
  5. D. Jack, "Virtual reality-enhanced stroke rehabilitation", IEEE Transactions on Neural Systems and Rehabilitation Engineering, 9(3), pp.308~318 (2001)
  6. A. S. Merians, "Virtual reality-augmented rehabilitation for patients following stroke", Physical Therapy, 82(9), pp.898~915 (2002)
  7. M. S. Cameirao, "The rehabilitation gaming system: a virtual reality based system for the evaluation and rehabilitation of motor deficits", Proceedings of the 6th 2007 International Workshop on Virtual Rehabilitation, pp.29~33 (2007)
  8. M. S. Cameirao, "The rehabilitation gaming system : a review", Studies in Health Technology and Informatics, 145, pp.65~83 (2009)
  9. "A virtual environment for post-stroke motor rehabilitation" (2012)[URL]
  10. M. Mihelj, "Virtual rehabilitation environment using principles of intrinsic motivation and game design", Presence, 21(1), pp.1~15 (2012)
  11. A. Rizzo, "A SWOT analysis of the field of virtual reality rehabilitation and therapy", Presence, 14(2), pp.119~146 (2005)
  12. K. Laver, "Virtual reality for stroke rehabilitation", Stroke, 43(2), pp.e20~e21 (2012)
  13. B. Lange, "Game-based telerehabilitation", European Journal of Physical and Rehabilitation Medicine, 45(1), pp.143~151 (2009)
  14. C. Kirner, "Development of an interactive artifact for cognitive rehabilitation based on augmented reality", Proceedings of 2011 IEEE International Conference on Virtual Rehabilitation (ICVR), pp.1~7 (2011)
  15. Y. M. Aung, "AR based upper limb rehabilitation system", Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp.213~218 (2012)
  16. J. C. Perry, "ArmAssist : development of a functional prototype for at-home telerehabilitation of post-stroke arm impairment", Proceedings of 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp.1561~1566 (2012)
  17. H. Kazemi, "Using a robotic interface and haptic feedback to improve grip coordination of hand function following stroke: case study", Proceedings of 2013 IEEE International Conference on Virtual Rehabilitation (ICVR), pp.196~197 (2013)
  18. C. Rodriguez-de-Pablo, "Development of computer games for assessment and training in post-stroke arm telerehabilitation", Proceedings of 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.4571~4574 (2012)
  19. M. Ma, "Serious games for movement therapy after stroke", Proceedings of 2008 IEEE International Conference on Systems, Man and Cybernetics, pp.1872~1877 (2008)
  20. M. J. Johnson, "TheraDrive : a new stroke therapy concept for home-based, computer-assisted motivating rehabilitation", Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.4844~4847 (2004)
  21. R. Ruparel, "Evaluation of the TheraDrive system for robot", computer assisted motivating rehabilitation after stroke, 2009, 811~814
  22. G. Alankus, "Stroke therapy through motion-based games: a case study", Proceedings of 12th International ACM SIGACCESS Conference on Computers and Accessibility, pp.219~226 (2010)
  23. A. Jacobs, "CONTRAST:gamification of arm-hand training for stroke survivors", Proceedings of the 31st Annual CHI Conference on Human Factors in Computing Systems Presence Abstract (CHI’13), pp.415~420 (2013)
  24. G. Alankus, "Reducing compensatory motions in video games for stroke rehabilitation", Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12), pp.2049~2058 (2012)
  25. R. Garris, "Games, motivation, and learning : a research and practice model", Simulation & Gaming, 33(4), pp.441~467 (2002)
  26. K. Salen, "Handbook of Computer Game Studies", pp.59~79 (2005), MIT Press
  27. J. W. Burke, "Optimising engagement for stroke rehabilitation using serious games", The Visual Computer, 25(12), pp.1085~1099 (2009)
  28. D. Johnson, "Effective affective user interface design in games", Ergonomics, 46(13-14), pp.1332~1345 (2003)
  29. J. Wiemeyer, "Serious games in prevention and rehabilitation: a new panacea for elderly people?", European Review of Aging and Physical Activity, 9(1), pp.41~50 (2012)
  30. K. Ogomori, "Requirements for a cognitive training game for elderly or disabled people", Proceedings of 2011 International Conference on Biometrics and Kansei Engineering, pp.150~154 (2011)
  31. E. Flores, "Improving patient motivation in game development for motor deficit rehabilitation", Proceedings of 2008 International Conference on Advances in Computer Entertainment Technology, pp.381~384 (2008)
  32. P. Rego, "Serious games for rehabilitation: a survey and a classification towards a taxonomy", Proceedings of 5th Iberian Conference on Information Systems and Technologies, pp.1~6 (2010)
  33. G. Alankus, "Towards customizable games for stroke rehabilitation", Proceedings of SIGCHI Conference on Human Factors in Computing Systems (CHI’10), pp.2113~2122 (2010)
  34. J. L. Gonzalez Sanchez, "From usability to playability: introduction to playercentred video game development process", Proceedings of 1st International Conference on Human Centered Design, pp.65~74 (2009)
  35. J. W. Burke, "Serious games for upper limb rehabilitation following stroke", Proceedings of 2009 Conference in Games and Virtual Worlds for Serious Applications, pp.103~110 (2009)
  36. A. Gouaich, "Digital-pheromone based difficulty adaptation in post-stroke therapeutic games", Proceedings of 2nd ACM SIGHIT International Health Informatics Symposium, pp.5~12 (2012)
  37. J. Chen, "Flow in games(and everything else)", Communications of the ACM, 50(4), pp.31~34 (2007)
  38. B. Kim, "Not just fun, but serious strategies: using meta-cognitive strategies in gamebased learning", Computers & Education, 52(4), pp.800~810 (2009)
  39. B. Paras, "Game, motivation, and effective learning: an integrated model for educational game design", Proceedings of 2005 Digital Games Research Association's 2nd International Conference: Changing Views: Worlds in Play (2005)
  40. E. Hudlicka, "Affective game engines: motivation and requirements", Proceedings of 4th International Conference on Foundations of Digital Games (FDG’09), pp.299~306 (2009)
  41. K. Tanaka, "A comparison of exergaming interfaces for use in rehabilitation programs and research", Loading, 6(9), pp.69~81 (2012)
  42. G. Saposnik, "Effectiveness of virtual reality using wii gaming technology in stroke rehabilitation a pilot randomized clinical trial and proof of principle", Stroke, 41(7), pp.1477~1484 (2010)
  43. M. R. Mouawad, "Wii-based movement therapy to promote improved upper extremity function post-stroke : a pilot study", Journal of Rehabilitation Medicine, 43(6), pp.527~533 (2011)
  44. B. Lange, "Development and evaluation of low cost game-based balance rehabilitation tool using the Microsoft Kinect sensor", Proceedings of 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp.1831~1834 (2011)
  45. K. LaBelle, "Evaluation of Kinect joint tracking for clinical and in-home stroke rehabilitation tools", Ph.D, pp.2011
  46. Y. J. Chang, "A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities", Research in Developmental Disabilities, 32(6), pp.2566~2570 (2011)
  47. C. Y. Chang, "Towards pervasive physical rehabilitation using Microsoft Kinect", Proceedings of 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, pp.159~162 (2012)
  48. J. E. Deutsch, "Use of a low-cost, commercially available gaming console(Wii)for rehabilitation of an adolescent with cerebral palsy", Physical Therapy, 88(10), pp.1196~1207 (2008)
  49. S. Brosnan, "The potential of Wii-rehabilitation for persons recovering from acute stroke", Physical Disabilities Special Interest Section Quarterly, 32(1), pp.1~3 (2009)
  50. A. Hartveld, "Augmented feedback and physiotherapy practice", Physiotherapy, 82(8), pp.480~490 (1996)
  51. B. Bongers, "Smart Health Applications and Services: Developments and Practices", pp.236~260 (2011), IGI-Global
  52. L. Beursgens, "Playful arm hand training after stroke", Proceedings of 2012 ACM Annual Conference on Human Factors in Computing Systems Extended Abstracts (CHI’12), pp.2399~2404 (2012)
  53. R. Sigrist, "Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning", Experimental Brain Research, 233(3), pp.909~925 (2015)
  54. J. Parker, "The provision of feedback through computer-based technology to promote self-managed post-stroke rehabilitation in the home", Disability and Rehabilitation : Assistive Technology, 9(6), pp.529~538 (2014)
  55. J. Parker, "Stroke patients utilisation of extrinsic feedback from computer-based technology in the home : a multiple case study realistic evaluation", BMC Medical Informatics and Decision Making, 14, pp.1~13 (2014)
  56. J. Parker, "A review of the evidence underpinning the use of visual and auditory feedback for computer technology in post-stroke upper-limb rehabilitation", Disability and Rehabilitation : Assistive Technology, 6(6), pp.465~472 (2011)
  57. R. Sigrist, "Augmented visual, auditory, haptic, and multimodal feedback in motor learning : A review", Psychonomic Bulletin & Review, 20(1), pp.21~53 (2013)
  58. A. Godbout, "Corrective sonic feedback for speed skating: a case study", Proceedings of 16th International Conference on Auditory Display, pp.23~30 (2010)
  59. F. van Wijck, "Making music after stroke : using musical activities to enhance arm function", Annals of the New York Academy of Sciences, 1252(1), pp.305~311 (2012)
  60. I. Wallis, "Real-time sonification of movement for an immersive stroke rehabilitation environment", Proceedings of 13th International Conference on Auditory Display, pp.26~29 (2007)
  61. R. Ronsse, "Motor learning with augmented feedback : modality-dependent behavioral and neural consequences", Cerebral Cortex, 21(6), pp.1283~1294 (2011)
  62. A. Alamri, "Haptic exercises for measuring improvement of post-stroke rehabilitation patients", Proceedings of 2007 IEEE International Workshop on Medical Measurement and Applications, pp.1~6 (2007)
  63. M. H. Milot, "Comparison of erroramplification and haptic guidance training techniques for learning of a timing-based motor task by healthy individuals", Experimental Brain Research, 201(2), pp.119~131 (2010)
  64. M. Mihelj, "A novel paradigm for patient-cooperative control of upper-limb rehabilitation robots", Advanced Robotics, 21(8), pp.843~867 (2007)
  65. D. Feygin, "Haptic guidance: experimental evaluation of a haptic training method for a perceptual motor skill", Proceedings of 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS2002), pp.40~47 (2002)
  66. G. Grindlay, "Haptic guidance benefits musical motor learning", Proceedings of 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp.397~404 (2008)
  67. M. C. Rhoads, "A meta-analysis of visual feedback for motor learning", Athletic Insight, 6(1), pp.17~33 (2014)
  68. S. Moya, "Animation of 3D Avatars for rehabilitation of the upper limbs", Proceedings of 2011 3rd International Conference on Games and Virtual Worlds for Serious Applications, pp.168~171 (2011)
  69. Z. Ruttkay, "Elbows higher! Performing, observing and correcting exercises by a virtual trainer", Proceedings of 8th International Conference on Intelligent Virtual Agents, pp.409~416 (2008)
  70. "Augmented motor skills learning" (2015)[URL]
  71. H. T. Jung, "Towards extended virtual presence of the therapist in stroke rehabilitation", Proceedings of 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), pp.1~6 (2013)
  72. D. L. Eaves, "The short-term effects of real-time virtual reality feedback on motor learning in dance", Presence, 20(1), pp.62~77 (2011)
  73. A. Jaume-i-Capo, "Improving vision-based motor rehabilitation interactive systems for users with disabilities using mirror feedback", The Scientific World Journal, 2014 (2014)
  74. H. Bunt, "Multimodal Human-Computer Communication: Systems, Techniques, and Experiments" (1998), Springer
  75. F. Quek, "Multimodal human discourse : gesture and speech", ACM Transactions on Computer-Human Interaction(TOCHI), 9(3), pp.171~193 (2002)
  76. R. Tadayon, "Interactive motor learning with the autonomous training assistant: a case study", Proceedings of 17th International Conference on Human-Computer Interaction, pp.495~506 (2015)
  77. M. Johnston, "Unification-based multimodal integration", Proceedings of 8th Conference on European Chapter of the Association for Computational Linguistics, pp.281~288 (1997)
  78. L. Wu, "Multimodal integration-a statistical view", IEEE Transactions on Multimedia, 1(4), pp.334~341 (1999)
  79. J. Y. Chai, "A probabilistic approach to reference resolution in multimodal user interfaces", Proceedings of 9th International Conference on Intelligent User Interfaces, pp.70~77 (2004)
  80. H. Mendonca, "A fusion framework for multimodal interactive applications", Proceedings of 2009 International Conference on Multimodal Interfaces, pp.161~168 (2009)
  81. Y. Song, "Multimodal human behavior analysis: learning correlation and interaction across modalities", Proceedings of 14th ACM International Conference on Multimodal Interaction, pp.27~30 (2012)
  82. L. Nigay, "A design space for multimodal systems: concurrent processing and data fusion", Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems, pp.172~178 (1993)
  83. R. S. Schaefer, "Auditory rhythmic cueing in movement rehabilitation : findings and possible mechanisms", Philosophical Transactions of the Royal Society B: Biological Sciences, 369, pp.1~9 (2014)
  84. S. Holland, "A pilot study using tactile cueing for gait rehabilitation following stroke", Proceedings of 2nd International Workshop on ICTs for Improving Patients Rehabilitation Research Techniques, pp.222~223 (2014)
  85. R. Sigrist, "Terminal feedback outperforms concurrent visual, auditory, and haptic feedback in learning a complex rowing-type task", Journal of Motor Behavior, 45(6), pp.455~472 (2013)
  86. H. B. Helbig, "Visual-haptic cue weighting is independent of modality-specific attention", Journal of Vision, 8(1), pp.21~21 (2008)
  87. H. S. Vitense, "Multimodal feedback : an assessment of performance and mental workload", Ergonomics, 46(1-3), pp.68~87 (2003)
  88. S. Oviatt, "When do we interact multimodally?: cognitive load and multimodal communication patterns", Proceedings of 6th International Conference on Multimodal Interfaces, pp.129~136 (2004)
  89. C. Schonauer, "Multimodal motion guidance: techniques for adaptive and dynamic feedback", Proceedings of 14th ACM International Conference on Multimodal Interaction, pp.133~140 (2012)
  90. B. Xiao, "Modeling multimodal integration patterns and performance in seniors: toward adaptive processing of individual differences", Proceedings of 5th International Conference on Multimodal Interfaces, pp.265~272 (2003)
  91. M. Sun, "Effects of multimodal error feedback on human performance in steering tasks", Information and Media Technologies, 6(1), pp.193~201 (2011)
  92. R. G. Carson, "Governing coordination : behavioural principles and neural correlates", Experimental Brain Research, 154(3), pp.267~274 (2004)
  93. C. D. Wickens, "Multiple resources and performance prediction", Theoretical Issues in Ergonomics Science, 3(2), pp.159~177 (2002)
  94. J. L. Burke, "Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis", Proceedings of 8th International Conference on Multimodal Interfaces, pp.108~117 (2006)
  95. M. A. Guadagnoli, "Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning", Journal of Motor Behavior, 36(2), pp.212~224 (2004)
  96. F. Chen, "Multimodal behavior and interaction as indicators of cognitive load", ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4) (2012)
  97. C. H. Anderson, "Neurobiology of Attention", pp.11~17 (2005), Elsevier Academic Press
  98. B. F. Miller, "Encyclopedia and Dictionary of Medicine, Nursing, and Allied Health" (1983), W.B. Saunders
  99. M. S. Sanders, "Human Factors in Engineering and Design" (1987), McGraw-Hill
  100. D. C. Ruspini, "The haptic display of complex graphical environments", Proceedings of 24th Annual Conference on Computer Graphics and Interactive Techniques, pp.345~352 (1997)
  101. M. O. Belardinelli, "Sonification of spatial information: audio-tactile exploration strategies by normal and blind subjects", Proceedings of 13th International Conference on Universal Access in Human-Computer Interaction, pp.557~563 (2009)
  102. S. J. Lederman, "Haptic perception: a tutorial", Attention, Perception, & Psychophysics, 71(7), pp.1439~1459 (2009)
  103. H. Z. Tan, "Optimum information transfer rates for communication through haptic and other sensory modalities", IEEE Transactions on Haptics, 3(2), pp.98~108 (2010)
  104. M. Slater, "How colorful was your day? Why questionnaires cannot assess presence in virtual environments", Presence, 13(4), pp.484~493 (2004)
  105. E. A. Alluisi, "An information analysis of verbal and motor responses in a forcedpaced serial task", Journal of Experimental Psychology, 53(3), pp.153~158 (1957)
  106. H. Z. Tan, "Information transmission with a multifinger tactual display", Perception & Psychophysics, 61(6), pp.993~1008 (1999)

관련 논문AI추천

멀티모달 가이던스가 독보 기능 습득에 미치는 영향: 드럼 타격 시퀀스에서의 사례 연구이인, 최승문한국로봇학회 · 로봇학회 논문지2013 · 제8권 제3호AI in Rehabilitation Medicine: Opportunities and ChallengesJayaraman Arun, Lanotte Francesco, O’Brien Megan K.대한재활의학회 · Annals of Rehabilitation Medicine2023 · 제47권 제6호A Multi-modal Interaction Glove for Immersive Education김상연, 변상규, 이재협, 최동수한국지식정보기술학회 · 한국지식정보기술학회 논문지2015 · 제10권 제6호Neuroscientific Review on Sensory Stimulation Therapy and Virtual Reality for Somatosensory Rehabilitation김요섭, 김태훈한국콘텐츠학회 · International Journal of Contents2010 · 제6권 제1호Advances in Markerless Motion Capture Systems: A Review of OpenCap and Its ApplicationsHyung Suk Yang대한운동학회 · 아시아 운동학 학술지2024 · 제26권 제4호Changes of Motor Control of Distal Joint and Motor Learning in Badminton Hitting Skills According to the Visual Feedback Types (Focused on Performance Curves)서동휘대한운동학회 · 아시아 운동학 학술지2023 · 제25권 제2호

이 간행물의 다른 논문 (14건)

Advanced Communications and Artificial Intelligence, and Introduction of the JIPS AWARD 2016박종혁pp. 1-7Efficient Implementation of the MQTT Protocol for Embedded SystemsOlivier Deschambault, Christian Légaré, Abdelouahed Gherbipp. 26-39Adaptive Medical Image Compression Based on Lossy and Lossless Embedded Zerotree MethodsTaleb-Ahmed Abdelmalik, Sid Ahmed Elhannachi, Nacéra Benamranepp. 40-56An Intuitionistic Fuzzy Approach to Classify the User Based on an Assessment of the Learner’s Knowledge Level in E-Learning Decision-MakingDivakar Yadav, Alka Tripathi, Mukta Goyalpp. 57-67Selective Encryption Algorithm Using Hybrid Transform for GIS Vector MapBang Nguyen Van, 이석환, 권기룡pp. 68-82
원문 보기