심층 강화 학습을 이용한 Luxo 캐릭터의 제어
Luxo character control using deep reinforcement learning
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
캐릭터로 하여금 시뮬레이션 내에서 사용자가 원하는 동작을 보이도록 물리 기반 제어기를 만들 수 있다면 주변 환경의 변화와 다른 캐릭터와의 상호작용에 대하여 자연스러운 반응을 보이는 캐릭터 애니메이션을 생성할 수 있다. 최근 심층 강화학습을 이용해 물리 기반 제어기가 더 안정적이고 다양한 동작을 합성하도록 하는 연구가 다수 이루어져 왔다. 본 논문에서는 다리가 하나 달린 픽사 애니메이션 스튜디오의 마스코트 캐릭터 Luxo를 주어진 목적지까지 뛰어 도착하게 하는 심층강화학습 모델을 제시한다. 효율적으로 뛰는 동작을 학습하도록 하기 위해서 Luxo의 각 관절의 각도값들을 선형 보간법으로생성하여 참조 모션을 만들었으며, 캐릭터는 이를 모방하면서 균형을 유지하여 목표한 위치까지 도달하도록 하는 제어 정책(control policy)을 학습한다. 참조 동작을 사용하지 않고 Luxo 동작을 제어하도록 학습된 정책과 비교한 실험 결과, 제안된방법을 사용하면 사용자가 지정한 위치로 Luxo가 점프하며 이동하는 정책을 더 효율적으로 학습할 수 있었다.
Abstract
Motion synthesis using physics-based controllers can generate a character animation that interacts naturally with the given environment and other characters. Recently, various methods using deep neural networks have improved the quality of motions generated by physics-based controllers. In this paper, we present a control policy learned by deep reinforcement learning (DRL) that enables Luxo, the mascot character of Pixar animation studio, to run towards a random goal location while imitating a reference motion and maintaining its balance. Instead of directly training our DRL network to make Luxo reach a goal location, we use a reference motion that is generated to keep Luxo animation’s jumping style. The reference motion is generated by linearly interpolating predetermined poses, which are defined with Luxo character’s each joint angle. By applying our method, we could confirm a better Luxo policy compared to the one without any reference motions.
참고문헌 (33)
- K. Yin, "Simbicon : Simple biped locomotion control", ACM Trans. Graph, 26(3), pp.105~ (2007)
- Y. Lee, "Data-driven biped control", ACM SIGGRAPH 2010 Papers, ser. SIGGRAPH ’10 (2010), Association for Computing Machinery
- X. B. Peng, "Deepmimic: Example-guided deep reinforcement learning of physics-based character skills", ACM Trans. Graph., 37(4), pp.143:1~143:14 (2018)
- J. Z. Kolter, "Hierarchical apprenticeship learning, with application to quadruped locomotion", Proceedings of the 20th International Conference on Neural Information Processing Systems, ser. NIPS’07, pp.769~776 (2007), Curran Associates Inc.
- P. Abbeel, "Autonomous helicopter aerobatics through apprenticeship learning", I. J. Robotic Res, 29, pp.1608~1639 (2010)
- N. M. O. Heess, "Emergence of locomotion behaviours in rich environments", ArXiv, vol. abs, 2017
- S. Park, "Learning predictand-simulate policies from unorganized human motion data", ACM Trans. Graph, 38(6) (2019)
- M. de Lasa, "Featurebased locomotion controllers", ACM Trans. Graph, 29(4) (2010)
- S. Agrawal, "Task-based locomotion", ACM Transactions on Graphics (Proc. SIGGRAPH 2016), 35(4) (2016)
- J. M. Wang, "Optimizing walking controllers for uncertain inputs and environments", ACM SIGGRAPH 2010 Papers, ser. SIGGRAPH ’10 (2010)
- K. Wampler, "Generalizing locomotion style to new animals with inverse optimal regression", ACM Trans. Graph, 33(4) (2014)
- R. J. Williams, "Simple statistical gradient-following algorithms for connectionist reinforcement learning", Mach. Learn, 8(3–4), pp.229~256 (1992)
- J. Schulman, "Trust region policy optimization", Proceedings of the 32nd International Conference on International Conference on Machine Learning, 37, pp.1889~1897 (2015)
- J. Schulman, "Proximal policy optimization algorithms" (2017)
- X. B. Peng, "Learning locomotion skills using deeprl: Does the choice of action space matter?", Proceedings of the ACM SIGGRAPH, 2017, ~
- "Learning symmetry and low-energy locomotion" (2018)[URL]
- X. B. Peng, "Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning", ACM Trans. Graph, 36(4) (2017)
- Jungdam Won, "Aerobatics control of flying creatures via self-regulated learning", ACM Transactions on Graphics, 37(6), pp.1~10 (2019), Association for Computing Machinery (ACM)[DOI]
- Andrew Witkin, "Spacetime constraints", ACM SIGGRAPH Computer Graphics, 22(4), pp.159~168 (1988), Association for Computing Machinery (ACM)[DOI]
- K. Yamane, "Animating nonhumanoid characters with human motion data", Proceedings of the 2010 ACM SIGGRAPH, 2010, 169~178[DOI]
- A. Sharma, "Phase-parametric policies for reinforcement learning in cyclic environments", AAAI (2018)
- T. Kwon, "Fast and flexible multilegged locomotion using learned centroidal dynamics", ACM Trans. Graph (2020)
- R. Sutton, "Reinforcement Learning: An Introduction, ser. Adaptive Computation and Machine Learning series" (1998), MIT Press
- J. Schulman, "High-dimensional continuous control using generalized advantage estimation" (2015)
- "stable baselines"[URL]
- S. Coros, "Generalized biped walking control", ACM Transctions on Graphics, 29(4), pp.130~ (2010)
- Igor Mordatch, "Discovery of complex behaviors through contact-invariant optimization", ACM Transactions on Graphics, 31(4), pp.1~8 (2012), Association for Computing Machinery (ACM)[DOI]
- Jie Tan, "Stable Proportional-Derivative Controllers", IEEE Computer Graphics and Applications, 31(4), pp.34~44 (2011), Institute of Electrical and Electronics Engineers (IEEE)[DOI]
- "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations"[URL]
- Yoonsang Lee, "Locomotion control for many-muscle humanoids", ACM Transactions on Graphics, 33(6), pp.1~11 (2014), Association for Computing Machinery (ACM)[DOI]
- D. Sharon, "Synthesis of controllers for stylized planar bipedal walking", Proceedings of the 2005IEEE International Conference on Robotics and Automation, pp.2387~2392 (2005)
- Kevin Bergamin, "DReCon", ACM Transactions on Graphics, 38(6), pp.1~11 (2019), Association for Computing Machinery (ACM)[DOI]
- Kyungho Lee, "Interactive character animation by learning multi-objective control", ACM Transactions on Graphics, 37(6), pp.1~10 (2019), Association for Computing Machinery (ACM)[DOI]