A Method for Learning Macro-Actions for Virtual Characters Using Programming by Demonstration and Reinforcement Learning
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
The decision-making by agents in games is commonly based on reinforcement learning. To improve the quality of agents, it is necessary to solve the problems of the time and state space that are required for learning. Such problems can be solved by Macro-Actions, which are defined and executed by a sequence of primitive actions. In this line of research, the learning time is reduced by cutting down the number of policy decisions by agents. Macro-Actions were originally defined as combinations of the same primitive actions. Based on studies that showed the generation of Macro-Actions by learning, Macro-Actions are now thought to consist of diverse kinds of primitive actions. However an enormous amount of learning time and state space are required to generate Macro-Actions. To resolve these issues, we can apply insights from studies on the learning of tasks through Programming by Demonstration (PbD) to generate Macro-Actions that reduce the learning time and state space. In this paper, we propose a method to define and execute Macro-Actions. Macro-Actions are learned from a human subject via PbD and a policy is learned by reinforcement learning. In an experiment, the proposed method was applied to a car simulation to verify the scalability of the proposed method. Data was collected from the driving control of a human subject, and then the Macro-Actions that are required for running a car were generated. Furthermore, the policy that is necessary for driving on a track was learned. The acquisition of Macro-Actions by PbD reduced the driving time by about 16% compared to the case in which Macro-Actions were directly defined by a human subject. In addition, the learning time was also reduced by a faster convergence of the optimum policies.
Abstract
The decision-making by agents in games is commonly based on reinforcement learning. To improve the quality of agents, it is necessary to solve the problems of the time and state space that are required for learning. Such problems can be solved by Macro-Actions, which are defined and executed by a sequence of primitive actions. In this line of research, the learning time is reduced by cutting down the number of policy decisions by agents. Macro-Actions were originally defined as combinations of the same primitive actions. Based on studies that showed the generation of Macro-Actions by learning, Macro-Actions are now thought to consist of diverse kinds of primitive actions. However an enormous amount of learning time and state space are required to generate Macro-Actions. To resolve these issues, we can apply insights from studies on the learning of tasks through Programming by Demonstration (PbD) to generate Macro-Actions that reduce the learning time and state space. In this paper, we propose a method to define and execute Macro-Actions. Macro-Actions are learned from a human subject via PbD and a policy is learned by reinforcement learning. In an experiment, the proposed method was applied to a car simulation to verify the scalability of the proposed method. Data was collected from the driving control of a human subject, and then the Macro-Actions that are required for running a car were generated. Furthermore, the policy that is necessary for driving on a track was learned. The acquisition of Macro-Actions by PbD reduced the driving time by about 16% compared to the case in which Macro-Actions were directly defined by a human subject. In addition, the learning time was also reduced by a faster convergence of the optimum policies.
참고문헌 (13)
- R. Tobi, "Hierarchical Reinforcement Learning on the Virtual Battlefield", pp.2007
- F. S. Melo, "Q-learning with Linear Function Approximation", Lecture Notes in Artificial Intelligence (LNAI), 4539, pp.308~322 (2007)
- N. Ono, "Multi-agent Reinforcement Learning: A Modular Approach", Proceedings of the Second International Conference on Multiagent Systems, Kyoto, December, pp.252~258 (1996)
- A. Mcgovern, "Roles of Macro-Actions in Accelerating Reinforcement Learning", Proceedings of Grace Hopper Celebration of Women in Computing, San Jose, CA, pp.13~18 (1997)
- T. Tateyama, "Automatic Generation of Macro-actions Using Genetic Algorithm for Reinforcement Learning", Proceedings of the 41st SICE Annual Conference, Osaka, August, 1, pp.286~289 (2002)
- J. Randløv, "Learning Macro-actions in Reinforcement Learning", pp.1999
- A. Cypher, "Watch What I Do: Programming by Demonstration" (1993), MIT Press
- N. Koenig, "Behavior-based segmentation of demonstrated task", Proceeding of IEEE 10th International Conference on Development and Learning, Bloomington (2006)
- 성연식, "Programming by Demonstration을 이용한 가상 캐릭터의 행동 생성 기법", 한국게임학회 논문지, 11(2), pp.141~150 (2011), 한국게임학회
- 성연식, "매크로 행동을 이용한 내시 Q-학습의 성능 향상 기법", 멀티미디어학회논문지, 11(3), pp.353~363 (2008), 한국멀티미디어학회
- R. Schoknecht, "Speeding-up Reinforcement Learning with Multi-step Actions", Lecture Notes in Computer Science (LNCS), 2415, pp.813~818 (2002)
- M. J Matarić, "Imitation in Animals and Artifacts : Sensory-Motor Primitives as a Basis for Imitation: Linking Perception to Action and Biology to Robotics", pp.391~422 (2000), MIT Press
- M. N. Nicolescu, "Natural Methods for Robot Task Learning: Instructive Demonstrations, Generalization and Practice", Proceedings of the Second International Joint Conference on Autonomous Agents and Multi-Agent Systems, Melbourne, pp.241~248 (2003)