Reinforcement learning with one-shot memory
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In modern reinforcement learning to solve problems in virtual environments and realities including games, artificial neural networks are used as approximations. However, since the current artificial neural network is based on statistics, a large amount of data is required, so if there is no simulator, there are difficulties in using and applying it. Besides, making an accurate simulator is often a very expensive and hard problem in most domains we want to solve, and getting appropriate data and rewards from the environment is also not easy task, since they are mostly partially observable and relatively scarce. Due to these difficulties of acquiring sufficient data, there are certain limitations in utilizing artificial neural networks as function approximators in reinforcement learning. Therefore, by using the memory structure, which is case-based learning, a model that can perform fast learning in an environment with little data and sparse rewards. In the experiment, we challenged the Open AI Cartpole problem based on the existing policy gradient and memory. The advantage function that evaluates the gain, was implemented by transforming the one-shot learning memory structure. In subsequent experiments, the model showed poor performance on average due to large variance during training. However, by comparing the learning speed with other algorithms, it was confirmed that the top 10 and 5 scores obtained higher scores than other algorithms compared to the same episode within 100 small episodes. In conclusion, we found possibility that the method of using the memory structure can be effective for small data. And in the future, research on techniques to reduce the deviation of learning is needed. The model is expected to help the decision-making of artificial intelligence NPCs interacting with users.
Abstract
In modern reinforcement learning to solve problems in virtual environments and realities including games, artificial neural networks are used as approximations. However, since the current artificial neural network is based on statistics, a large amount of data is required, so if there is no simulator, there are difficulties in using and applying it. Besides, making an accurate simulator is often a very expensive and hard problem in most domains we want to solve, and getting appropriate data and rewards from the environment is also not easy task, since they are mostly partially observable and relatively scarce. Due to these difficulties of acquiring sufficient data, there are certain limitations in utilizing artificial neural networks as function approximators in reinforcement learning. Therefore, by using the memory structure, which is case-based learning, a model that can perform fast learning in an environment with little data and sparse rewards. In the experiment, we challenged the Open AI Cartpole problem based on the existing policy gradient and memory. The advantage function that evaluates the gain, was implemented by transforming the one-shot learning memory structure. In subsequent experiments, the model showed poor performance on average due to large variance during training. However, by comparing the learning speed with other algorithms, it was confirmed that the top 10 and 5 scores obtained higher scores than other algorithms compared to the same episode within 100 small episodes. In conclusion, we found possibility that the method of using the memory structure can be effective for small data. And in the future, research on techniques to reduce the deviation of learning is needed. The model is expected to help the decision-making of artificial intelligence NPCs interacting with users.
참고문헌 (13)
- Richard S. Sutton, "Reinforcement Learning: An Introduction" (2018), A Bradford Book
- Volodymyr Mnih, "Asynchronous methods for deep reinforcement learning", Proceedings of the 33rd International Conference on International Conference on Machine Learning, 48, pp.1928~1937 (2016)
- John Schulman, "Trust Region Policy Optimization", CoRR abs, 2015
- John Schulman, "Proximal Policy Optimization Algorithms", CoRR abs, 2017
- 양의홍, "경영 시뮬레이션 게임에서 PPO 알고리즘을 적용한 강화학습의 유용성에 관한 연구", 한국게임학회 논문지, 19(6), pp.61~70 (2019), 한국게임학회
- 정원조, "사용자와의 협력 플레이를 위한 강화학습 인공지능 프로세스 구축", 한국게임학회 논문지, 20(1), pp.57~66 (2020), 한국게임학회[DOI]
- Schulman, John, "High-dimensional continuous control using generalized advantage estimation", arXiv preprint arXiv:1506.02438, ~ (2015)
- Kaiser, Łukasz, "Learning to remember rare events", arXiv preprint arXiv:1703.03129, ~ (2017)
- Lake, B. M., "Human-level concept learning through probabilistic program induction", Science, 350(6266), pp.1332~1338 (2015)
- L. Bertinetto, "Learning feed-forward oneshot learners", Advances in Neural Information Processing Systems, pp.523~531 (2016)
- "Openai"[URL]
- AG Barto, "Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problem", IEEE Transactions on Systems, Man, and Cybernetics (1983)
- "seungeunrho"[URL]