멀티-스텝 누적 보상을 활용한 Max-Mean N-Step 시간차 학습
Max-Mean N-step Temporal-Difference Learning Using Multi-Step Return
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
n-스텝 시간차 학습은 몬테카를로 방법과 1-스텝 시간차 학습을 결합한 것으로, 적절한 n을 선택할 경우 몬테카를로 방법과 1-스텝 시간차학습보다 성능이 좋은 알고리즘으로 알려져 있지만 최적의 n을 선택하는 것에 어려움이 있다. n-스텝 시간차 학습에서 n값 선택의 어려움을해소하기 위해, 본 논문에서는 Q의 과대평가가 초기 학습의 성능을 높일 수 있다는 특징과 Q≈Q* 경우, 모든 n-스텝 누적 보상이 비슷한 값을가진다는 성질을 이용하여 1≤k≤n에 대한 모든 k-스텝 누적 보상의 최댓값과 평균으로 구성된 새로운 학습 타겟인 Ω-return을 제안한다. 마지막으로 OpenAI Gym의 Atari 게임 환경에서 n-스텝 시간차 학습과의 성능 비교 평가를 진행하여 본 논문에서 제안하는 알고리즘이 n-스텝시간차 학습 알고리즘보다 성능이 우수하다는 것을 입증한다.
Abstract
n-step TD learning is a combination of Monte Carlo method and one-step TD learning. If appropriate n is selected, n-step TD learningis known as an algorithm that performs better than Monte Carlo method and 1-step TD learning, but it is difficult to select the bestvalues of n. In order to solve the difficulty of selecting the values of n in n-step TD learning, in this paper, using the characteristicthat overestimation of Q can improve the performance of initial learning and that all n-step returns have similar values for Q≈Q* ,we propose a new learning target, which is composed of the maximum and the mean of all k-step returns for 1≤k≤n. Finally, inOpenAI Gym's Atari game environment, we compare the proposed algorithm with n-step TD learning and proved that the proposedalgorithm is superior to n-step TD learning algorithm.
참고문헌 (24)
- R. S. Sutton, "Learning to Predict by the Methods of Temporal Differences", Machine Learning, 3(1), pp.9~44 (1988)
- H. Seijen, "True Online TD(λ)", Proceedings of the 31st International Conference on Machine Learning, 32, pp.692~700 (2014), Proceedings of Machine Learning Research
- K. D. Asis, "Multi-Step Reinforcement Learning: A Unifying Algorithm", Association for the Advancement of Artificial Intelligence (2018)
- S. L. Chen, "Multi-Step Truncated Q Learning Algorithm", 2005 International Conference on Machine Learning and Cybernetics, 1, pp.194~198 (2005)
- K. De Asis, "Per-decision Multi-step Temporal Difference Learning with Control Variates", arXiv:1807.01830, ~ (2018)
- A. R. Mahmood, "Multi-step Offpolicy Learning without Importance Sampling Ratios", arXiv: 1702.03006, ~ (2017)
- L. Yang, "A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning", Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp.2984~2990 (2018)
- C. J. C. H. Watkins, "Q-learning", Machine Learning, 8(3), pp.279~292 (1992)
- V. Mnih, "Playing Atari with Deep Reinforcement Learning", Comment: NIPS Deep Learning Workshop (2013)
- V. Mnih, "Human-level Control through Deep Reinforcement Learning", Nature, 518, pp.529~533 (2015)
- S. Thrun A., "Issues in using function approximation for reinforcement learning", Proceedings of the Fourth Connectionist Models Summer School (1993)
- H. van Hasselt, "Double Q-learning", Advances in Neural Information Processing Systems, 23, pp.2613~2621 (2010)
- H. van Hasselt, "Deep Reinforcement Learning with Double Q-learning", Association for the Advancement of Artificial Intelligence (2016)
- R. S. Sutton, "Reinforcement Learning: An Introduction" (2018), The MIT Press
- J. Peng, "Incremental Multi-Step Q-Learning", Machine Learning, 22(1), pp.283~290 (1996)
- M. Hessel, "Rainbow: Combining Improvements in Deep Reinforcement Learning", Association for the Advancement of Artificial Intelligence, pp.3215~3222 (2018), AAAI Press
- C. J. C. H. Watkins, "Learning from delayed rewards", Doctoral
- Z. Wang, "Dueling Network Architectures for Deep Reinforcement Learning", Proceedings of The 33rd International Conference on Machine Learning, 48, pp.1995~2003 (2016), PMLR
- T. Schaul, "Prioritized Experience Replay", ICLR 2016 (2016)
- S. J. Bradtke, "Reinforcement learning methods for continuous-time Markov decision problems", Proceedings of the 7th International Conference on Neural Information Processing Systems, pp.393~400 (1994), MIT Press
- J. Hernandez-Garcia, "Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target", NIPS Deep Learning Workshop 2018 (2019)
- M. J. Kearns, "Bias-variance error bounds for temporal difference updates", Proceedings of the Thirteenth Annual Conference on Computational Learning Theory, pp.142~147 (2000)
- D. Horgan, "Distributed prioritized experience replay", ICLR 2018 (2018)
- Q. Lan, "Maxmin Q-learning:Controlling the Estimation Bias of Q-learning", ICLR 2020 (2020)