Self-Imitation Learning을 이용한 개선된 Deep Q-Network 알고리즘
Improved Deep Q-Network Algorithm Using Self-Imitation Learning
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Self-Imitation Learning은 간단한 비활성 정책 actor-critic 알고리즘으로써 에이전트가 과거의 좋은 경험을 활용하여 최적의 정책을 찾을 수 있도록 해준다. 그리고 actor-critic 구조를 갖는 강화학습 알고리즘에 결합되어 다양한 환경들에서 알고리즘의 상당한 개선을 보여주었다. 하지만 Self-Imitation Learning이 강화학습에 큰 도움을 준다고 하더라도 그 적용 분야는actor-critic architecture를 가지는 강화학습 알고리즘으로 제한되어 있다. 본 논문에서 Self-Imitation Learning의 알고리즘을 가치 기반 강화학습 알고리즘인 DQN에 적용하는 방법을 제안하고, Self-Imitation Learning이 적용된 DQN 알고리즘의학습을 다양한 환경에서 진행한다. 아울러 그 결과를 기존의 결과와 비교함으로써 Self-Imitation Leaning이 DQN에도 적용될 수 있으며 DQN의 성능을 개선할 수 있음을 보인다.
Abstract
Self-Imitation Learning is a simple off-policy actor-critic algorithm that makes an agent find an optimal policy byusing past good experiences. In case that Self-Imitation Learning is combined with reinforcement learning algorithmsthat have actor-critic architecture, it shows performance improvement in various game environments. However, itsapplications are limited to reinforcement learning algorithms that have actor-critic architecture. In this paper, we proposea method of applying Self-Imitation Learning to Deep Q-Network which is a value-based deep reinforcement learningalgorithm and train it in various game environments. We also show that Self-Imitation Learning can be applied to DeepQ-Network to improve the performance of Deep Q-Network by comparing the proposed algorithm and ordinary DeepQ-Network training results.
참고문헌 (13)
- Sutton, Richard S, "Reinforcement learning: An introduction" (2018), MIT press
- Kuleshov, Volodymyr, "Algorithms for multi-armed bandit problems" (2014)
- Oh, Junhyuk, "Self-imitation learning", International Conference on Machine Learning (2018), PMLR
- Mnih, V, "Human-level control through deep reinforcement learning", Nature, 0518, pp.529~533 (2015)
- Watkins, Christopher JCH, "Q-learning", Machine learning, 8(3-4), pp.279~292 (1992)
- Van Hasselt, Hado, "Deep reinforcement learning with double q-learning", Proceedings of the AAAI conference on artificial intelligence, 30(1) (2016)
- Wang, Ziyu, "Dueling network architectures for deep reinforcement learning", International conference on machine learning (2016), PMLR
- Schaul, Tom, "Prioritized experience replay" (2015)
- Fortunato, Meire, "Noisy networks for exploration" (2017)
- Andrychowicz, Marcin, "Hindsight experience replay" (2017)
- Mnih, Volodymyr, "Asynchronous methods for deep reinforcement learning", International conference on machine learning (2016), PMLR
- Schulman, John, "Proximal policy optimization algorithms" (2017)