3D-Pinball 게임 기반 강화학습 알고리즘 및 모델 성능 분석
Performance Analysis of Reinforcement Learning Algorithms and Models Based on the 3D-Pinball Game
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
강화학습을 적용한 게임 연구는 인공지능이 사람보다 뛰어날 수 있다는 것을 입증하였으며, 다양한 방법들이 제안되었다. 본 논문에서는 기존 연구에서 뛰어난 성능을 보인 ‘Video Pinball’과 유사하지만, 화면 및 구성이 더 복잡한 ‘3D-Pinball’에 다양한 모델 구성 및 개선된 강화학습을 적용하였다. 시각적 데이터를 사용하는CNN 기반의 모델과 고차원 데이터에서 적절한 특징을 추출하는 MLP 기반의 모델에 DQN, Double DQN, Dueling Double DQN 총 3가지 강화학습 알고리즘을 ‘3D-Pinball’에 적용하여 성능을 비교 및 평가한다. 실험은 빠른 진행을 위해 게임 플레이 시간 및 게임 목숨에 제한을 두고 진행했으며 실험 결과, MLP 기반의Dueling Double DQN 알고리즘을 적용한 모델이 가장 높은 성능증가율과 게임 성적을 보여 제한된 환경에서는 단순한 모델의 성능이 우수함을 보였다. 따라서, 학습 및 게임 환경의 우수한 성능을 위해서는 제약사항을고려한 모델 개발이 필요할 것으로 보인다.
Abstract
Game research applying reinforcement learning has demonstrated that AI can outperform humans. In this study, we extend this approach to '3D-Pinball', a game similar to 'Video Pinball' but with greater complexity. Three reinforcement learning algorithms - DQN, Double DQN, and Dueling Double DQN - are applied to both CNN-based models using visual data and MLP-based models extracting appropriate features from high-dimensional data. These are then evaluated in '3D-Pinball'. Restrictions on gameplay time and lives were implemented for efficient experimentation. Results showed that the Dueling Double DQN algorithm applied to the MLP-based model yielded the highest performance increase and game scores, underscoring the superior performance of simpler models in restricted environments. Consequently, it appears that model development considering constraints is required for superior performance in learning and game environments.
참고문헌 (13)
- Volodymyr Mnih, "Human-level control through deep reinforcement learning", Nature, 518(7540), pp.529~533 (2015), Springer Science and Business Media LLC[DOI]
- V. Mnih, "Playing Atari with Deep Reinforcement Learning", arXiv preprint arXiv:1312.5602, ~ (2013)
- Christopher J. C. H. Watkins, "Technical Note: Q-Learning", Machine Learning, 8(3-4), pp.279~292 (1992), Springer Science and Business Media LLC[DOI]
- Warren S. McCulloch, "A logical calculus of the ideas immanent in nervous activity", The Bulletin of Mathematical Biophysics, 5(4), pp.115~133 (1943), Springer Science and Business Media LLC[DOI]
- H. van Hasselt, "Double Q-learning", Proceedings of the 24th Annual Conference on Neural Information Processing Systems, pp.2613~2621 (2010)
- H. van Hasselt, "Deep Reinforcement Learning with Double Q-learning", arXiv preprint arXiv:1509.06461, ~ (2015)
- T.P. Lillicrap, "Continuous Control with Deep Reinforcement Learning", arXiv preprint arXiv:1509.02971, ~ (2015)
- Z. Wang, "Dueling Network Architectures for Deep Reinforcement Learning", arXiv preprint arXiv:1511.06581, ~ (2015)
- D.P. Kingma, "Adam: A Method for Stochastic Optimization", International Conference on Learning Representations (ICLR) (2015)
- R. Girshick, "Fast R-CNN", arXiv preprint arXiv:1504.08083, ~ (2015)
- "Imagenet Classification with Deep Convolutional Neural Networks" (2012)[URL]
- David E. Rumelhart, "Learning representations by back-propagating errors", Nature, 323(6088), pp.533~536 (1986), Springer Science and Business Media LLC[DOI]
- C.M. Bishop, "Neural Networks for Pattern Recognition" (1995), Oxford University Press