강화학습을 이용한 대전 슈팅 게임의 플랫폼 형태에 따른 레벨 디자인 영향 분석
A Study on the Influence of Platform Design in Level Design by utilizing Multi-agent Reinforcement Learning
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
다중 에이전트 강화학습의 발전과 함께 게임 분야에서 강화학습을 레벨 디자인에 적용하려는 연구가 계속되고 있다. 플랫폼의 형태가 레벨 디자인의 중요한 요소임에도 불구하고 지금까지의 연구들은 플레이어의 스킬 수준이나, 스킬 구성 등 플레이어의 매트릭에 초첨을 맞춰 강화학습을 활용하였다. 따라서 본 논문에서는 레벨 디자인에 플랫폼의 형태가 사용될 수 있도록 시각 센서의 가시성과 구조물의 복잡성을 고려하여 플랫폼이 플레이 경험에 미치는 영향을 연구한다. 이를 위해Unity ML-Agents Toolkit과MA-POCA 알고리즘, Self-play 방식을 기반으로 2vs2 대전 슈팅 게임 환경을 개발하였으며 다양한 플랫폼의 형태를 구성하였다. 분석을 통해 플랫폼의 형태에 따른 가시성과 복잡성의 차이가 승률 밸런스에는 크게 영향을 미치지 않으나 전체 에피소드 수, 무승부 비율, Elo의 증가폭에 유의미한 영향을 미치는 것을 확인했다.
Abstract
As multi-agent reinforcement learning advance, it has been utilized for level design in game with multiple players and complex interactive environments. Despite the significant impact that the platform's design has on game level design, player-related metrics including behavior, skill level, and skill composition have been the primary focus of most reinforcement learning studies. In this paper, we study the impact of platform design on playing experience by utilizing multi-agent reinforcement learning. we use the Unity Engine to create various platform designs based on a 2vs2 shooting game and visual sensor visibility and structural complexity are taken into consideration for analysis. With Unity ML-Agents Toolkit, we construct a reinforcement learning environment based on the MA-POCA algorithm and the self-play method. Though the agent, we analyze changes in play experience with win rate, the number of episode, draw rate, and the Elo rating value. With the study, we figure out that visibility and complexity of platform design exhibit minimal impact on win rate balance in self-play method, but does influence the total number of episodes, draw rate, and the increase in Elo rating value.
참고문헌 (15)
- Hyeon-Chang Jeon, "RaidEnv: Exploring New Challenges in Automated Content Balancing for Boss Raid Games" (2013)
- Conor Stephens, "Balancing Multiplayer Games across Player Skill Levels using Deep Reinforcement Learning", ICAART, 3, pp.827~833 (2022)
- Andrew Cohen, "On the Use and Misuse of Absorbing States in Multi-agent Reinforcement Learning" (2021)
- Arthur Juliani, "Unity : A General Platform for Intelligent Agents" (2018)
- Karina Litwynenko, "Analysis of the possibilities for using machine learning algorithms in the Unity environment", Journal of Computer Sciences Institute, 20, pp.197~204 (2021)
- John Schulman, "Trust Region Policy Optimization", Proceedings of the 32nd International Conference on Machine Learning, PMLR, 37, pp.1889~1897 (2015)
- John Schulman, "Proximal Policy Optimization Algorithms" (2017)
- Bangchu Zhang, "Intelligent Close Air Combat Design based on MA-POCA Algorithm", 2022 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), pp.408~414 (2022)
- Dae-Wook Kim, "Mastering Fighting Game Using Deep Reinforcement Learning With Self-play", 2020 IEEE Conference on Games (CoG), pp.576~583 (2020)
- Inseok Oh, "Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learning", IEEE Transactions on Games, 14(2), pp.212~220 (2021)
- Lauro Victor Ramos Cavadas, "Training human-like bots with Imitation Learning based on provenance data", 2022 21st Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pp.1~6 (2022)
- João Rodrigues, "Character Simulation Using Imitation Learning With Game Engine Physics" (2023)
- Conor Stephens, "Assessing Multiplayer Level Design Using Deep Learning Techniques", Proceedings of the 15th International Conference on the Foundations of Digital Games, pp.1~3 (2020)
- Ahmed Khalifa, "PCGRL: Procedural Content Generation via Reinforcement Learning", Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 16(1), pp.95~101 (2020)
- Tristan Tomilin, "LevDoom: A Benchmark for Generalization on Level Difficulty in Reinforcement Learning", 2022IEEE Conference on Games (CoG), pp.72~79 (2022)