Ray RLlib 기반 QMIX와 RND를 이용한 희소 보상 전장 환경에서의 멀티에이전트 강화학습 협업
Cooperative Multi-agent Reinforcement Learning on Sparse Reward Battlefield Environment using QMIX and RND in Ray RLlib
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
멀티에이전트는 전장 교전 상황, 무인 운송 차량 등 다양한 실제 협동 환경에 사용될 수 있다. 전장 교전 상황에서는 도메인 정보의 제한으로 즉각적인 보상(Dense Reward) 설계의 어려움이 있어 명백한 희소 보상(Sparse Reward)으로 학습되는 상황을 고려해야 한다. 본 논문에서는 전장 교전 상황에서의 아군 에이전트 간 협업 가능성을 확인하며, 희소 보상 환경인 Multi-Robot Warehouse Environment(RWARE)를 활용하여 유사한 문제와 평가 기준을 정의하고, 강화학습 라이브러리인 Ray RLlib의 QMIX 알고리즘을 사용하여 학습 환경을 구성한다. 정의한 문제에 대해QMIX의 Agent Network를 개선하고 Random Network Distillation(RND)을 적용한다. 이를 통해 에이전트의 부분 관측값에 대한 패턴과 시간 특징을 추출하고, 에이전트의 내적 보상(Intrinsic Reward) 을 통해 희소 보상 경험 획득 개선이 가능함을 실험을 통해 확인한다.
Abstract
Multi-agent systems can be utilized in various real-world cooperative environments such as battlefield engagements and unmanned transport vehicles. In the context of battlefield engagements, where dense reward design faces challenges due to limited domain knowledge, it is crucial to consider situations that are learned through explicit sparse rewards. This paper explores the collaborative potential among allied agents in a battlefield scenario. Utilizing the Multi-Robot Warehouse Environment(RWARE) as a sparse reward environment, we define analogous problems and establish evaluation criteria. Constructing a learning environment with the QMIX algorithm from the reinforcement learning library Ray RLlib, we enhance the Agent Network of QMIX and integrate Random Network Distillation(RND). This enables the extraction of patterns and temporal features from partial observations of agents, confirming the potential for improving the acquisition of sparse reward experiences through intrinsic rewards.
참고문헌 (13)
- C. G. Lee, "Multi-agent based Manned", unmanned Collaboration System for Combatant’s Battlefield Situation Awareness, 2022, pp.59(2), 126~134
- U. H. Choi, "A Study on Multi-agent Deep Reinforcement Learning for Distributed Cooperative Multi-UAV Mission", Proc. of the KSAS 2021 Fall Conf, pp.958~959 (2021)
- A. P. Pope, "Hierarchical Reinforcement Learning for Air-to-Air Combat", Proc. of the International Confernse on Unmanned Aircraft System (2021)
- M. H. Park, "The Specification of Air-to-Air Combat Tactics Using UML Sequence Diagram", Journal of the KIMST, 24(6), pp.664~675 (2021)[DOI]
- G. Papoudakis, "Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks", Proc. of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) (2021)
- T. Rashid, "QMIX : Monotonic Value Funtion Factorisation for Deep Multi-Agent Reinforcement Learning", Proc. of the 35th International Conference on Machine Learning (2018)
- "Anyscale"[URL]
- M. Samvelyan, "The StarCraft Multi-Agent Challenge", Proc. Deep Reinforcement Learning at the 33rd Conference on Neural Information Processing Systems (NeurIPS) (2019)
- Tan, M, "Multi-Agent Reinforcement Learning Independent vs Cooperative Agents", Proc. of the Tenth International Conference on Machine Learning, pp.330~337 (1993)
- P. Sunehag, "Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward", Proc. of the 17th International Conference on Autonomous Agents and Multiagent Systems (2017)
- "Random network distillation"[URL]
- H. J. Kim, "The Fault Diagnosis Model of Ship Fuel System Equipment Reflecting Time Dependency in Conv1D Algorithm Based on the Convolution Network", J. Navig. Port Res, 46(4), pp.367~374 (2022)[DOI]