협력적 다중 에이전트 강화학습을 위한 확산모델 기반 데이터 증강기법
Improving Cooperative Multi-Agent Reinforcement Learning through Diffusion-Driven Data Augmentation
1
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
Offline Reinforcement Learning (RL) enables agents to learn from pre-collected data without real-time interaction, offering a safe and cost-effective approach in high-risk fields like autonomous vehicles and healthcare. This method is crucial in Offline Multi-Agent RL (MARL), where robust policies are developed from static datasets. However, dataset biases and limited coverage can impair policy effectiveness. To address this, we study three generative models-variational autoencoders (VAE), generative adversarial networks (GAN), and diffusion models-for data augmentation in MARL. Identifying diffusion models as most effective, we then develop a novel diffusion-based approach, integrating the Q-total function to guide augmentation and optimize global returns without additional training for Q-total value estimation. Focusing on cooperative multi-agent settings where agents work toward shared goals, we demonstrate that diffusion-driven data augmentation significantly enhances learning outcomes in cooperative environments.
Abstract
Offline Reinforcement Learning (RL) enables agents to learn from pre-collected data without real-time interaction, offering a safe and cost-effective approach in high-risk fields like autonomous vehicles and healthcare. This method is crucial in Offline Multi-Agent RL (MARL), where robust policies are developed from static datasets. However, dataset biases and limited coverage can impair policy effectiveness. To address this, we study three generative models-variational autoencoders (VAE), generative adversarial networks (GAN), and diffusion models-for data augmentation in MARL. Identifying diffusion models as most effective, we then develop a novel diffusion-based approach, integrating the Q-total function to guide augmentation and optimize global returns without additional training for Q-total value estimation. Focusing on cooperative multi-agent settings where agents work toward shared goals, we demonstrate that diffusion-driven data augmentation significantly enhances learning outcomes in cooperative environments.
참고문헌 (24)
- S. Fujimoto, "Off-policy deep reinforcement learning without exploration", ~ (2019)
- A. Kumar, "Conservative q-learning for offline reinforcement learning", ~ (2020)
- C. Formanek, "Off-the-grid marl: Datasets with baselines for offline multi-agent reinforcement learning", ~ (2023)
- M. Laskin, "Reinforcement learning with augmented data", Advances in Neural Information Processing Systems, 33, pp.19884~19895 (2020)
- T. Rashid, "Qmix:Monotonic value function factorization for deep multi-agent reinforcement learning", ~ (2018)
- M. Janner, "Planning with diffusion for flexible behavior synthesis", International Conference on Machine Learning (2022)
- Y. Chen, "Boosting continuous control with consistency policy", ~ (2024)
- H. He, "Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning", Advances in Neural Information Processing Systems, 36 (2024)
- W. Li, "Efficient planning with latent diffusion", ~ (2023)
- S. E. Ada, "Diffusion policies for out-of-distribution generalization in offline reinforcement learning", IEEE Robotics and Automation Letters, 9(4), pp.3116~3123 (2024)[DOI]
- J. Ho, "Denoising diffusion probabilistic models", Advances in Neural Information Processing Systems, 33, pp.6840~6851 (2020)
- H. Chen, "Offline reinforcement learning via high-fidelity generative behavior modeling", ~ (2023)
- B. Kang, "Efficient diffusion policies for offline reinforcement learning", Advances in Neural Information Processing Systems, 36 (2024)
- F. Ni, "Metadiffuser: Diffusion model as conditional planner for offline meta-rl", ~ (2023)
- Z. Wang, "Diffusion policies as an expressive policy class for offline reinforcement learning", ~ (2023)
- M. Samvelyan, "The StarCraft Multi-Agent Challenge", ~ (2019)
- L. van der Maaten, "Visualizing data using t-sne", Journal of Machine Learning Research, 9(86), pp.2579~2605 (2008)
- C. Lu, "Synthetic experience replay", Advances in Neural Information Processing Systems, 36 (2024)
- A. Ramesh, "Hierarchical text-conditional image generation with clip latents", ~ (2022)
- R. S. Sutton, "Reinforcement learning : An introduction" (2018), MIT Press
- Z. Zhu, "Madiff: Offline multi-agent learning with diffusion models", ~ (2023)