다중 에이전트 강화 학습을 이용한 트랜스포터 혼잡 회피 경로 계획 최적화
Multi-Agent Reinforcement Learning for Optimizing Path Planning of Transporters Considering the Congestion
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
첨단 기술이 적용된 대규모 유통 센터(mega DC)는 지능형 물류 운영을 실현하는 데 필수적인 역할을 한다. 그러나 다수의 자율 운송 장치가 동적으로 운용되는 대규모 mega DC의 효율적 관리는 기존 경로 계획 방식으로는 효과적으로 대응하기 어렵다. 이러한 문제를 해결하기 위해 본 연구는 다중 에이전트 강화 학습(MARL)과 보조 학습 기법을 결합한 혁신적인 경로 계획 프레임워크를 제안한다. 제안된 프레임워크는 학습의 가속화와 안정성을 위해 특화된 보상 구조와 추가적인 기법들을 도입하여 시스템 전반의 성능을 최적화하는 데 중점을 둔다. 소규모 환경에서 제한된 수의 운송 장치에 대한 경로 최적화를 목표로 MARL 네트워크를 훈련하여 효율성을 검증하였으며, 훈련된 모델 파라미터는 대규모 환경으로의 적용 시 초기 조건으로 활용하여 적응 과정을 가속화하였다. 또한, 복잡한 대규모 환경에서 수렴을 촉진하기 위해 학습을 난이도에 따라 단계적으로 진행하는 커리큘럼 학습 접근법을 적용하였다. 실험 결과, 제안된 접근법은 충돌, 혼잡, 교착 상태와 같은 문제를 효과적으로 해결함으로써 시스템 전체 성능이 유의미하게 향상되는 것을 확인하였다.
Abstract
Advanced mega distribution centers(mega DCs) equipped with cutting-edge technologies are essential for enabling intelligent logistics operations. However, the efficient management of large-scale mega DCs with a dynamic array of autonomous transportation devices presents complexities that traditional path planning approaches struggle to address. To overcome these challenges, this study proposes an innovative path planning framework that integrates Multi-Agent Reinforcement Learning (MARL) with supplementary learning techniques to facilitate accelerated and stable training. This framework incorporates a specialized reward schema alongside auxiliary methodologies aimed at optimizing system-wide performance. Initially, we train MARL networks within a controlled, small-scale environment, focusing on path optimization for a limited number of transportation agents to validate efficiency. The trained model parameters are then leveraged as initial conditions for deployment in a larger-scale environment, thus expediting adaptation. Additionally, we employ a curriculum learning approach, segmenting the training process into levels of progressive difficulty to enhance convergence in complex, large-scale scenarios. Empirical results demonstrate that the proposed approach effectively mitigates operational challenges such as collisions, congestion, and deadlock, leading to significant improvements in overall system performance.
참고문헌 (32)
- Bazzan, A. L., "Opportunities for multiagent systems and multiagent reinforcement learning in traffic control", Autonomous Agents and Multi-Agent Systems, 18, pp.342~375 (2009)
- Bengio, Y., "Curriculum learning", Proceedings of the 26th annual international conference on machine learning, pp.41~48 (2009)
- Boutsioukis, G., "Transfer learning in multi-agent reinforcement learning domains", Recent Advances in Reinforcement Learning: 9th European Workshop, EWRL 2011, pp.249~260 (2011), Springer Berlin Heidelberg
- Chen, X., "A fast two-stage ACO algorithm for robotic path planning", Neural Computing and Applications, 22, pp.313~319 (2013)
- Chen, Z., "Integrated task assignment and path planning for capacitated multi-agent pickup and delivery", IEEE Robotics and Automation Letters, 6(3), pp.5816~5823 (2021)
- Christianos, F., "Shared experience actor-critic for multi-agent reinforcement learning", Advances in neural information processing systems, 33, pp.10707~10717 (2020)
- Dorigo, M., "Ant system: optimization by a colony of cooperating agents", IEEE transactions on systems, man, and cybernetics, part b (cybernetics), 26(1), pp.29~41 (1996)
- Ebben, M., "Container Terminals and Automated Transport Systems: Logistics Control Issues and Quantitative Decision Support", pp.287~306 (2005)
- Fazlollahtabar, H., "Methodologies to optimize automated guided vehicle scheduling and routing problems : a review study", Journal of Intelligent & Robotic Systems, 77, pp.525~545 (2015)
- Fragapane, G., "Planning and control of autonomous mobile robots for intralogistics : Literature review and research agenda", European Journal of Operational Research, 294(2), pp.405~426 (2021)
- Gendreau, M., "Parallel tabu search for real-time vehicle routing and dispatching", Transportation science, 33(4), pp.381~390 (1999)
- Hu, H., "Anti-conflict AGV path planning in automated container terminals based on multi-agent reinforcement learning", International Journal of Production Research, 61(1), pp.65~80 (2023)
- Hussein, A., "Metaheuristic optimization approach to mobile robot path planning", 2012 international conference on engineering and technology (ICET), pp.1~6 (2012), IEEE
- Lowe, R., "Multi-agent actor-critic for mixed cooperative-competitive environments", Advances in neural information processing systems, 30 (2017)
- Lei, X., "Dynamic path planning of unknown environment based on deep reinforcement learning", Journal of Robotics, 2018 (2018)
- Oliehoek, F. A., "Optimal and approximate Q-value functions for decentralized POMDPs", Journal of Artificial Intelligence Research, 32, pp.289~353 (2008)
- Oliehoek, F. A., "A concise introduction to decentralized POMDPs (Vol. 1)" (2016), Springer International Publishing
- Panov, A. I., "Grid path planning with deep reinforcement learning : Preliminary results", Procedia computer science, 123, pp.347~353 (2018)
- Qie, H., "Joint optimization of multi-UAV target assignment and path planning based on multi-agent reinforcement learning", IEEE access, 7, pp.146264~146272 (2019)
- Rashid, T., "Monotonic value function factorisation for deep multi-agent reinforcement learning", The Journal of Machine Learning Research, 21(1), pp.7234~7284 (2020)
- Sedighi, K. H., "Autonomous local path planning for a mobile robot using a genetic algorithm", Proceedings of the 2004 congress on evolutionary computation (IEEE Cat. No. 04TH8753), 2, pp.1338~1345 (2004), IEEE
- Tampuu, A., "Multiagent cooperation and competition with deep reinforcement learning", PloS one, 12(4), pp.e0172395~ (2017)
- Tan, M., "Multi-agent reinforcement learning: Independent vs. cooperative agents", Proceedings of the tenth international conference on machine learning, pp.330~337 (1993)
- Ulusoy, Gündüz, "A genetic algorithm approach to the simultaneous scheduling of machines and automated guided vehicles", Computers & Operations Research, 24(4), pp.335~351 (1997)
- Vinyals, O., "Grandmaster level in StarCraft II using multi-agent reinforcement learning", Nature, 575(7782), pp.350~354 (2019)
- Wang, K. C., "MAPP : a scalable multi-agent path planning algorithm with tractability and completeness guarantees", Journal of Artificial Intelligence Research, 42, pp.55~90 (2011)
- Xue, T., "A reinforcement learning method for multi-AGV scheduling in manufacturing", 2018 IEEE international conference on industrial technology (ICIT), pp.1557~1561 (2018), IEEE
- Multiagent reinforcement learning for multi-robot systems: A survey, "Yang, E.", 1~23 (2004)
- Yang, Y., "Multi‐robot path planning based on a deep reinforcement learning DQN algorithm", CAAI Transactions on Intelligence Technology, 5(3), pp.177~183 (2020)
- Yun, W. J., "Distributed deep reinforcement learning for autonomous aerial eVTOL mobility in drone taxi applications", ICT Express, 7(1), pp.1~4 (2021)
- Wei, G., "COLREGs-compliant multi-ship collision avoidance based on multi-agent reinforcement learning technique", Journal of Marine Science and Engineering, 10(10), pp.1431~ (2022)