Exploring the performance impact of soft constraint integration on reinforcement learning-based autonomous vessel navigation: Experimental insights
1
조회
0
다운로드
1
인용
0
참고문헌
키워드
초록
Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain understudied. This research addresses this gap by investigating the implications of soft constraints in the context of the risk-averse ship navigation problem. Four distinct soft constraint functions are proposed, which are integrated with two widely used RL algorithms, resulting in the creation of eight risk-averse autonomous vessel navigation models. To ensure a comprehensive evaluation of their performance, comparative analyses are conducted across seven virtual digital channel environments. Additionally, a novel metric, known as Large Helm Momentum (LHM), is introduced to quantify the smoothness of autonomous vessel navigation. Through thorough experimentation, key considerations for the design of soft constraint functions in the domain of autonomous ship navigation are identified. A comprehensive understanding of how different soft constraint functions influence autonomous driving behavior has been achieved. Key considerations for designing soft constraint functions in the domain of autonomous ship navigation have also been identified. Five principles, namely the constraint association principle, dominance of hard constraints, reward-balance principle, mapping requirement principle, and iterative improvement principle, are proposed to optimize the design of soft constraint functions for autonomous ship navigation, providing valuable guidance and insights.
Abstract
Reinforcement learning has shown promise in enabling autonomous ship navigation, allowing vessels to adapt and make informed decisions in complex marine environments. However, the integration of soft constraints and their impact on performance in RL-based autonomous vessel navigation research remain understudied. This research addresses this gap by investigating the implications of soft constraints in the context of the risk-averse ship navigation problem. Four distinct soft constraint functions are proposed, which are integrated with two widely used RL algorithms, resulting in the creation of eight risk-averse autonomous vessel navigation models. To ensure a comprehensive evaluation of their performance, comparative analyses are conducted across seven virtual digital channel environments. Additionally, a novel metric, known as Large Helm Momentum (LHM), is introduced to quantify the smoothness of autonomous vessel navigation. Through thorough experimentation, key considerations for the design of soft constraint functions in the domain of autonomous ship navigation are identified. A comprehensive understanding of how different soft constraint functions influence autonomous driving behavior has been achieved. Key considerations for designing soft constraint functions in the domain of autonomous ship navigation have also been identified. Five principles, namely the constraint association principle, dominance of hard constraints, reward-balance principle, mapping requirement principle, and iterative improvement principle, are proposed to optimize the design of soft constraint functions for autonomous ship navigation, providing valuable guidance and insights.
참고문헌 (49)
- Bai, X., "A review of current research and advances in unmanned surface vehicles", J. Mar. Sci. Appl, 21(2), pp.47~58 (2022)
- Bellemare, M. G., "Autonomous navigation of stratospheric balloons using reinforcement learning", Nature, 588(7836), pp.77~82 (2020)
- Chen, C., "A knowledge-free path planning approach for smart ships based on reinforcement learning", Ocean Eng, 189, pp.106299~ (2019)
- Chen, C., "Controlling a cargo ship without human experience using deep Q-network", J. Intell. Fuzzy Systems, 39(5), pp.7363~7379 (2020)
- Cheng, Y., "Concise deep reinforcement learning obstacle avoidance for underactuated unmanned marine vessels", Neurocomputing, 272, pp.63~73 (2018)
- Chun, D. H., "Deep reinforcement learning-based collision avoidance for an autonomous ship", Ocean Eng, 234, pp.109216~ (2021)
- Chun, D. H., "Method for collision avoidance based on deep reinforcement learning with path-speed control for an autonomous ship", Int. J. Nav. Archit. Ocean Eng, 16, pp.100579~ (2024)
- Analysis of Regulatory Barriers to the Use of Autonomous Ships, "Danish Maritime Authority", Danish Maritime Authority, 141~ (2017)
- Deng, Y., "Event-triggered output-feedback adaptive tracking control of autonomous underwater vehicles using reinforcement learning", Appl. Ocean Res, 113, pp.102676~ (2021)
- Deraj, R., "Deep reinforcement learning based controller for ship navigation", Ocean Eng, 273, pp.113937~ (2023)
- Dong, Z., "Autonomous cooperative formation control of underactuated USVs based on improved MPC in complex ocean environment", Ocean Eng, 270, pp.113633~ (2023)
- Du, B., "Safe deep reinforcement learning-based adaptive control for USV interception mission", Ocean Eng, 246, pp.110477~ (2022)
- Farazi, N. P., "Deep reinforcement learning in transportation research : A review", Transp. Res. Interdiscip. Perspect, 11, pp.100425~ (2021)
- Feng, S., "Dense reinforcement learning for safety validation of autonomous vehicles", Nature, 615(7953), pp.620~627 (2023)
- Hadi, B., "Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle", Appl. Ocean Res, 129, pp.103326~ (2022)
- Hao, B., "A path planning approach for unmanned surface vehicles based on dynamic and fast Q-learning", Ocean Eng, 270, pp.113632~ (2023)
- Higaki, T., "Human-like route planning for automatic collision avoidance using generative adversarial imitation learning", Appl. Ocean Res, 138, pp.103620~ (2023)
- Kim, J. H., "Collision avoidance based on predictive probability using Kalman filter", Int. J. Nav. Archit. Ocean Eng, 14, pp.100438~ (2022)
- Le, A. V., "Reinforcement learning-based optimal complete water-blasting for autonomous ship hull corrosion cleaning system", Ocean Eng, 220, pp.108477~ (2021)
- Li, J., "LFLD-CLbased NET : A curriculumlearning-based deep learning network with leap-forward-learning-decay for ship detection", J. Mar. Sci. Eng, 11(7), pp.1388~ (2023)
- Li, L., "A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field", Appl. Ocean Res, 113, pp.102759~ (2021)
- Miao, R., "Coordination of distributed unmanned surface vehicles via model-based reinforcement learning methods", Appl. Ocean Res, 122, pp.103106~ (2022)
- Mnih, V., "Human-level control through deep reinforcement learning", Nature, 518(7540), pp.529~533 (2015)
- Qiao, Y., "Survey of deep learning for autonomous surface vehicles in marine environments", IEEE Trans. Intell. Transp. Syst (2023)
- Qu, X., "Pursuit-evasion game strategy of USV based on deep reinforcement learning in complex multi-obstacle environment", Ocean Eng, 273, pp.114016~ (2023)
- Seo, C., "Ship collision avoidance route planning using CRI-based A* algorithm", Int. J. Nav. Archit. Ocean Eng, 15, pp.100551~ (2023)
- Shen, H., "An intelligent collision avoidance and navigation approach of unmanned surface vessel considering navigation experience and rules", J. Harbin Eng. Univ, 39(6), pp.998~1005 (2018)
- Silver, D., "Mastering the game of Go with deep neural networks and tree search", Nature, 529(7587), pp.484~489 (2016)
- Silver, D., "Mastering the game of go without human knowledge", Nature, 550(7676), pp.354~359 (2017)
- Sun, Y., "AUV path following controlled by modified deep deterministic policy gradient", Ocean Eng, 210, pp.107360~ (2020)
- Wang, N., "Data-driven performance-prescribed reinforcement learning control of an unmanned surface vehicle", IEEE Trans. Neural Netw. Learn. Syst, 32(12), pp.5456~5467 (2021)
- Wang, N., "Reinforcement learning-based optimal tracking control of an unknown unmanned surface vehicle", IEEE Trans. Neural Netw. Learn. Syst, 32(7), pp.3034~3045 (2020)
- Wang, P., "Obstacle avoidance for environmentally-driven USVs based on deep reinforcement learning in large-scale uncertain environments", Ocean Eng, 270, pp.113670~ (2023)
- Wang, S., "Adaptive and extendable control of unmanned surface vehicle formations using distributed deep reinforcement learning", Appl. Ocean Res, 110, pp.102590~ (2021)
- Wang, C., "Collision avoidance for autonomous ship using deep reinforcement learning and prior-knowledge-based approximate representation", Front. Mar. Sci, 9, pp.1084763~ (2023)
- Woo, J., "Deep reinforcement learning-based controller for path following of an unmanned surface vehicle", Ocean Eng, 183, pp.155~166 (2019)
- Xu, X., "COLREGs-abiding hybrid collision avoidance algorithm based on deep reinforcement learning for USVs", Ocean Eng, 247, pp.110749~ (2022)
- Xu, X., "Intelligent collision avoidance algorithms for USVs via deep reinforcement learning under COLREGs", Ocean Eng, 217, pp.107704~ (2020)
- Xu, H., "Deep reinforcement learning-based path planning of underactuated surface vessels", Cyber-Phys. Syst, 5(1), pp.1~17 (2019)
- Zhang, Q., "Model-reference reinforcement learning control of autonomous surface vehicles", 2020 59th IEEE Conference on Decision and Control. CDC, pp.5291~5296 (2020), IEEE
- Zhang, X., "Collision-avoidance navigation systems for Maritime Autonomous Surface Ships : A state of the art survey", Ocean Eng, 235, pp.109380~ (2021)
- Zhang, R., "Ship collision avoidance using constrained deep reinforcement learning", 2018 5th International Conference on Behavioral, Economic, and Socio-Cultural Computing. BESC, pp.115~120 (2018), IEEE
- Zhao, Y., "Decision-making for the autonomous navigation of USVs based on deep reinforcement learning under IALA maritime buoyage system", Ocean Eng, 266, pp.112557~ (2022)
- Zhao, Y., "Path following optimization for an underactuated USV using smoothly-convergent deep reinforcement learning", IEEE Trans. Intell. Transp. Syst, 22(10), pp.6208~6220 (2020)
- Zhao, Z., "Safe reinforcement learning for single train trajectory optimization via shield SARSA", IEEE Trans. Intell. Transp. Syst, 24(1), pp.412~428 (2022)
- Zhong, W., "USV path following controller based on DDPG with composite state-space and dynamic reward function", Ocean Eng, 266, pp.112449~ (2022)
