Deep reinforcement learning for a multi-objective operation in a nuclear power plant
2
조회
0
다운로드
3
인용
0
참고문헌
키워드
초록
Nuclear power plant (NPP) operations with multiple objectives and devices are still performed manually by operators despite the potential for human error. These operations could be automated to reduce the burden on operators; however, classical approaches may not be suitable for these multi-objective tasks. An alternative approach is deep reinforcement learning (DRL), which has been successful in automating various complex tasks and has been applied in automation of certain operations in NPPs. But despite the recent progress, previous studies using DRL for NPP operations have limitations to handle complex multi-objective operations with multiple devices efficiently. This study proposes a novel DRL-based approach that addresses these limitations by employing a continuous action space and straightforward binary rewards supported by the adoption of a soft actor-critic and hindsight experience replay. The feasibility of the proposed approach was evaluated for controlling the pressure and volume of the reactor coolant while heating the coolant during NPP startup. The results show that the proposed approach can train the agent with a proper strategy for effectively achieving multiple objectives through the control of multiple devices. Moreover, hands-on testing results demonstrate that the trained agent is capable of handling untrained objectives, such as cooldown, with substantial success
Abstract
Nuclear power plant (NPP) operations with multiple objectives and devices are still performed manually by operators despite the potential for human error. These operations could be automated to reduce the burden on operators; however, classical approaches may not be suitable for these multi-objective tasks. An alternative approach is deep reinforcement learning (DRL), which has been successful in automating various complex tasks and has been applied in automation of certain operations in NPPs. But despite the recent progress, previous studies using DRL for NPP operations have limitations to handle complex multi-objective operations with multiple devices efficiently. This study proposes a novel DRL-based approach that addresses these limitations by employing a continuous action space and straightforward binary rewards supported by the adoption of a soft actor-critic and hindsight experience replay. The feasibility of the proposed approach was evaluated for controlling the pressure and volume of the reactor coolant while heating the coolant during NPP startup. The results show that the proposed approach can train the agent with a proper strategy for effectively achieving multiple objectives through the control of multiple devices. Moreover, hands-on testing results demonstrate that the trained agent is capable of handling untrained objectives, such as cooldown, with substantial success
참고문헌 (38)
- D. Lee, "Comparison of deep reinforcement learning and PID controllers for automatic cold shutdown operation", Energies, 15, pp.2834~ (2022)
- C. K. Lee, "Functional Descriptions for Instrumentation and Control System in Nuclear Power Plants" (2003)
- W. Jo, "Bayesian belief network-based human reliability analysis methodology for start-up and shutdown operations in nuclear power plants", Ann. Nucl. Energy, 179, pp.109403~ (2022)
- D. Silver, "Mastering the game of Go without human knowledge", Nature, 550, pp.354~359 (2017)
- O. Vinyals, "Grandmaster level in StarCraft II using multiagent reinforcement learning", Nature, 575, pp.350~354 (2019)
- A. Fawzi, "Discovering faster matrix multiplication algorithms with reinforcement learning", Nature, 610, pp.47~53 (2022)
- E. Prianto, "Path planning for multi-arm manipulators using deep reinforcement learning : soft actorecritic with hindsight experience replay", Sensors, 20, pp.5911~ (2020)
- M. H. Lee, "Deep reinforcement learning-based UAV navigation and control: a soft actor-critic with hindsight experience replay approach", arXiv preprint arXiv:2106.01016, ~ (2021)
- Y. H. Chae, "A methodology for diagnosing FAC induced pipe thinning using accelerometers and deep learning models", Ann. Nucl. Energy, 143, pp.107501~ (2020)
- J. Bae, "Limit surface", states searching algorithm with a deep neural network and Monte Carlo dropout for nuclear power plant safety assessment, 2022, pp.124, 109007~
- S. Ryu, "Probabilistic deep learning model as a tool for supporting the fast simulation of a thermalehydraulic code", Expert Syst. Appl, pp.200~ (2022)
- H. Kim, "Application of a deep learning technique to the development of a fast accident scenario identifier", IEEE Access, 8, pp.177363~177373 (2020)
- 신지현, "Abnormal state diagnosis model tolerant to noise in plant data", Nuclear Engineering and Technology, 53(4), pp.1181~1188 (2021), 한국원자력학회[DOI]
- G. Lee, "A convolutional neural network model for abnormality diagnosis in a nuclear power plant", Appl. Soft Comput, 99 (2021)
- J. Choi, "A sensor fault-tolerant accident diagnosis system", Sensors, 20, pp.1~17 (2020)
- 김재민, "Abnormality diagnosis model for nuclear power plants using two-stage gated recurrent units", Nuclear Engineering and Technology, 52(9), pp.2009~2016 (2020), 한국원자력학회[DOI]
- 채영호, "Graph neural network based multiple accident diagnosis in nuclear power plants: Data optimization to represent the system configuration", Nuclear Engineering and Technology, 54(8), pp.2859~2870 (2022), 한국원자력학회[DOI]
- 류승형, "Development of deep autoencoder-based anomaly detection system for HANARO", Nuclear Engineering and Technology, 55(2), pp.475~483 (2023), 한국원자력학회[DOI]
- J. Ahn, "Deep learning-based procedure compliance check system for nuclear power plant emergency operation", Nucl. Eng. Des, pp.370~ (2020)
- J. Bae, "Real-time prediction of nuclear power plant parameter trends following operator actions", Expert Syst. Appl, pp.186~ (2021)
- 강정성, "Concept of an intelligent operator support system for initial emergency responses in nuclear power plants", Nuclear Engineering and Technology, 54(7), pp.2453~2466 (2022), 한국원자력학회[DOI]
- M. I. Radaideh, "Physics-informed reinforcement learning optimization of nuclear assembly design", Nucl. Eng. Des, 372, pp.110966~ (2021)
- J. Park, "Control automation in the heat-up mode of a nuclear power plant using reinforcement learning", Prog. Nucl. Energy, 145, pp.104107~ (2022)
- D. Lee, "Algorithm for autonomous power-increase operation using deep reinforcement learning and a rule-based system", IEEE Access, 8, pp.196727~196746 (2020)
- 김재민, "Strategy to coordinate actions through a plant parameter prediction model during startup operation of a nuclear power plant", Nuclear Engineering and Technology, 55(3), pp.839~849 (2023), 한국원자력학회[DOI]
- T. Haarnoja, "Soft actor-critic algorithms and applications", arXiv preprint arXiv:1812.05905, ~ (2018)
- M. Andrychowicz, "Hindsight experience replay", Adv. Neural Inf. Process. Syst, pp.30~ (2017)
- R. S. Sutton, "Reinforcement Learning: an Introduction" (2018), MIT press
- V. Mnih, "Human-level control through deep reinforcement learning", nature, 518, pp.529~533 (2015)
- T. Haarnoja, "Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor", International Conference on Machine Learning, PMLR, pp.1861~1870 (2018)
- J. Bae, "An Autonomous Pressure Controller Based on Approximation of Action Value Function", Transactions of the Korean Nuclear Society (2020)
- W. Zhang, "Soft ACTOR-CRITIC reinforcement learning for robotic manipulator with hindsight experience replay Tao yan", Int. J. Robot Autom, 34 (2019)
- Compact Nuclear Simulator and its Upgrade Plan, "K. -C. Kwon", ~ (1997)
- J. Miettinen, "Development and assessment of the SBLOCA code SMABRE", Proceedings of the CSNI Specialists' Meeting on Small Break LOCA Analyses in LWRs, pp.23~27 (1985)
- C. R. Harris, "Array programming with NumPy", Nature, 585, pp.357~362 (2020)
- M. Abadi, "Tensorflow: large-scale machine learning on heterogeneous distributed systems", arXiv preprint arXiv:1603.04467, ~ (2016)
- J. H. Shin, "An interpretable convolutional neural network for nuclear power plant abnormal events", Appl. Soft Comput, 132, pp.109792~ (2023)
- J. Garcıa, "A comprehensive survey on safe reinforcement learning", J. Mach. Learn. Res, 16, pp.1437~1480 (2015)