자율주행 차량 시뮬레이션에서의 강화학습을 위한 상태표현 성능 비교
Comparing State Representation Techniques for Reinforcement Learning in Autonomous Driving
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
딥러닝과 강화학습을 활용한 비전 기반 엔드투엔드 자율주행 시스템 관련 연구가 지속적으로 증가하고 있다. 일반적으로 이러한 시스템은 위치, 속도, 방향, 센서 데이터 등 연속적이고 고차원적인 차량의 상태를 잠재 특징 벡터로 인코딩하고, 이를 차량의 주행 정책으로 디코딩하는 두 단계로 구성된다. 도심 주행과 같이 다양하고 복잡한 환경에서는Variational Autoencoder(VAE)나 Convolutional Neural Network(CNN)과 같은 네트워크를 이용한 효율적인 상태 표현방법의 필요성이 더욱 부각된다. 본 논문은 차량의 이미지 상태 표현이 강화학습 성능에 미치는 영향을 분석하였다. CARLA 시뮬레이터 환경에서 실험을 수행하였고, 차량의 전방 카메라 센서로부터 취득한 RGB 이미지 및 Semantic Segmented 이미지를 각각 VAE와 Vision Transformer(ViT) 네트워크로 특징 추출하여 상태 표현 학습에 활용하였다. 이러한 방법론이 강화학습에 미치는 영향을 실험하여, 데이터 유형과 상태 표현 기법이 자율주행의 학습 효율성과 결정능력 향상에 어떤 역할을 하는지를 실험하였다.
Abstract
Research into vision-based end-to-end autonomous driving systems utilizing deep learning and reinforcement learning has been steadily increasing. These systems typically encode continuous and high-dimensional vehicle states, such as location, velocity, orientation, and sensor data, into latent features, which are then decoded into a vehicular control policy. The complexity of urban driving environments necessitates the use of state representation learning through networks like Variational Autoencoders (VAEs) or Convolutional Neural Networks (CNNs). This paper analyzes the impact of different image state encoding methods on reinforcement learning performance in autonomous driving. Experiments were conducted in the CARLA simulator using RGB images and semantically segmented images captured by the vehicle’s front camera. These images were encoded using VAE and Vision Transformer (ViT) networks. The study examines how these networks influence the agents’ learning outcomes and experimentally demonstrates the role of each state representation technique in enhancing the learning efficiency and decision- making capabilities of autonomous driving systems.
참고문헌 (29)
- C. Chen, "Deepdriving:Learning affordance for direct perception in autonomous driving", Proceedings of the IEEE international conference on computer vision, pp.2722~2730 (2015)
- J. Chen, "Autonomous driving motion planning with constrained iterative lqr", IEEE Transactions on Intelligent Vehicles, 4(2), pp.244~254 (2019)
- B. Paden, "A survey of motion planning and control techniques for self-driving urban vehicles", IEEE Transactions on intelligent vehicles, 1(1), pp.33~55 (2016)
- F. Codevilla, "End-to-end driving via conditional imitation learn-ing", 2018 IEEE international conference on robotics and automation (ICRA), pp.4693~4700 (2018), IEEE
- R. S. Sutton, "Reinforcement learning: An introduction" (2018), MIT press
- I. Higgins, "Darla: Improving zeroshot transfer in reinforcement learn- ing", International Conference on Machine Learning, pp.1480~1490 (2017), PMLR
- A. Kendall, "Learning to drive in a day", 2019 international conference on robotics and automation (ICRA), pp.8248~8254 (2019), IEEE
- A. Dosovitskiy, "Carla: An open urban driving simulator", Con- ference on robot learning, pp.1~16 (2017), PMLR
- D. A. Pomerleau, "Alvinn: An autonomous land vehicle in a neural network", Advances in neural information processing systems, 1 (1988)
- D. Chen, "Learning by cheating", Conference on Robot Learning, pp.66~75 (2020), PMLR
- B. R. Kiran, "Deep reinforcement learning for autonomous driving : A survey", IEEE Transactions on Intel- ligent Transportation Systems, 23(6), pp.4909~4926 (2021)
- V. Mnih, "Asynchronous methods for deep reinforcement learning", International conference on machine learning, pp.1928~1937 (2016), PMLR
- X. Liang, "Cirl: Control- lable imitative reinforcement learning for vision-based self- driving", Proceedings of the European conference on com- puter vision (ECCV), pp.584~599 (2018)
- J. Chen, "Model-free deep reinforcement learning for urban autonomous driving", 2019 IEEE intelligent transportation systems conference (ITSC), pp.2765~2771 (2019), IEEE
- H. Van Hasselt, "Deep reinforcement learning with double q-learning", Proceedings of the AAAI conference on artificial intelligence, 30(1) (2016)
- S. Fujimoto, "Addressing function approximation error in actor-critic methods", International conference on machine learning, pp.1587~1596 (2018), PMLR
- T. Haarnoja, "Soft actor-critic : Off-policy maximum entropy deep reinforce-ment learning with a stochastic actor", International conference on machine learning, pp.1861~1870 (2018), PMLR
- V. Mnih, "Human-level control through deep reinforcement learning", nature, 518(7540), pp.529~533 (2015)