언어-기반 제로-샷 물체 목표 탐색 이동 작업들을 위한 인공지능 기저 모델들의 활용
Utilizing AI Foundation Models for Language-Driven Zero-Shot Object Navigation Tasks
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
In this paper, we propose an agent model for Language-Driven Zero-Shot Object Navigation (L-ZSON) tasks, which takes in a freeform language description of an unseen target object and navigates to find out the target object in an inexperienced environment. In general, an L-ZSON agent should able to visually ground the target object by understanding the freeform language description of it and recognizing the corresponding visual object in camera images. Moreover, the L-ZSON agent should be also able to build a rich spatial context map over the unknown environment and decide efficient exploration actions based on the map until the target object is present in the field of view. To address these challenging issues, we proposes AML (Agent Model for L-ZSON), a novel L-ZSON agent model to make effective use of AI foundation models such as Large Language Model (LLM) and Vision-Language model (VLM). In order to tackle the visual grounding issue of the target object description, our agent model employs GLEE, a VLM pretrained for locating and identifying arbitrary objects in images and videos in the open world scenario. To meet the exploration policy issue, the proposed agent model leverages the commonsense knowledge of LLM to make sequential navigational decisions. By conducting various quantitative and qualitative experiments with RoboTHOR, the 3D simulation platform and PASTURE, the L-ZSON benchmark dataset, we show the superior performance of the proposed agent model.
Abstract
In this paper, we propose an agent model for Language-Driven Zero-Shot Object Navigation (L-ZSON) tasks, which takes in a freeform language description of an unseen target object and navigates to find out the target object in an inexperienced environment. In general, an L-ZSON agent should able to visually ground the target object by understanding the freeform language description of it and recognizing the corresponding visual object in camera images. Moreover, the L-ZSON agent should be also able to build a rich spatial context map over the unknown environment and decide efficient exploration actions based on the map until the target object is present in the field of view. To address these challenging issues, we proposes AML (Agent Model for L-ZSON), a novel L-ZSON agent model to make effective use of AI foundation models such as Large Language Model (LLM) and Vision-Language model (VLM). In order to tackle the visual grounding issue of the target object description, our agent model employs GLEE, a VLM pretrained for locating and identifying arbitrary objects in images and videos in the open world scenario. To meet the exploration policy issue, the proposed agent model leverages the commonsense knowledge of LLM to make sequential navigational decisions. By conducting various quantitative and qualitative experiments with RoboTHOR, the 3D simulation platform and PASTURE, the L-ZSON benchmark dataset, we show the superior performance of the proposed agent model.
참고문헌 (16)
- S. Y. Gadre, "Cows on pasture : baselines and benchmarks for language-driven zero-shot object navigation", 2023 IEEE, 2023, 23171~23181
- V. S. Dorbala, "Can an embodied agent find your"Cat-shaped Mug"? LLM-based zero-shot object navigation", IEEE Robotics and Automation Letters, 9(5), pp.4083~4090 (2024)[DOI]
- K. Zhou, "Esc : exploration with soft commonsense constraints for zero-shot object navigation", 40th International Conference on Machine Learning (ICML), pp.42829~42842 (2023)
- M. Minderer, "Simple open-vocabulary object detection", European Conference on Computer Vision (ECCV), pp.728~755 (2022)
- L. H. Li, "Grounded language-image pre-training", 2022IEEE, 2022, 10965~10975
- T. Brown, "Language models are few-shot learners", 34th International Conference on Neural Information Processing Systems, 33, pp.1877~1901 (2020)
- M. Deitke, "RoboTHOR : An open simulation-to-real embodied ai platform", 2020 IEEE, 2020, 3164~3174
- J. Redmon, "You only look once : Unified, real-time object detection", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.779~788 (2016)
- N. Carion, "End-to-end object detection with transformers", European Conference on Computer Vision (ECCV), pp.213~229 (2020)
- A. Dosovitskiy, "An image is worth 16x16words: transformers for image recognition at scale", 2021 International Conference on Learning Representations (ICLR) (2021)
- Z. Liu, "Swin transformer : Hierarchical vision transformer using shifted windows", 2021IEEE, 2021, 10012~10022
- H. Du, "Learning object relation graph and tentative policy for visual navigation", European Conference on Computer Vision (ECCV), pp.19~34 (2020)
- S. Zhang, "Hierarchical object-to-zone graph for object navigation", 2021IEEE, 2021, 15130~15140