In recent years, visual language models (Vision-Language Model, VLM) have demonstrated strong capabilities in tasks such as image understanding and semantic reasoning. However, when these models actually enter the physical world, allowing robots to navigate autonomously according to a single natural language instruction still faces a huge challenge.
For example, a robot completes a simple instruction:
She walked through the hallway and found a chair by the window.
Not only does the robot need to understand the phrase, but it also needs to know: where is the corridor? what is the spatial relationship between the windows and the chairs? is the current perspective obscured? where should it move first? how to avoid collisions when approaching the target?
To this end, AgentVLN has proposed the concept of the VLM-as-Brain approach, which allows VLM to be responsible for high-level cognitive decision-making while completing specific executions through a modular skill pool, thus achieving more efficient and flexible robot navigation, namely: VLM Agent → Skill Calling → SLAM/Perception → Robot Navigation.
The paper proposes that AgentVLN performs at the forefront of mainstream VLN benchmarks such as R2R-CE and RxR-CE, while using only 3B-scale models and supporting real-time operation of edge devices such as Jetson.
The title of the paper is AgentVLN: Towards Agentic Vision-and-Language Navigation.
The code link:https://github.com/Allenxinn/AgentVLN
The main page of the project:https://allenxinn.github.io/AgentVLN/
The data collection links to:https://huggingface.co/datasets/allenxinn/AgentVLN-Instruct
The core idea
Agent VLN enables robots to no longer require a large model to do everything directly, but to take responsibility for the visual language model: to understand the objectives of the task, to analyze the environment and to schedule different skill sets. While specific execution functions, such as building maps, avoiding obstacles, moving and positioning, are handed over to specially designed skills modules. The work modeled the navigation process into part of the observable half-Marcoff decision-making process, and divided the skills into perceptual skills and planning skills, allowing intelligent agents to call different dynamics according to the state of the environment. This design is similar to the way humans complete navigation tasks:
Seeing the environment → Thinking paths → Acting → Adjusting according to feedback.
Robots don't need to re-learn every step of the way, but are able to call like humans. The same Agent VLN can replace the corresponding underlying skill modules based on different robots' physical abilities to quickly adapt to different robot platforms.
By designing a plug-in underlying skill library, AgentVLN can adapt to new environments and new tasks by expanding or replacing skills without re-training large-scale visual language models.
The 3D world translates to 2D tips that can be understood by VLM
The biggest advantage of a visual language model is that it understands the semantic information in two-dimensional images. But the problem with robot navigation is that the real world is three-dimensional. The robot needs to know: where should the mouse go?
For this purpose, AgentVLN proposed a cross-space representation mapping mechanism ("Cross-space Representation Mapping"). Its core idea was to use SLAM as a perceptual skill to calculate viable paths in real-world environments, and then to map three-dimensional path information back into two-dimensional image space, allowing VLM to make decisions directly in visual space.
The robot first invokes sensory skills to analyze the current environment: 1) builds three-dimensional spatial maps; 2) obtains environmental topological structures; 3) calculates navigable areas; 4) generates candidate navigation path points. Subsequently, AgentVLN uses a camera model to project these three-dimensional navigable path points into two-dimensional images from the current camera's perspective.
Choose the path point in the image that best meets the language instruction.
In other words, the model no longer needs to predict a precise coordinate directly from open space, but will instead generate the question of: