AdaRoboVLG: Adaptive visual language crawling framework for tasks for different mechanics
It's also a cup, where should the robot catch it?
If the task is to pour water into the cistern, holding the cup handle is usually more appropriate; if the cup is to be delivered to someone else, the robot is better to leave the cup handle to the other; if the cup is still moving on the transmission belt, the position must be updated over time.
The goal of Vision-Language-Grasping, VLG, is to allow robots to make crawling based on natural language and specific contexts that correspond to mission intentions. This requires simultaneous processing of space, cognition, timing and the challenges of the four types: identifying targets and avoiding collisions, understanding the use of objects, tracking the state of continuous change, and adapting to different mechanics.
Current VLG methods use end-to-end routing, directly mapping visual observations and language instructions to capture action. The adaptability of these strategies is often limited by trained data and manual configuration; new task situations and entities often require data replenishment or retraining.
Researchers from institutions such as the Chinese University of Science and Technology, Beijing University of Technology, and Linglong Intelligent have proposed AdaRoboVLG. The framework unravels "how to grab a task" and "how to grab a machine" and uses unified crawler interface to connect task understanding and physical crawling: the upper layer transforms the task situation into a crawler constraint, and the lower layer generates a stable posture for different machine hands. Thus, new capabilities can access the same crawler base, forming a scalable robotic crawler route.
Title: Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
It was written by Sixu Yan, Shikang Wang, Binhua Huang, Xuanlai Tang, Guohua Fan, Fan Huang, Haoxuan Li, Yongkang Li, Yuhan Li, Bencheng Liao, Zeyu Zhang, Wenyu Liu, Hangxin Liu, Xinggang Wang.
The paper:
What are you doing here?
The code is:https://github.com/AdaRoboVLG/AdaRoboVLG
It is a simulation.
01 to understand and grab and tear apart,
Let different mechanics serve in the same basement
Humans are able to flexibly adjust crawling when challenges such as shadowing, functional needs and target movement arise at the same time, not because they have experienced every possible situation, but because they call on the rich precedent formed by common sense and past experience to synchronize perception, reasoning and action control. The basic model exhibits similarities to these precedents in visual recognition, spatial understanding, functional reasoning and target tracking. Inspired by this, the Ada RoboVLG will transform the model precedent provided by the base into a combination of crawling constraints, and then hand over to a cross-mechanical hand replication strategy, generating a mission-intentional and physically achievable crawling gesture.
Human use of common sense and experience adjustment crawling; combined robotic vision, division and language model precedent
A crawling constraint is defined as a set of unified crawling interfaces composed of the geometry of the target object, the contact grasp representation (CGR) and the crawling type compatible with the mechanist. The CGR describes the location of the contact, the probability, the approach to closing direction and the crawling width; the crawling type is classified from human crawling, and is filtered based on the mechanist's ability to move.
General framework for AdaRoboVLG: combining pioneering strategies through structured interface connectivity
The basic strategy is to generate candidates for specific mechanics based on the CGR and the crawling type; stability scores are then given by the shared model. Hand-Object Interaction Representation, HOI) encodes the location of the contact points, provides local geometry information for the closure of the associated stability predictions. Different mechanics generate candidates according to their own structure and are responsible for scoring the shared model.
In Isaac Sim, the team used DH3, Allegro, and Inspire, three mechanical hands, and collected 4.4 million marked crawling experiments to train basic crawling strategies.
Simulation of data collection processes for basic strategies
Experiment One verified learning efficiency and cross-hand prediction. HOI converges faster and less training losses compared to independent coders and objects; three-handed combined-training models achieved 82.95% and 88.02% and 91.07% crawling quality prediction accuracy on DH3, Allegro, and Inspire respectively.
Basic strategic experiments: trained clutches (left) and cross-mechanical forecasting accuracy (right)
02 Three types of predecessors, respectively
"Can't catch, how to catch, how to follow".
In addition to basic strategies, space, cognition and timing pioneers are responsible for transforming mission situations into interface constraints. Experiments 2 through 4 examine each capability separately to provide the basis for the real system combination that follows.
Space pioneers: Can mechanics reach into the garbage?
A gesture may collide with the surrounding object even if it is able to grasp the target. Space pioneers use multi-point color and depth (RGB-D) observation to raise the image characteristics extracted from DINOv3 to three-dimensional, forming a scenic-level semantic representation; then combine point clouds to provide access candidates as a base strategy through Contact-GraspNet's predictive scenic-level CGR.
Space pioneers: DH3, Allegro, Inspire captured visualization in six scenarios.
Contact-GraspNet in space modules is trained on the GraspClutter 6D and then used for the following benchmarks:
The AdaRoboVLG has an average success rate of 88.4% in DexGraspNet 2.0's three levels of clutter and three mechanical hands. It has the best chart results on the Random, Loose, and Dense scores are still higher.
DexGraspNet 2.0: How to Get Successful
AnyDexGrasp* maintains the original candidate generation process and uses this article to share scores for differentiation models.
Cognitive pioneering: first infer functions, then verify objectives
Functional crawling is both an understanding of task needs, but also a positioning, splitting of corresponding parts, and generating physically feasible crawling gestures. The paper crawled two complementary experiments through open cluster functional reasoning and language-guiding objectives, respectively, to test reasoning ability and landing ability.
Task A: Open Cluster Function Reasoning. Below are the results of 16 objects: given task instructions and input images, reasoning on the corresponding functional parts and crawling types.
Open Cluster Function Logic: 16 items with a choice of parts and hands
The open set evaluation contained 125 cluster images with instructions for 57 operational tasks and 80 objects. Functional sections of the basic LLM, crawling type accuracy of 82% and 63%, respectively; increased to 94% and 87% after joining RAG and CoT. In this cluster fusion experiment, CoT improved functional section judgement more significantly, and RAG was more helpful for crawling type selection.
Open Cluster Functionality Reasoning and Melting Experiments
Task B: Language-guided target grabbing. Under the same cognitive module, target positioning and splitting results, compare whether different methods can generate successful target grabbing gestures.
The goal of language guidance is to capture: Allegro four-fingered mechanic.
AdaRoboVLG's three-handed average success rate on GraspClutter6D and GraspNet-1Billion was 81.3% and 86.0%, respectively, with the highest success rate on both of the two benchmarks and the three mechanical hands in the table.
Language guides target capture: two sets of data, three mechanical hands
Each method uses the same cognitive reasoning modules; AnyDexGrasp* maintains the original candidate generation process and uses this article to share scores for differentiation models.
Timeline: After the target moves, grab the interface continuously updated
Good gestures calculated in static scenarios may fail quickly after the target moves. The time sequence precursor combines SAM3's mask tracking with DINOv3's cross-sectional characteristic consistency, estimates the target's rigid motion, and updates the target's geometry and CGR. The module itself does not require additional training.
In GraspNet-1Billion's Seen, Similar, and Novel scores, AdaRoboVLG is at the top of the three indicators for multiple tracking accuracy, parallel error, and rotation error.
This is a dynamic tracking: Seen / Similar / Novel
Long-term tracking of representative objects: projection of three-dimensional semantic features in continuous images
The results of the experiments and long-term visualization show that the AdaRoboVLG has a significant advantage in long-range tracking, being able to maintain a stable association during the continuous movement of the target, and continuously updating the tracking constraints. However, symmetrical objects may still appear to rotate, and severe shadowing may lead to tracking loss.
03 From static turmoil to dynamic interference:
Authentic machine-verified combined capability
Previous experiments examined each linkage, real-machine experiments further examined whether they could work in sync. Experiment 5 started from static chaos scenes, first combining space with cognitive pioneers, and then joining timeline pioneers to process movement goals; experiment 6 examined additional tasks and expanded capabilities after sensing modules.
Find the right target and grab the right parts: Experiment 5 A)
The test covers 102 everyday objects, including food, household items, tools, toys, electronics, and textiles. Each test involves identifying targets, fulfilling functional intentions, avoiding collisions, and lifting objects 10 cm high, keeping them off for 5 seconds.
On the XMAN-R1 and TianJi Marvin platform equipped with a ROHand, each object underwent five randomized trials, with a total of 510 real-crawling attempts, with an overall success rate of 83.3%. The research team also deployed the system in parallel clutches to demonstrate its adaptability to cross-end executors.
Functional grappling guided by language in real-life mess scenes
XMAN-R1 + ROHand: Functional gripping in a real mess scene.
TianJi Marvin + ROHand: Functional grabbing in a real mess scene.
After the target moves, figure out how to keep up.
Mobile transmission bands make "catch where" a time-changing problem. The experiment uses the AgileX Piper equipped with the ROHand to select 16 representative objects and demonstrate functional crawling at a speed of 2.5 cm/s. Space and cognitive pioneers construct initial crawling constraints, timing pioneers continuously update according to the target movement, and grab gesture updates online at a frequency of 5 Hz.
The following six task groups demonstrate how the system keeps its grip in line with the language's intent as it moves toward the target.
Dynamic transmission bands: tennis, sneakers, red microphones, green toy cars, pumps, water pumps are shown simultaneously for six task groups.
Subsequently, the team used parallel claws on the same mechanical arm to test the robustness of human crawling under dynamic interference. The overall crawling success rate of the system was 89.7% in the face of four types of interference, such as removal of field of view recovery, rapid movement, continuous shadowing, parallel movement and rotation.
There are four types of human interference: removal of the field of view, recovery, rapid movement, continuous shading, flattening and rotation.
Access to new modules, expansion of tasks and perception capabilities (experiment 6)
After adding the task assignment module, the AdaRoboVLG can perform two-arm sorting; after accessing the full module, it can also process transparent objects. Both extensions change the task organization and perceptual input, respectively, without the need to re-train the basics of capturing strategies.
These two demonstrations make scalable meaningful: new capabilities do not have to be written all into the same strategy, but can be accessed via modules, taken over by existing crawlers, and provide a viable path for expansion of tasks and perceptual capabilities.
The two arms collaborated in sorting.
04 Not only to make big models, but also to expand the capacity to capture
AdaRoboVLG's value lies not only in individual success rates, but also in making tasks adaptable, strategies replicable, and intrinsically adaptable. Structured interfaces connect task constraints to physical clutches, leaving room for more powerful perception and reasoning models to be accessed later.
From a broader systemic point of view, the Scaling of robotic abilities depends not only on the size of the model, but also on the synergy between perception, reasoning, skills and the body. AdaRoboVLG connects basic model pioneers, reusable crawling skills with different mechanics, and shows how to promote the expansion of robotic abilities through a combination of abilities.
Modularization still has boundaries. If the upper layer identifies faulty functional parts, the lower layer may generate steady but unsuitable grappling; multicolored positioning inconsistencies, deep rebuilding noise and lack of touch feedback, also limit reliability.
From different mechanical hands to static, dynamic real-world scenarios, the AdaRoboVLG provides a concrete example of scalable robotic crawling. This design, which combines task understanding and physical crawling, is expected to transform the continuous advancement of the basic model into robotic capabilities more efficiently, opening up new development space for multi-tasking and intrinsic robotic systems.
