As one of the mainstream routes of physical intelligence, VLA has developed rapidly and has been plagued by its various problems. One of the main pain points is the ability to direct generalization.
Most of the paper's ability to demonstrate is to browse through the consistent data distribution of training and test sets of instructions, resulting in a dramatic decrease in performance after simply rewriting instructions, even for data that now looks simple like LIBERO. As shown below, thevla-adapter has a 51% decrease in performance on LIBERO-para. This is an unacceptable problem for physical AGI, and we expect a robot that can understand the various expressions of human instructions in a variety of environments.
So it's really hard for VLA not to take more than a few points on a fixed-direction template, but to let the same task, in different words, end up on the same set of robot actions. That is, the target object doesn't change, the visual scene doesn't change, the success conditions don't change, just change the language of expression, and the robot should still complete the original task.
A joint team from the Shanghai Intermediate and Infinite Dynamics Intelligent Laboratories has proposed an extremely simple but very effective solution to the problem of VLA language generalization. By redesigning language semantics into visual and motion computation pathways, the model is less dependent on extensive paraphrase training and can more steadily execute rewrite instructions.
Subject: Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
Address of the paper:https://arxiv.org/abs/2608.02497
This is the first time I've seen this.
The instructions change only a few words and the VLA's actions may be completely different.
Currently, many VLA works are constantly refreshing success rates on benchmarks such as LIBERO, RoboTwin and others. However, the language in the standard benchmark is usually more fixed, and the model expresses highly similar instructions seen during training and testing stages.
This overshadows a fundamental question: does the model understand the mission meaning expressed by Instruction, or does it simply label a fixed statement as a task such as 0,1,2?
For example: grab a larger red object; rewrite as: pick up a larger red square. For humans, the two phrases correspond to the same task. But for the VLA, this seemingly minor change could lead to a direct failure of the task that would have been able to stabilize it.
This article first evaluates the three models VLA-Adapter, SmolVLA and π0.5 on LIBERO-Para. LIBERO-Para is a linguistically generic benchmark built on the basis of the LIBERO-Goal task. It keeps the scenarios, objects, initial state and success standards unchanged, only rewriting the expression of movement in the task index, the description of the object, or both, to test whether the model can continue to perform the task correctly if the task does not change, only replacing an equivalent expression.
In other words, VLA performance will drop so much.
The results showed that SmolVLA had a success rate of 72.0% on the original instruction, but only 4.47% when faced with rewriting instruction; the VLA-Adapter dropped from 98.2% to 46.82%.
In other words, a high success rate on standard instructions is not the same as a model with a stable Instruction follow-up capability.
However, the decline in success rates only explains the existence of the problem and does not explain where the problem came from.
For post-training mitigation, a common solution is to continue collecting more equation statements and add rewrite statements to the training. For pre-training, large-scale data training, such as pi, enables the model to see more of a language task form and enhance instruction generalization. However, the paper finds that there may be a simpler solution.
Analysis 1: Maintaining the correct structure of the task in failed moves
In order to determine whether mission information has disappeared, the paper first designed an action-level retrieval experiment.
For the same visual observation, the model first receives a rewriting instruction for the current task, generating an action chunk. Subsequently, the original instruction for the ten LIBERO-Goal tasks is entered with exactly the same visual input, and ten candidate moves are obtained.
Next, it is necessary to judge whether the action generated by rewriting the instruction is the original one closest to the correct task, or closer to other erroneous tasks. If the model has completely misinterpreted the instruction, the action generated by rewriting the instruction should not be steadily pointing to the correct task. On the contrary, if it still contains task-related information, it should be closer to the corresponding original action than the erroneous task.
Retrieval@1 in all three models is significantly higher than the random level (0.1), and Semantic Retention is also all positive. Replication operations at π0.5 are particularly noticeably close to the correct task; the VLA-Adapter and SmolVLA retain a certain task differentiation structure, although the final success rate decreases significantly.
This set of results does not mean that the model fully understands the instructions in every failed episode. But it at least indicates that language rewriting does not make the correct task information completely disappear.
Mission-related information still exists, and problems are more likely to occur when this information is converted into continuous actions.
Analysis 2: Layer-by-Layer Replacement: How language deviation becomes movement deviation
The above experiments can only indicate that the task structure still exists. In order to further determine whether internal language characteristics actually control the movement, the paper made step-by-step characteristic interventions on the model.
In the case of the VLA-Adapter, the same episode runs separately using the original instruction and the rewriting instruction, and the visual input, robot state, initial environment, and action generation conditions in the two runs are all consistent. Before the rewriting instruction enters the final Bridge-Attention block, the experiment does not change the image token or the state token, replacing only the language characteristics received at that location with the language characteristics corresponding to the original instruction.
The results show:
The difference between the forecasted movements was restored by 96.8%;
The success rate of pairing closed rings has increased from 60% to 96%.
The recovery effect of replacing specific layers
This shows that the rewriting instruction does not put the model in a completely unrelated state. Closer to reality is that a certain language deviation is produced between the original instruction and the rewriting instruction, and the downstream Action Expert is very sensitive to this deviation, eventually magnifying the smaller feature difference into different robot actions. To confirm that this location does participate in task control, the paper also carried out a wrong-task intervention: after replacing the language characteristic with the characteristic of another erroneous task, the model's generated action will move to the injected error task deviation.
However, this is not a common feature of all VLAs.
In the VLA-Adapter, a single-layer replacement for the last Bridge-Attention block can recover 96.8% of the motion difference; in the SmolVLA and π0.5, the same replacement for any single-layer can only recover about 10.5% and 31.3% respectively.
This shows that the three types of architecture do not work in the same way: VLA-Adapter sends language information into a relatively independent Bridge-Attention policy, with more centralized control positions; SmolVLA and π0.5 continue to merge language, vision, and state in multi-layer multi-modal computing, with mission information more distributed.
Therefore, the problem of language generalization cannot simply be summarized as "All VLAs' Action Heads fail at some level". Differing information flows in different architectures require changes in the position of semantic repair as well.
Analysis 3: Two control experiments: both dynamic images and phrases disrupt language representations
After confirming that differences in language characteristics would affect the action, the paper continued to track how these differences were generated. The analysis mainly revolved around the auxiliary Qwen branch of the VLA-Adapter. The branch simultaneously receives current images and task instructions, so language tokens are not encoded separately from the text, but are generated in the process of modeling the visual language.
The first experiment was used to isolate the effect of dynamic visual input on language representation. The model went forward and inward twice. The first time forward and inward, Qwen received real images and mission instructions; the second forward and inward, Qwen received the same instructions, but the image was replaced with a fixed image. The second forward-generated language token was then sent back to the computing path of the first forward, while the main visual branch of the robot still used real Observation.
In other words, robots are still able to see the current scene, but no longer allow the task language to communicate with the dynamic images in the Qwen branch.
Analysis 3 overview of the chart
With this processing alone, the VLA-Adapter success rate on the complete LIBERO-Para increased from 46.82% to 61.58%. The same trend can also be observed after replacing the dummy image with a fixed natural image.
The result indicates that in this structure, additional characteristic fluctuations may be introduced when the task description is co-encoded with the dynamic image. For robot movements, these fluctuations do not represent a change in the task itself, but still change the result of subsequent controls.
Analysis 4: Removing the "wording" direction, the closed-loop success rate increased from 55% to 90%.
The second experiment was used to analyze the wording differences between the original instruction and the rewritten instruction.
Using cross-validation, the paper first uses eight tasks to estimate 32 characteristic directions that can distinguish between the original instruction and the rewriting instruction, and then removes these directions from the remaining two unseen task's rewriting instruction characteristics.
After removal:
The difference in movement between the original instruction and the rewriting instruction decreased from 0.4361 to 0.2282;
The success rate for 20 closed-circuit episodes increased from 55% to 90%;
Removing the same number, the random direction of the same parameter, the difference in motion remains essentially unchanged.
This suggests that the equalization instruction does not necessarily cause the model to completely forget the task, but may introduce systematic wording biases in the characteristic space. The downstream Action Expert did not learn to ignore these unrelated changes, but instead mapped them into different actions.
This also constitutes a direct motivation for GSR: instead of continually covering different statements through a large amount of rewriting data, it is better to provide a more stable source of task semantics and then relearn how this part of the semantics affects the action.
Analysis of the 4 overview chart
Solution: GSR rebuilds the flow of information from language to action
Based on the previous analysis, the paper proposes Grounded Semantic Re-Binding, the basic process of GSR.
First, the frozen T5 encoder handles language instructions separately. T5 neither receives images nor robot status, and is only responsible for extracting the text semantics of the task itself, reducing the representational interference from the source caused by dynamic images and wording changes.
Subsequently, the projected T5 semantics were re-injected into the model to integrate task, visual and state information locations based on the various VLAs' original multi-modal calculations. Note that, as described in the above analysis, the convergence locations of different models are different, so the injection locations are not the same.
Finally, re-initialize and train Action Expert to learn how to use new language conditions from the beginning of the training, instead of continuing with the action mapping that has already adapted to the old language path.
GSR architecture as a whole
T5 provides "what tasks need to be completed", the location of the target object, the current state of the robot, and the specific movement trajectory, all of which are still determined by the original visual and state paths of the model.
Therefore, the GSR is not concerned with mere text coding, but with how linguistic semantics can re-enter real-world scenario computation and ultimately control continuous action.
The results: Lightweight models directed to a surge in generalization capabilities.
Bigger models can also continue to benefit from large-scale robotic data pre-training.
All GSR models use only the original instruction training in LIBERO-Goal, with no additional rewriting instruction.
The complete LIBERO-Para contains 4,092 episodes, including 870 motion expressions rewritten, 259 object descriptions rewritten, and 2,963 combined rewritten.
VLA-Adapter's Full Para success rate increased from 46.82% to 70.94%, an increase of 24.12 percentage points; SmolVLA increased from 4.47% to 49.12%, an increase of 44.65 percentage points.
π0.5 has already gained stronger language generalization capabilities through large-scale isomorphic data, but GSR has still pushed its Full Para success rate from 73.60% to 75.59%, achieving the highest PRIDE score reported in the paper, 70.4%. This score is higher than the recently released large-scale pre-training model Xiaomi-Robotics, demonstrating its potential to further enhance the generalization capabilities of the current VLA model instruction.
More importantly, simply adding a T5 next to the native path of the Action Expert can't get the same result. The Native + T5 of the VLA-Adapter increased from just 46.82% to 47.31%; the corresponding version of the SmolVLA was only 13.49%.
Analysis: Path Conflict: After accessing two language sources, who really controls the movement
The paper also designed a set of language-route authority experiments to input correct or incorrect tasks into native language paths and T5 paths, respectively.
C/C indicates that both paths receive the correct instruction; W/C indicates that the native path receives the wrong instruction; T5 receives the correct instruction; C/W indicates the opposite.
The success rate of the VLA-Adapter under C/C was 47.31%. The success rate dropped to 5.11% when the native Qwen path received the wrong task; while the success rate remained 44.0% when the T5 received the wrong task.
This shows that even if Qwen is poorly generalized to rewriting instructions, it still controls the main action. Only by adding a more stable language encoder next to it can't prevent the native path from continuing to dominate the robot. SmolVLA will decrease significantly under both conflict conditions, indicating a clear disturbance between the two language paths.
π0.5 relies heavily on the indigenous PaliGemma, and the pathway itself is already more reliable, so it is suitable to retain indigenous instructions and use T5 as an auxiliary.
This group of results explains why GSRs must adopt architectural-related designs: whether linguistic features exist is one thing, and which pathways actually have motion control is another.
ParaVLA: Exploring the possibilities of the new VLA architecture
In addition to GSR, the paper attempted a more thorough experimental architecture of ParaVLA.
ParaVLA uses a frozen T5 processing instruction and uses DINOv2 visual processing. Language and images remain relatively independent at the front end, eventually merging with robot state and motion information in Action Expert.
Contrasting ParaVLA with the classic VLA architecture
On LIBERO-Goal's original instruction and rewriting instruction, ParaVLA achieved a success rate of 92% and 91%, respectively, with only 1 percentage point difference.
By contrast, after replacing T5 with SmolVLM decoders in the same structure, the original instruction success rate remained at 85%, but the rewriting instruction success rate dropped to 41% (multi-module base modules cannot provide stable semantics). This group of experiments further supported the paper's core observation: a relatively stable, independent of the language conditions of current visual input, helps to reduce the interference of equivalence phrases with motion generation.
However, the complete solution of the puzzle also poses new problems. As the visual model expanded further, ParaVLA did not demonstrate the scaling capabilities common to traditional VLMs. Therefore, future work will continue in this paper to explore how to maintain stable task semantics, adequate scenario grounding, and the expansion capabilities of large-scale multi-modal models at the same time.
From adding language data to redesigning language to controlling movement
The focus of GSR is not to prove that the data expansion has no value, nor simply to replace existing VLMs with T5.
It focuses on a more easily overlooked link: whether language models retain mission information, where that information interacts with vision and states, and which pathways they ultimately control movement through.
For VLA, the failure of language generalization may not always mean that the model completely does not understand the instructions. The information related to the task may still exist, but in the process of multi-modal integration and motion generation, the characteristic deviation between equation expressions is constantly amplified.
Therefore, the real reliable VLAs that need to be addressed are not only the size of the model and the amount of training data, but also: where linguistic semantics enter the model; at what stage it combines with visual and robotic states: which path has control when multiple language paths collide; and whether Action Expert can maintain stability over semantic equivalence and surface language representations.