Imagine this scenario: you have a robot put the cup on the table into a tray. It has already seen the cup, and the machine arm is stretching past the cup.
For humans, it's good to adjust the position of the hand afterwards. For a robot, things can be much more complicated: it sees the old position before, and the movements that have been generated point to the old position. If it continues to be executed, the hand can reach an empty table.
This little clip points to the problem that robots must face when entering real life: the environment changes as robots operate, and the task must continue to be completed.
How can a robot take a "test"?
In recent years, robots have begun using visual language action models (VLAs): inputting a camera image and instructions such as "putting the cup in the tray" to generate the movements of the control arm. Some models have also introduced world prediction mechanisms, attempting to use prediction to help action in subsequent scenarios.
Judging how well these models perform requires a common set of test questions. Robot evaluation benchmarks provide standardized tasks and simulation environments, allowing different models to be tested under the same conditions and then compare who can complete more tasks.
There have been many robust assessments that change the conditions before the mission starts, such as changing the position of a cup, adjusting the lighting or changing the perspective.
But "the cup is on the other side at the beginning" and "the hand is stretched to half and the cup is taken away" make different demands on the robot. In the latter case, the robot has already acted according to the old picture, and there may be a series of actions that have not yet been performed.
To address this issue, researchers from Stanford, CMU, MIT, UIUC, and UT Austin have proposed LIBERO-MAX, which turns environmental changes in the execution process into repeatable, comparable tests.
This work has been selected for NeurIPS’26 Robotics World Modeling Workshop Oral Report: Yunbei Zhang and Zijian Jin co-authored the first, and co-authors include UT Austin professor Peter Stone, Stanford professor Marco Pavone, MIT postdoctoral fellow Dr. Weirui Ye, and NTU assistant professor Jianfei Yang.
Libero-MAX: Do Robot Policies Adapt When the World Changes?
It was written by Yubei Zhang*, Zijian Jin*, Yuanzhe Liu, Janet Wang, Xilun Zhang, Yuyou Zhang, Zhenyu Zhang, Daoan Zhang, Shuaicheng Niu, Gen Li, Jianfei Yang, Jihun Hamm, Ismini Lourentzou, Weirui Ye, Bo Liu, Peter Stone, and Marco Pavone.
Link to the paper:https://arxiv.org/abs/2609.36518
The code link:https://github.com/liberomax/LIBERO-MAX
The main page of the project:https://liberomax.github.io/
The same robot, two sequels
Back to the story of grabbing the cup. If the robot ultimately fails, how do we know if the cup was taken or if it didn't?
LIBERO-MAX arranges for two executions of the same task -- two executions of shared initial states, instructions, model random seeds, and all actions prior to the change -- at the same critical moment, and then allows them to experience different sequences:
Base: No additional additions to the mid-range changes, and robots continue to execute.
Dynamic: In addition to an environmental change, such as moving target objects, robots continue to attempt to complete tasks.
If a robot succeeds in Base, if it fails in Dynamic, it can recognize that this change destroys a task that could have been completed. Even if it fails both times, it will remain in the results, helping to distinguish between inadequate capabilities and the effects of intermediate changes.
The design, based on the LIBERO-Plus and LIBERO-PRO tasks, constructed 8,000 sets of paired cases and eight classes of execution intermediate changes: moving targets, moving containers, changing camera angles, sensor noise, switching lights, changing visual themes, adding interference and inserting obstacles.
Figure 1 shows the consistency of the two executions before the change and the completion of the task after the change. The eight events cover the position of objects, visual observations, appearance and impurities, and path constraints.
What is the impact of a change in the middle?
The team evaluated 14 current robotic strategies, including VLA, hybrid architecture, and world action models (WAM), using their respective published reasoning configurations. Each strategy completed 16,000 executions, with the main evaluation totaling 224,000 simulations.
The results showed that the success rate of all measurement strategies decreased to 11.0 to 25.7 percentage points.
Figure 2 shows that each line represents a strategy. The empty circle is the success rate of no new intermediate events, and the true square is the success rate of adding changes. All strategies have decreased.
Even if MolmoAct2 and π0.5 performed best under dynamic conditions, success rates of only 66.9% and 65.7%, respectively, lost many of the tasks they could have accomplished. For one of these testing strategies, more than half of the original successes turned into failures after the change.
Pair ratings thus provide two pieces of information: how many tasks can be completed after the change, and which tasks could have been completed before the change was interrupted.
The rankings of different types of models are also intertwined, with no single model family always prevailing. Currently tested world action models also need to face changes in the middle of execution, whether world predictive capabilities can be transformed into reliable actions, and whether they need to be tested through actual missions.
Why is it so hard to move the cup, change the perspective?
The losses caused by the eight types of changes are not the same. The movement of targets or containers, as well as changes in cameras and sensors, cause more obvious common difficulties; lighting and visual subject changes usually affect less.
This formed an intuitive link with the story of grabbing the cup: the cup changed position, the original reaching target needed to be adjusted; the tray moved, the positioning needed to be updated; the camera's perspective changed, and the robot needed to understand its relationship to the object from a new image.
In Figure 3, each circle represents a change in the performance of a strategy. Decomposed by event, the model can be seen in which changes are more likely to fail.
This breakdown allows researchers to further locate questions: should visual recognition be improved, target updates, or action planning?
Can we get the robot to look at it a few more times and fix it?
A natural idea is that if the environment changes, robots will see images more frequently and reproduce their actions.
Robot strategies often predict a series of actions at a time, then perform a number of steps to obtain new observations. This can reduce model calls, but when changes occur, the actions generated based on old images may still be executed.
In the same group of 800 mating cases, the team tested 4 strategies, 18 effective action execution intervals. Adjusting intervals does change performance, but with all test settings, the success rate of changes is still lower than the corresponding Base. More frequent feedback does not automatically eliminate this gap.
Camera control experiments also found that the same perspective changes occurred in the first shot and only occurred halfway through the execution. All three measured strategies were more likely to succeed in mid-change conditions than using a changed perspective from the start.
This indicates that the evaluation also needs to focus on where the robot has gone, what actions it has performed before. Faced with the same new picture, different execution histories will also affect subsequent results.
How many success rates can be traced back after finding the problem?
LIBERO-MAX can also be used to test targeted improvements.
The team selected 300 fixed sensor noise cases and added image quality detection modules before the X-VLA: when the current image quality was found to be poor, they repaired, de-noise, and handed over to strategy processing. The strategy did not need to be retrained, and the detection modules would not receive a "change has occurred" notification in advance.
1 Explain how to compare the original observation, always repair and trigger repair according to image quality.
After quality-triggered repair, the success rate after the change increased from 40.7% to 47.7%, an increase of 7.0 percentage points, while the Base success rate remained basically the same.
This small-scale experiment shows that compensatory measures for specific changes can reverse some of the failures. Coupled evaluation can check at the same time: how much it improves the disrupted tasks and whether it affects the tasks that could have been completed.
Let "The world is changing, can it be done?" be at the heart of robotic assessment.
LIBERO-MAX offers rating codes, fixed case lists, and paired diagnostics, as well as a Lite version containing 800 group cases, which reduces execution to one-tenth of the full version, enabling researchers to quickly verify and then complete the evaluation.
After the robot enters the home, office or factory, environmental changes are accompanied by the continuous execution of tasks. LIBERO-MAX hopes to provide a replicable test platform for change detection, action adjustment, reprogramming and autonomous recovery, so that models can be improved to accommodate the same set of dynamic environmental tests.
Back to the cup that was taken from the beginning: a reliable robot that needs to keep it in the tray after the situation changes.