A mechanical arm is reaching for the sushi on the turntable. At this point, is it steadily approaching the target, or has it missed the best time to grab?
However, in the visual-language-action model reinforcement learning (VLA-RL) methodology, the Critic Model, which is responsible for evaluating the current state of the critical model, relies primarily on single-mode observations or single-mode VLM characteristics to score. This is like letting a referee who only looks at a match screenshot without feedback judge whether an athlete has completed a critical move or has lost his balance.
Recently, the OpenMOSS team opened up the World Critic Model (WCM) which poses a straightforward but often overlooked question: Robot control of nature is a partly observable problem.
Figure 1 shows that the WCM is based on the limitations of the single-player Critic and combines world prediction with value learning.
The answer to WCM is not simply to stack more images, but to let the Critic, while estimating the value, learn how to predict the next moment after performing the action. In other words, it not only answers how well the mouse is now, but also answers: If the mouse does this, what happens next?
WCM's code, data, and weight are all open source, and it's only 15 minutes of training to get a real machine value curve!
WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
The paper:
The code is:https://github.com/sylvestf/WCM
The data:
WCM performed system verification on four simulation benchmarks, 149 missions, and seven real robot missions, covering grab, flexible object, long-range, and dynamic target operations. At the same time, WCM can be paired with three mainstream VLA models: π0,π0.5,OpenVLA-OFT.
One of the most striking results came from ManiSkill: a VLA model with no previous exposure to ManiSkill data, with an initial success rate of only 0.78%, reaching 98.7% after WCM-guided reinforcement training, up 97.9 percentage points. More importantly, the improvement did not only occur within the training distribution, but also with an average success rate of 72.7% in OOD.
The group's findings point to the core value of WCM: it not only makes training tasks better, but also makes it harder for Critics to overlook changes in the environment.
01/01 by Critic of VLA-RL
Maybe you haven't seen the most critical information.
In the Actor-Critic reinforcement class, the strategy determines what the policy will do, and the critic is responsible for judging how good the status is, and for calculating the advantage, guiding the policy update.
Figure 2 The Critic is responsible for determining the current state of the WCM value estimate curve, which begins to decline when the object falls in a collision.
The problem is that robotic operations can usually be modeled as part of the observable Markov decision-making process (POMDP). A single snapshot can tell where the model object is, what posture the machine arm is about, what visual elements the current scene contains. But it often cannot reliably tell the model: the machine arm is approaching or moving away from the target; whether the claw has just slipped; whether the current contact is a stable support or is about to collide; whether the target object is still in motion; whether the sequence of movements is progressing successfully or has entered an irreversible failure....
Figure 3 Robot operations have some observable properties, and intelligent agents can only reconstruct systems by observing their state.
For example, a desk wiper is the same as an end-executor near a desk, which may represent that the desk is being cleaned, or it may represent that the machine arm is attached to the desk. The pixel is similar, but the value is completely different. This is why in real-life robotic experiments, the single-cylinder Critic is easy to learn some dangerous pseudo-relationships: for example, to overview the end of the desk near a desktop as a high value, ultimately causing the machine arm to repeatedly stay on the floor.
It's just that I don't know what to do.
It's not necessarily helpful to see a few clips for Critic.
Since a single image is not enough, the most natural solution seems to be to stack multiple images directly, or add a sequence of Transformers.
What is really missing is not the historical input of the chip itself, but the need for the model to understand the monitoring signals of historical change.
The core goal of traditional Critic is usually to return a metrical return. This supervision is too sparse in the face of high-dimensional image sequences. Even if the input contains the last few lines, the model may take them as a larger static feature rather than to learn how objects move, how contact changes, and where the movement will lead the environment.
Three notable phenomena emerged in the experiment: 1) to add historical observations directly to the original Critic, the performance may decline; 2) to replace the Transformer with timely modeling capabilities, but without reaching the world prediction goal, it is still difficult to exploit history; 3) only after joining the future state prediction, multi-channel history is stablely transformed into a better value estimate.
FIGURE 4 MLP, in contrast to the complete WCM, the complete WCM achieves the highest overall performance on different backgrounds and tasks.
This conclusion also explains a common misconception: a memory chip is not the same as a dynamic chip; only predictable historical features are closer to what a robot really needs.
How do you do it?
Let's get the Critic on the scoreboard and make a future move.
WCM is based on the lightweight LeJEPA style architecture and consists of four key parts: the observation encoder, the world predictor, the value head, and the dynamic predictor head. The entire process can be broken down into four steps:
The core architecture of WWCM: Shared historical representation serves both as a value estimate and as a prediction of the next potential state.
Step one: Code sequence history. WCM receives the most recent K-bone observation. Each sequence is independently encoded as a potential representation. In different implementations, the encoder can be either ViT or directly replicate VLA's VLM Backbone.
The second step: adding language conditions. The robot does not perform unconditional actions. The mouse puts the carrots in the dish and folds the towel pads, even if the visual image is similar, the definition of the status value is completely different. Therefore, WCM injected the task instructions with a historical marker through language coding and adaptation modules.
The third step is to aggregate history using a cause-time model. After language-conditioned polynomial characteristics are sent to the cause-time Transformer World Predictor, forming the current historical state representation. This representation is not a simple polynomial spelling, but supports two things at once: judging the expected return of the current state; and combining current actions to predict the potential state of the next moment.
Step 4: Dual-head combined learning. WCM sets two lightweight predictive heads: Value Head output current state values; World Head receives action conditions to predict the next potential state.
Its overall goal can be summarized as: total loss = value return loss + world prediction loss + character space normal loss. In this, character space uses SIGReg to suppress the potential representation of dimensional contraction. Here is a very critical design detail: value estimates remain action-free, while state transfer predictions are action-conditioned.
WCM is also not an additional pixel-generating, large-scale video world model. It predicts the next potential state of the task, and the goal is very clear: to let Critic learn to capture the dynamics that are really useful for judging value.
It doesn't bind to an RL.
On-policy and off-policy are compatible
Another feature of WCM is that it does not reinvent an entire set of reinforcement learning algorithms, but rather that it acts as a Critic access to an existing training process. In the same on-policy learning scenario: self-return VLA can access PPO, Flow Matching VLA can access Flow-SDE, WCM provides value estimates to calculate Return and GAE Advantage; in the off-policy learning scenario: OpenVLA-OFT can be combined with AWR, π0.5 can be combined with RECAP, SFT data, Rollout and failed Rollout are placed in a unified data buffer for continuous critical policy updates and success.
Figure 6 WCM can be directly embedded in both On-policy and Off-policy VLA-RL pipelines.
This modular design means that WCM is more like a one-time upgrade to the way Critic represents learning, rather than a skill that only works on specific policies or algorithms.
Results of the experiment: double SOTA of IND and OOD
In the RL4VLA-set Maniskill mission, the dual SOTA performance of IND and OOD was achieved with WCM's reinforcement learning. More interestingly, in the Zero-Shot setup, the WCM directly increased its almost ineffective 0.8 points to 98.7 points, up to a high of 97.9 points, fully verifying its excellent generalization and reinforcement learning adaptive effects.
Table 1 The WCM obtained SOTA results from Maniskill's in- and out-of-distributed test sets.
The team designed a very convincing set of contrast experiments on the LIBERO-Plus benchmark. One-SFT said that each task provided only one sample data, while Full-SFT provided up to 50 samples. The results showed that, based on One-SFT's WCM method, only about 250 steps of reinforcement modeling could be taken from a very simple initial strategy to successfully counter the over-dependent Full-SFT baseline on a large number of samples in overall performance.
In the LIBERO-Plus standardized test benchmark, 250-step RL training is the only way to exceed Full-SFT.
This is not to say that demonstration data is not important, but rather that when Critic understands dynamics and outcomes more accurately, the failure of environmental interaction can become a very valuable learning signal.
At MetaWorld and CALVIN, WCM also provides steady improvement in the performance of continuous contact and long-range operations.
Figure 7 shows that the WCM performed well in MetaWorld's success rate and CALVIN's average mission completion.
On the real machine side, the team tested 7 missions on the Widow X-250S: carrots, bananas, sweet potatoes, picking and placing; clothing, towels, table sorting; rolling sushi dynamic picking. Each mission first collected 100 remote control tracks, followed by eight RL updates; each round collected 50 Rollouts.
Table 3 uses WCM for real-machine reinforcement learning to achieve better results than VLM Critic.
In real-world robot settings, the WCM has about 107.2M trainable parameters. At the data scale of hundreds to thousands of trajectories, the WCM can complete rapid rotation in less than half an hour. This improvement is not only reflected in the final success, but also in the more coherent policy movements trained by the WCM, with fewer stops and ineffective small movements, resulting in greater throughput in unit time.
Figure 8 The RECAP + WCM has a higher throughput than the Gemma Critic when it comes to ordering and grabbing rotating sushi.
06: Visualization of the value curve
The WCM warehouse provides a value curve video generation tool. After the model's predicted values are superimposed on the real trajectory, you can see intuitively how it evaluates the progress of the move.
Figure 9 for Success Trajectory: The overall rise in value as the task progresses.
Figure 10 For failure trajectories: Value decreases rapidly from the moment of failure.
Figure 11 For ineffective trajectories: Random movements have not formed a stable success trend.
Visualization illustrates that what WCM learns is not a simple terminal classifier, but a continuous evaluation signal that reflects process changes. In real data, stops, slippages, collisions, poor grip or folding disorder can cause a value curve to stagnate or decline.
07/07/2021 in English
WCM provides code, data processing scripts, training configurations, model weights, and value curve visualization tools to work together as a new replicable baseline.
When a robot is no longer just looking at a mouse, it's more likely to understand where it's going.