The official open-source version of Ola-Sem is here!UniWAM explores the Scaling trend of Human-Robot Co-training to achieve global understanding, unified learning for future prediction and motion generation, full open-source code and weights.
Ola-Sem has previously co-published Ola-Sem and achieved SOTA results on the global mainstream charts. Today, the open-source version of Ola-Sem, UniWAM (Unified World-Action Model, Unified World with Action Model) is officially released, incorporating semantic understanding, future vision prediction and motion generation into a unified architecture, revealing for the first time the Scaling trend of Human-robot Co-training.
UniWAM received multiple SOTA ratings, with an average success rate of 99.2% on LIBERO, an overall success rate of 92.6% on LIBERO-Plus, and a 68.32% success rate on RoboTwin 2.0 C2R, all achieving a leading level in the methodology described in the paper. At the same time, UniWAM further verified the model's ability to execute instruction-following, complex interaction and long-range tasks through real-time experiments.
These results are driven by UniWAM's exploration of semantic understanding, unified learning for world prediction and action generation. Currently, model code and weights are officially open source.
One thought on "Picking up tableware"
What capabilities do robots need?
When you're faced with a table full of trash, you're instinctively saying to the robot: "Please clean up the table". Oh, that sentence doesn't list every step. The robot needs to identify the object, judge the collection strategy, and track the scene changes in execution: is the cup cleared? is the drawer open? what is the next step?
In UniWAM's long-range desktop sorting experiment, the robot receives the same high-level instruction when reasoning: "Tidy up the desk". The task includes multiple subtasks such as cup placement, drawer receiver pen placement, etc. These subtasks exist and are related by cause and effect: objects can be placed in the drawer after the drawer is opened, and the drawer must be closed after the reception is completed.
Figure 1 shows multiple objects on the arm platform capturing, interacting, positioning relative to each other, pulling, receiving and long-range tasks.
Three capabilities, one unified architecture
UniWAM's goal of successfully executing continuous tasks raises a fundamental question: Can robots no longer view understanding, prediction, and action as fragmented, but instead learn in a unified model, thereby reducing the disconnect between understanding and execution and better responding to continuous decision-making and environmental changes in complex tasks? Around this goal, UniWAM will integrate semantic understanding, visual dynamics, and motion generation into a unified architecture, making task requirements, scenarios, and motion information interrelated in training to support more stable and coherent robot operations.
VLA inherits the semantic ability of the visual language model, but learning about the dynamics of the world is limited only by the supervision of the movement. World WAM introduces the time and space precedent of video generation models, which are equally important in the face of environmental change, meaningful understanding and reasoning.
UniWAM connects three classes of components to a unified structure:
Physical Reasoner: Based on Qwen3-VL-2B-Instruct, it provides semantic information to understand current visual scenarios and mission instructions.
World Generator: Based on Wan2.2-TI2V-5B, learn to predict future visual changes and provide dynamic information for operations.
Action Predictor (action predictor): simulates the flow field of continuous action to generate the action snippets needed by the robot to execute.
The three components exchange information by combining multi-modal attention, enabling action prediction to combine task requirements with scene dynamics. UniWAM's core exploration is to make understanding, prediction and action interrelated and learn together in a unified architecture.
Figure 2 on the left is an overview of three components and combined multi-modal attention, and on the right is an overview of human, robotic and visual language data.
Unified architecture that efficiently connects semantics and dynamics
The three components retain their own characteristic processing methods and project the tokens of different modules into a common attention space. Semantic, visual and motion information interact in this unified space, allowing motion prediction to combine task semantics and visual dynamic clues.
World generators learn future changes in the frozen visual encoder potential space, and action predictors learn continuous control of signals. The two generating targets connect by sharing information while maintaining their respective modeling patterns.
Physical reasoning also maintains language prediction goals, receives physical language and visual questionnaire supervision. This provides direct training signals for semantic ability: robots need to learn to operate as well as continue learning.
From architectural design to data efficiency
Human-operated videos contain a wealth of experience, but the action labels often need to be obtained by estimating the accuracy of the trajectory. Handshaking, camera movements and depth estimation errors all affect trajectory accuracy.
The team observed in the data pipeline they analyzed that the shape of two-dimensional trajectories and movement trends were often more reliable, with errors more concentrated in depth and absolute scale. This meant that a trajectory could retain available information such as movement direction and operational intent, even if it was difficult to provide precise three-dimensional displacement.
UniWAM therefore explores a shared way of monitoring: while learning about visual changes, using these still valid motion information to help the model establish links between action and scene changes.
Expressions of movement in physical language accelerate movement learning
UniWAM designs physical language (physical language) to describe movements in natural language and provide shared supervision for human and robot data.
For example, a section of motion can be written as moving forward 5.5 cm, followed by a 30° turn of the clock. The values here are used to illustrate the form: the language can organize direction, position, and rotation through an existing language, matching the pre-trained expressions of the VLM visual language model.
This design focuses on preserving movement trends and intentions, reducing reliance on precise continuous labeling. The language describes the understanding of the components as providing auxiliary supervision, while the motion predictor still learns continuous control, without requiring robots to first output a step-by-step description of the movement and then execute it.
Improve tens of thousands of hours of data quality and data scale
The team built a cleaning and labeling pipeline, processed more than 10,000 hours of robot and human first-person data, and mixed VQA Q&A data for pre-training. Robot demonstrations provided control experiences, expanded human operation coverage of scenarios and behaviors, and VQA supplemented space, object and interactive state understanding.
Robot data is timed, geometry and visual quality checked; human data is used to describe subtasking levels. The team built the EgoANT for EgoDex, which was preceded by rough to fine-tuned operation events, reproduced and filtered descriptions to match language to local video events; the already labeled use of EgoVerse and VITRA is maintained.
It is noteworthy that data cleaning not only removes low-quality samples, but also preserves information that has physical interaction value. For example, small segments of visual changes may contain clavicle closures that cannot be removed simply because the image is still.
Figure 3 provides a description of the corresponding sub-tasks from a long video to locate the operation event.
Physical AI is finally on the move.
In addition to understanding, effective and robust action is needed.
For Physical AI, action should not only be the final output of a model's understanding of the world, but should be an important source of information for learning about the physical world. In continuous operations, the historical actions of robots not only record previous execution processes but also provide important clues for predicting subsequent actions.
Based on this idea, UniWAM introduced History-Conditioned Flow Matching, which no longer starts generating movements entirely from random noise, but uses disrupted historical movements as an initialization to predict future movements by updating their sections with a small amount of noise.
At the same time, robotic predictions of future scenarios are not always accurate. For this reason, UniWAM introduced Future Visual Noise Augmentation in post-training to reduce the reliance of motion generation on precise future predictions by imposing disturbances on future vision representations, allowing models to better cope with the uncertainty in predictions.
Figure 4 uses historical movements as a starting point to predict future movements by making noise updates.
Small-scale generation with high success rates
The mean is the arithmetic mean of RoboTwin C2C and C2R.
The experimental results showed that historical action initialization can significantly reduce the number of update steps needed to generate the action. In just 1 step, the average success rate reached 64.99%, up 22.79 percentage points compared to noise initialization; at 4 steps, it reached 71.73%, with only 0.06 percentage points difference from 10-step generation. This suggests that historical action information can be used to effectively reduce iterative updates in the generation process while maintaining operational performance.
Further dilution experiments in Figure 6 (c) also confirmed the role of two designs: historical motion initialization will increase the average success rate from 64.80% to 69.73%, and further increase to 71.34% after the addition of future visual noise enhancement.
From simulation to virtual machines
Testing the ability of robots to operate in complex scenarios
Simulation reviews: Changes from standard operations to complex environments
UniWAM is a leader in multiple robotics benchmarks, demonstrating the ability of the model to execute tasks and to generalize the environment.
On the standard operating benchmark LIBERO, UniWAM has an average success rate of 99.2%; on the introduction of the multi-environmental disruptive LIBERO-Plus, the overall success rate is 92.6%; and on the randomized environment testing of RoboTwin 2.0, the success rate is 68.32%.
The above three are for multiple benchmarks SOTA: the ranking is limited to the highest value in the methodology listed in the paper correspondence table.
The real environment often does not fully meet the conditions of training. Light can change, objects can be moved, and the camera's perspective can be distorted. Therefore, in addition to the standard mission success rate, whether robots can adapt to changes in the environment is an even more important indicator for measuring model ability.
LIBERO-Plus assesses the robustness of the seven-dimensional model from the background, camera perspective, language instructions, lighting, object layout, robot startup status and sensor noise.
The results of the LIBERO-Plus evaluation.
UniWAM scored the highest in the methodology described in the paper in the four robotic initial state, language, lighting and background assessments. Even in the face of more challenging object layout changes, the success rate was 84.6%, demonstrating the model's adaptability to different environmental conditions. At the RoboTwin 2.0 benchmark covering 50 tasks, UniWAM continued to accept randomized environmental testing.
The average success rate is the arithmetic mean of C2C and C2R; the maximal and sub maximal values of the columns in the table are marked with a rough and bottom line, respectively.
Switching from C2C to C2R, UniWAM's success rate dropped by only 6.82 percentage points and achieved the highest score of the following approaches in C2R testing. This suggests that UniWAM is not only capable of performing standard operating tasks but also performs better generalizations in the face of environmental changes.
Machine-to-machine verification: from understanding instructions to continuous operation
Simulation benchmarks validate the model's performance in different tasks and environments, while real robot experiments further test whether these capabilities can be converted into actual operations.
UniWAM conducted real-world experiments on the two-armed AgileX Piper robot platform involving up to 18 objects and designed four different types of difficulty operations:
Multi-object grabbing: Is it possible for explorers to correctly select target objects according to the instructions?
Different interactions: Whether the examiner can understand both the object of operation and the requirements of the action.
Relative positioning: whether the surveyors can understand and execute spatial relationship instructions.
Drawer receiver: Examination of two-arm collaboration and complex operational capabilities.
The experiments compared UniWAM with π0.5·Motus, each using 200 sample data per task, and combined individual training strategies under the same training protocol.
UniWAM achieved an average mission success rate of 67.5% and an instruction follow-up rate of 82.5% for all four types of missions. Among the various interactive missions, the UniWAM mission success rate was 72.5%, significantly higher than 40.0% for π0.5; the instruction follow-up rate was 80.0%, up from 50.0% for π0.5.
These two indicators measure whether a robot correctly understands the task requirement and whether it actually completes the operation. Understanding instructions does not mean being able to execute them successfully, only by effectively combining task understanding and action generation can the user's requirements be translated into actual results.
In addition to single operations, UniWAM further challenges long-range desktop organizing tasks.
The robot needs to follow a single high-level instruction to continuously complete six interrelated operating stages, such as the placement of the cups, extraction and collection of objects. The training stage uses a microscopic task description to provide supervision, while the reasoning stage only enters high-level task instructions.
In this task, UniWAM scored a task progression score of 5.0/6 on average, better than 4.8 of π0.5 and 3.2 of Motus, demonstrating the model's performance in multi-stage continuous operations.
Figure 5 shows the success rate and follow-up rate of the four types of missions, as well as the average progress score for long-range missions.
Dissolving experiments
The Value of Combined Training with Physical Language Supervision
Pre-training data combinations are dissolved.
Robot data pre-training increased the average success rate from 46.23% to 67.53%, including human data to 71.34%, and three-source mixing to 71.73%. However, VQA was only included in robot data and did not exceed the robot single-source program on average indicators, indicating that a specific design is required for good model effects, data sources and combinations.
In the robot+human settings, after removing language-specific action monitoring, C2R dropped from 67.56% to 56.40%, a difference of 11.16 percentage points. This is experimental support for the shared monitoring value.
Figure 6 shows the merging of three groups, each examining pre-training data, understanding component training strategies and movement generation.
Figure 6 (b) The cumulative melting of understanding components also showed gradual improvement: an average of 42.61 at VLM freezing, 46.49 after melting, 51.74 after joining physical language supervision, and 71.34% after further pre-training.
Scalable Joint Pilot Training
Scaling trends show
How to truly transform larger-scale human operation experiences into robotic capabilities is an important question on the path of Scaling, a body-based model.
UniWAM's experiments show two notable trends: the downstream performance of the model continues to improve with the expansion of pre-trained data by robots; and the introduction of first-person data to humans can bring further benefits when the robot data reaches a certain size, and the gains are more pronounced as the scale expands.
At the same time, in current experiments, the convergence verification loss of mixed pre-training of humans with the data scale shows an approximate numerical linear relationship, matching R2 to 0.983, initially demonstrating the pre-trained Scaling potential. This provides the experimental basis for further exploration of larger-scale combined pre-training of humans.
Figure 7 On the left is the verification loss, on the right is the RoboTwin C2R success rate; the horizontal axis is the number of hours of robot data, and the hybrid group adds additional human data, each group containing VQA.
Is there a more common structure for human expression?
In t-SNE visualization, with the addition of physical language supervision, the representation of human and robotic data is more overlapping. This phenomenon is consistent with the research goal of shared supervision to promote shared representation.
Combined with the previous operational evaluation and data fusion, the role of the design can be observed from two perspectives: from the representation of changes and from the performance of tasks. Visualization provides supporting clues as to whether transnational experience can be used effectively for operations, which still needs to be judged by specific tasks.
From UniWAM to Real-World RSI
Let's keep robots growing in the real world
UniWAM is not an isolated model study, but rather a phased exploration along the technical path of the Action-Native Foundation Model, undertaken in collaboration with HKG. Through unified learning of semantics, future vision prediction and motion generation, UniWAM not only focuses on how robots execute movements, but also attempts to engage movements in the learning of physical representations of the world, establishing connections between movement and environmental change, and providing a model basis for more complex real-world tasks.
This exploration is also an important foundation for the long-term construction of the Real-World RSI. On the basis of UniWAM, robots are continually entering the real-world to execute tasks, identifying failures, anomalies and insufficient capabilities in the execution process by the AI system, and converting this information into the next round of models, infrastructure or intrinsic improvement inputs. The worlds that UniWAM explores are combined with action, model-machine data scaling, and multidimensional mission evaluation are building the foundation for such a continuous process of improvement. Our long-term goal is not to train a better model on the Benchmark, but to allow the ability of real-world robots to accumulate and grow with experience in the real world.
At present, UniWAM's models with the training code, pre- and post-training weights are fully open source and provide training and evaluation resources for relevant benchmarks, and researchers are welcome for further exploration based on open source results.
Open source entry:
GitHub code:https://github.com/UniWAM/UniWAM
ModelScope model weight:
It's an AI-powered piece of content.