If the training videos shown to the robot come from cluttered homes, busy factories and pharmacies, and the camera is still vibrating as people move, will it still learn to operate precisely from it?
Recent work by the MMLab team at the University of Hong Kong suggests that this "not clean enough" data can also be used to train robots.
Recently, the team introduced EgoWild2Dex, which uses 538.9 hours of unintentionally organized real human first-person data to train robots. With less than 1 hour of real machine data per downstream mission, the robot is able to continuously complete sophisticated hand-to-hand tasks such as decommissioning, hot-shot fixing, and cold-drinking, with an average success rate of 96.7%.
Instead of "cleaning" the data in the lab, EgoWild2Dex chose to directly use natural data that preserved real factors such as perspective oscillations, complex backgrounds, and long tail object distributions.
EgoWild2Dex learns from real-world first-person human experiences and transfers those experiences to two-armed robotics to achieve complex operations over time.
Currently, the research paper is available on arXiv, and the project's homepage is available online.
The main page of the project:
Link to the paper:https://arxiv.org/pdf/2609.23755
The robot data is expensive.
But human experience is everywhere.
In the long term, robot learning faces a real problem: real robot data is too expensive.
To allow robots to learn complex operations, it often requires a number of remote operators to repeatedly operate on real robots, collect expert data and process failed trajectories. Once tasks involve arm coordination, easy use of tools, precise contact and long-range operations, the cost of data collection increases further. Human beings perform mass operations in the real world every day: unpacking, organizing goods, assembling parts, picking up tools, making drinks.
As a result, more and more studies have begun to attempt to use human first-person perspective (Egocentric) data to help robots learn. However, many datasets are still being collected around pre-designed tasks in relatively controlled environments.
EgoWild2Dex's approach is more straightforward: no longer requiring humans to "take a clean shot" of the data, but directly into the real world to record what humans are actually doing. The research team collects natural human operations from scenes such as homes, factories, pharmacies, hospitals, supermarkets, packing stations. These videos do not intentionally maintain a steady view and do not remove background interference as much as possible.
By the standards of traditional robotic data collection, this set of data is indeed quite "dirty". But the "dirty" here is not an indication of error or poor data quality, but rather that it retains the complexity of the real world.
The "noise" that seems to impede model learning.
How to become a key resource for robot training?
The first obstacle to using human data to train robots is the difference in perspective between humans and robots.
A human first-person view camera is typically installed in the head. When performing a task, the person moves, turns, and lowers the head, so the image has very strong dynamic changes. In contrast, the perspective of a robot camera is often more stable. If the two data are directly mixed together, the model actually needs to face two sets of very different visual distributions at the same time.
For this purpose, EgoWild2Dex has developed GeoFormer (Geometric Transformer), a lightweight, microscopic geometric transformation module.
Simply put, GeoFormer transforms "the image that people see while moving" into a perspective closer to a robotic camera.
GeoFormer learns the geometrical relationship between the human first point of view and the robot's point of view, and dynamically transforms images to reduce the visual difference between human data and robot data. This means that humans do not have to intentionally maintain camera stability when collecting data in order to adapt to robots. Compared to traditional Project + Inpaint processes that rely on precise camera marking, 3D projection and image repair, GeoFormer relies less on precise marking, increases processing speed by 21.9 times, and improves visual alignment.
GeoFormer's perspective aligns with the training process. (a) perspective alignment: predicting the geometric relationship between a robot's perspective and a human's first perspective, and aligning human observations with a robot's perspective; (b) geometric transformation learning: gradually optimizing transformation parameters through multi-level prediction and using gauges to monitor more natural results; (c) VLA training: aligning human and robot data for strategic training.
GeoFormer aligns dynamic human first point of view observations with those closer to a robotic camera, thereby reducing observation differences between humans and robots.
After the alignment of perspectives, there is also a need to address the differences between human and robotic movements.
EgoWild2Dex uses action retargeting and inversion dynamics to map three types of data into a unified robot native action space: 1) human first-person perspective data collected in the natural environment; 2) glove handheld demonstration data collected from a robot perspective; 3) real robot demonstration data (Robot). After retargeting, the three types of data can be used to represent unified robot actions and participate in strategy training.
Open-scene human first-person view data, human first-person view data related to downstream missions, and glove demonstration data from a robot's perspective will be mapped to a unified robot native action space.
After completing the alignment of the visual and action space, EgoWild2Dex adopted a progressive human-robot co-training paradigm ("Progressive Human-Robot Training"). The entire training was divided into three stages: pre-training with large-scale human data, then adding mission-related data, and finally refining with a small amount of machine data.
1. Open Human-to-Robot Learning: First, models use large-scale In-the-Wild human first-person data to pre-train. This stage does not require all of the data to come from the tasks that robots ultimately need to perform. VLA models thus have access to more scenarios, objects and modes of operation to learn general vision and operational priorities.
Human-Robot Co-training: Next, the model adds further Ego data related to downstream tasks, handheld demonstration data, and a small amount of real robot data. With a unified operating space, the extensive experience offered by humans begins to build correspondence with specific robot tasks.
Robot-Domain Refinement: Finally, a small amount of real robot demonstration and recovery trajectory is used to fine-tune the model. This phase is mainly used to further adapt the physical contact, dynamics and execution errors of real robots. Real robot data used for each downstream task is less than 1 hour.
Ego data, glove data and machine data
The team simultaneously launched the massive "EgoWild" real human first-person perspective data set. The data set contains 538.9 hours of operational data, covering approximately 179,000 operational segments, 126,000 separate mission descriptions, and 1,282 object categories. Here, "Dirty" does not refer to incorrect labeling or low-quality data, but indicates that the data does not deliberately circumvent the complexities of the real environment. EgoWild collects data directly from households, factories, pharmacies, hospitals, supermarkets, packing stations, and maintains camera movements, complex backgrounds, long-tail object distributions, and natural human operation processes.
EgoWild records natural human operations in real environments such as homes, factories, pharmacies, hospitals, supermarkets, keeping dynamic perspectives, complex backgrounds and diverse operating processes.
Experimental testing
In the real-world experiments, the team tested three complex long-range dual-arm operations: "Using a machine tool to split the ribbon and open the box", "Using a hot-shot fixture" and "Using a pumpkin ice and make an ice drink", the model achieved an average mission success rate of 96.7%, and demonstrated in the experiment the ability to cope with external disruption and restore tasks. In the open-box removal expedition, the team also tested the model in the face of unseen objects in training, with zero sampling of unseen objects in training, with an average success rate of 33.3%.
A series of sub-tasks such as grabbing tools, cutting ribbons, opening the casing and removing objects.
Hand-to-hand operation of the rifle and the projectile, complete continuous precision operations such as alignment, stretching, grabbing and fixing
Complete multi-stage arm operations such as ice-making, pouring ice, holding cups, allowing water out and taking water
In the face of unseen in-box objects in training, the robot can still attempt to complete the crawling and removal.
When the task is disrupted, the robot is able to adjust the movements according to the current state and continue to do the follow-up.
When the grabbed swing is removed, the robot is able to rearrange the hands and arms and complete the grabbing again.
In the face of human-taken interference, the robot is able to retrieve the cup and continue the task.
EgoWild2Dex moves further from Piper's mechanical arm to Tianji Marvin Pro
Summary and Prospects
Where EgoWild2Dex is most notable is perhaps not the success rate of a specific task, but its expansion into robotic data sources: in addition to expensive real-world demonstrations, unintentional human day-to-day operations, it can also be a source of robot-trained data.
Of course, from the first point of view of human video to the strategy of robots that can be stably executed, the middle still involves multiple differences such as perspective, essence, dynamics. But if such data can be used more effectively, the amount of data that can be used by robots is expected to expand significantly.
EgoWild2Dex is just an initial exploration in this direction. We look forward to future robots truly learning from everyday human experience, adapting, accumulating and evolving in a complex and changing real world.
© THE