How can human data become the real capabilities of robots?

The Heart of the Machine

As the number of first-person videos increases, how do researchers turn these records into training information that can improve robot capabilities?

A person picks up a cup on the table and the camera can record the entire process of movement. But how to extract the camera trajectory, hand movements and changes in the position of the cups and align them with the images and text descriptions is still a problem that has not been completely solved. And whether this information can make models predict more accurately and make robots operate better needs to be tested through specific tasks.

In addition to collecting and organizing human experiences, the team also needed to test which of these experiences were useful for the model and whether the system could use the information in specific tasks. So, from October 19 to 23, Ropedia hosted the first Ropedia Super Week, which focused on six studies around these issues.

Ropedia W0 and DONIC, respectively, use human experience to predict future images and manually manipulate them to directly test whether models can benefit from them. HoliWorld and EgoXperience focus on how the experience is accurately recorded: the former organizes the interaction history of objects and turns it into training questions that can retrieve observational data; the latter jointly reconstruct cameras, scenes, manuals and objects, allowing different types of motion information to match in space.

The S-Agent 2.0 study model uses tools and scene memory to make judgments based on spatial evidence; the Ropedia A0 allows programming intelligence to write observations, decisions, and checks into control codes that are executed by the program. Researchers can track the causes of code failures, maintain repair in shared skills, and record execution trajectories to prepare data for subsequent research.

Figure 1 Ropedia W0 Predicts images of different objects using the same world model. The model uses camera trajectories and skeletons to describe movement conditions, covering objects such as humans, claw-like mechanical arms and clever hand-held humanoid robots.

01 Ropedia W0

Humans and robots share a model of the world

In Ropedia W0, the research team examined whether models learned from human video could help it predict future images of robots. The research team used camera posture to describe changes in perspective, projected skeletal movements on the image, and shared a set of motion interfaces with human hands, claws, and mechanical arms. The model combined the initial image, text, and motion conditions to predict the subsequent first perspective video.

Figure 2 Ropedia W0 receives different physical conditions of motion through a shared motion interface. The camera trajectory describes changes in perspective, the skeleton describes movements of the hands or claws. The model combines initial images, text and motion conditions to generate subsequent first-person video.

To train this model, the research team constructed Xp-Data, integrating about 10,000 hours of human first-person video and 3,000 hours of robot video, unifying camera trajectories, motion skeletons, and mission descriptions.

In addition to the size of the data, precise movements and visual alignment are equally important. The research team found that the open-source robot dataset used did not provide reliable results for hand-eye marking, or that there were problems with hand-eye marking, which made it difficult to accurately match the robot's alignment with the motion in the video image. For this reason, the team improved the accuracy of motion information and visual image alignment by artificially marking and modifying key points of the robot.

The same group of Ropedia W0 weights achieved a PSNR of 21.98 dB, higher than the Cosmos3-Nano (19.02 dB), which was adapted using the same robot mixed data, without additional training for targeted evaluation data.

The human experience has also significantly reduced the need for data for post-robot training. In the GR-1 assessment, with models with W0 pre-training weights requiring only 1% of post-robot training data, the predictive quality exceeds the initialized version of the general video generation model using full data. Further Scaling Law experiments show that the predictive quality in both human and robot assessments continues to improve as human pre-training videos increase to 10,000 hours.

In addition to predicting future images, Ropedia W0 is also used for robotic strategy evaluation (Policy Evaluation) and model-based planning (Model-based Planning). Researchers allow world models to simulate the execution results of different strategies and evaluate strategies based on predictive images. In the Unitree G1 real-machine experiment, the model evaluation results with a coefficient related to actual performance reached 0.996.

Ropedia W0 also demonstrates real-time interaction capabilities. After accelerating model distillation, the system reaches a video generation speed of 21.3 FPS on a single GPU. Users can wear VR headsets and control the first visual images generated in real time through head and hand movements, achieving continuous interactions of more than a minute.

The team also used the pre-training weight of Ropedia W0 for robot action learning. They used the video backbone of the Ropedia W0 5B to initialize the world action model FastWAM to keep the action forecast head and training program unchanged. On the LIBERO-Plus scale, the success rate of the initial version of the Ropedia W0 increased from 50.06% to 56.82% and improved in all seven categories of environments, including camera, lighting, and background disruption.

From real-world forecasting to strategic evaluation, action planning and real-time interaction, Ropedia W0 demonstrates the many uses of large-scale human experience in robotic learning and provides new directions for reducing the need for robotic proprietary data.

Figure 3 Ropedia W0 supports real-time VR interaction with real-time deployments. Left: Operators wear Quest 3 real-time drive models to generate a robot or human perspective image at 21.3 fps. Right: Run tasks on Unitree G1 using the Ropedia W0 initialization strategy, with each row left to the external perspective and right to the robot perspective.

02 HoliWorld

A glass of tracking chips can also be turned into training data.

HoliWorld organizes the objects, location changes and interactions in the first-person video into questionable records, and then converts the records into training questions. For example, a person takes the cup from the table and puts it somewhere else, the model needs to identify the same cup after the first one, and connect the two positions and the operations in between.

According to the paper, the Holi World-Foundry first uses synchronous binary imaging with IMU data recovery scene geometry and camera positioning, then deals with static objects and dynamic targets respectively. Static branches combine images with point cloud detections, then verify by VLM; dynamic branches combine interactive descriptions with key string-related object identities, track object regions, and then map results to unified world coordinates. Current dynamic tagging processes assume that at least one object is moving at the same time.

After this set of processing, the team was able to connect the same object's state in different segments. The three-dimensional frames stored by the system, the trajectory of movement and the semantics of interaction are all associated with the observational evidence, which allows the researchers to find a time of movement and corresponding images. The office below shows an example: a mobile car moves from A to B, with its trajectory and the structure of the room displayed in three-dimensional scenes.

Figure 4 shows that HoliWorld keeps the scene and the movement of objects in the same world record. The system puts the camera path, static objects and dynamic targets in the same coordinate system.

It is reported that the team has grouped about 100 scenes and 444 videos into HoliWorld-Data, and generated about 780,000 group spaces, time and interaction questions. The answers to each question can be traced back to specific observations.

The team also used the data to fine-tune Qwen 3.5-4B, obtained HoliWorld-VL, and test learning effects on external evaluation of EgoDynamic 4D. The Token F1 of the model increased from 18.29% to 66.39% before fine-tuning.

03 EgoXperience

Let the camera, the scene, the hands and the objects in the first view go back to the same world.

EgoXperience enabled the model to reconstruct cameras, scenes, hands and objects at the same time, helping researchers to reconstruct how operations occurred in 3D space. It estimated that these elements were spliced together when the camera was moving, hands were operating, and objects were moving, respectively.

EgoXperience has proposed a comprehensive model of real-world collection, automated markup, unified modeling and evaluation that aims to transform large-scale first-person experiences into unified representations that correspond to the geometry of the physical world. The entire process includes the data processing system EgoX-Engine, the data set EgoX-Data-6M, the evaluation benchmark EgoX-Bench, and the joint reconstruction of the model EgoX.

Among them, EgoX-Engine grouped the real-life experiences collected by HOMIE into RGB, depth, camera position, MANO, handles, object masks, object 6DoF trajectories, measurement object grids, and eight synchronous models describing tasks. The team built an EgoX-Data-6M of 50 hours, 540,000 tons, 270 missions and 101 objects, covering single-area operations, inter-room and outdoor mobility, and organized an hour-long EgoX-Bench to assess joint reconstruction capabilities.

On the model side, EgoX extends VGGT's frontal reconstruction paradigm from scene to first-person video. The model only receives single-camera video and first-person object masks, which can be used to simultaneously predict cameras, scene depths, hands and object movements, and place them in a shared measurement world coordinate system.

Figure 5 shows the unified reconstruction capability of the EgoXperience. Given the ego video and the first object masquerade, the EgoX requires only one forward direction, which is to combine recovery scenes, cameras, hands and objects 6DoF in a unified coordinate system. The motion shows the reconstruction effect of the model in different object and operating scenes.

The results of the experiments showed that the strongest control method in the comparative paper was that EgoX reduced the camera error by 68.9 percent on HOI4D, the hand world coordinate error and depth error by 65.6 percent and 42.9 percent on EgoX-Bench, respectively, and the object position score by 28.0 percent on HOT3D. These results showed that when multiple surveillance signals are organized into the same geometric system, the human experience can provide more complete, higher-density learning information for the intelligence of the body.

04 S-Agent 2.0

Strengthening tools for call and scene memory

How to make models understand space better

S-Agent 2.0 uses sensing tools and query-able scenario memory to help existing programming intelligences respond based on spatial evidence. A case of failure in the paper illustrates the problem: the baseline system SpatialClaw once calculated that the fireplace was located on the west side of the stairs and finally answered on the east side.

The team is equipped with a set of command line perception tools for existing general programming smartphones, with no additional model training required. The team can continue to use the original code execution, file access and verification modification processes. The team has validated this method in three frameworks: OpenHands, Codex and Claude Code.

For example, a smart body needs to adopt the point of view that the subject is aiming for. It first reconstructs the scene, then registers the person lying on the bench as a positioned and oriented observer, and then calls the tool to inquire about the person's orientation. The model can also nuclear perceive the output, remove instances of erroneous registration, so that the object it understands matches the object measured by the tool.

The toolkit is responsible for storing this geometric information and maintaining a queryable scenario memory for each question. The smart body can then search for evidence by instance name. Each time the tool returns the readings, it marks the objects, units and reference systems, and accompanies spatial explanations, such as a parking lot on the observer's right side at 2.78 meters. If further verification is needed, the smart body can also open the file to view the full measurement results.

Figure 6 shows that S-Agent 2.0 reads from the point of view of the intended observer according to the title. The intelligent body registers the camel lying on the bed as an observer with a position and orientation, and then calls the tool to inquire about the location of the parking lot. The result of the return of the tool indicates that the parking lot is 2.78 meters to its right; this is the ratio in the right direction.

The team compared the performance of the same general-purpose programming smart body access tool before and after. On the Qwen 3.8-27B backbone, the ReVSI score increased from 61.8 to 82.3; in five spatial reasoning assessments, the system led the smart body baseline listed in the paper. When switching to the Gemini 3.8-Flash, the team used the same tool- and tool-free control settings, and the ReVSI score increased from 80.9 to 90.5.

Figure 7 shows that the ReVSI score of the same programming intelligence has been improved. The research team has remained consistent in comparing the performance before and after the access tool: Qwen 3.8-27B improved from 61.8 to 82.3, Gemini 3.8-Flash improved from 80.9 to 90.5.

05 DONIC

A set of strategies for robots to learn how to work hand in hand

DONIC transformed the human hand presentation into a motion reference, using a shared strategy to allow the two clever hands to track these references in a simulation, coordinating the manipulation of objects. In the paper's contrast experiments, there are methods that, although the gesture of the hand is close to the presentation, the object does not reach the target location.

Figure 8 shows different strategies executing the same two-handed reference in a simulation. From left to right, the diagram shows a sequence of redirected references, DexTrack-bi, Dexplore-bi and DONIC. The image is taken from a demonstration video, showing the relative relationship between the two hands and the object when performing each method.

For this purpose, the Ropedia team proposed DONIC, which uses a shared strategy to track different human hands. The paper reports that in simulated single-object assessments, the same DONIC strategy for training with only open data does not require a test reference mitigation, achieving a tracking success rate of 86.79% on 53 unseen trajectories of the trained object, reaching 57.32% on 164 unseen object references.

According to the team, the researchers first reconstructed the three-dimensional movements of human hands and objects, then re-directed hand movements to the robot reference. DONIC then combined PPO and BC to conduct training: the strategy is learned through simulation interaction, while the system will successfully execute the verified trajectory back to the model library and be updated for subsequent use.

The timing of hand-to-hand contact is often different, and DONIC therefore continuously evaluates whether the object moves by reference, and rewards according to each hand's contact role. When one hand first restrains the object, the other hand subsequently participates, the strategy also receives the corresponding learning signal. For the differences in difficulty between demonstration segments, the team also designed Clip-Centered Advantages: the system subtracts the average value of each segment's own strengths, retains the relative differences in movement within the segment, and mitigates the disruption of sharing training.

The team also added 1575 hand-interaction references to the training set in the self-constructed BiHOI data set, keeping the training method unchanged. The success rate of the two groups of tests increased to 88.68% and 59.76% respectively, improving on seven indicators.

After adding BiHOI data, DONIC's simulation tracking performance improved further. The left-hand side evaluated 53 unseen tracks on the training object, and the right-hand side used 164 unseen object references. After adding 1575 unseen object references to the BiHOI data, DONIC's success rate increased from 86.79%, 57.32%, to 88.68%, 59.76% respectively.

06 Ropedia A0

SWE's idea of disarming missions

A0 explores a new paradigm: using pure code as a robot strategy. Programming intelligent agents, like software engineers, write perceptions, decisions, and checks into code during the development phase; robots run only code when they execute: read images and link readings, measure objects, calculate movements, and check results.

From the team's point of view, the code itself can make closed-loop decisions, handle errors. It turns robotic strategies into solving a physical problem, without having to scramble the trajectory: every step in the field solves a problem, first looks at the object and its state, then calculates the physical properties such as shape, size, and so on. A0 uses Code-Only-as-Policy (COAP) developed by Embodied Turing Machines, to write these decisions together with checks and fail re-tests, which was originally left to the online decision makers of the large model to automatically execute the code.

Generalization: Skills can be written according to physical properties, without recognizing object names, changing objects, changing tasks.
It can be explained that each decision is a line of readable code that allows the problem to be located in a specific line.
Reuse: Skills are stuck in shared libraries, new tasks are replicated and can't be expanded.

Figure 10 The ability to pick and choose the way to grasp according to the shape and intention, when lifted up and do not see the object rising, to look up and confirm the grasp and continue.

The paper reports that according to RoboDojo's public protocol self-assessment, COAP has a 70.24% success rate on 42 dual-arm simulation missions.

The same set of skills can be replicated across the body: changing the body requires only an adaptation layer and an original specification file, skills and checks remain unchanged. The team performed a simulation verification on LIBERO's Franka Panda and MARVIN, equipped with 22 free-thinking hands, with the handy pick-and-place capturing an accuracy of 94% for the first time. On sim-to-real, AgileX PiPER had a 95% success rate for the first time in the simulation lifecycle, and the real machine only added an adaptation layer and pick-and-place was successful.

Figure 11 shows the same set of skills being reused across the body.

A0 wants to answer this question: where are the limits of pure code as a robotic strategy? Under this paradigm, code has gone from being a tool for model calls to being a self-contained decision-making, self-checking and error-correcting closed-loop skill. A0 has also validated the generalization of code: the skills don't require each task to be rewritten, a shared library can be cross-tasked, cross-body replicated, and sim-to-real transferred.

Testing the Value of Experience with Task Results

Returning to the opening of the cup, the researchers were concerned about more than just whether the video was complete. At the same time, the camera trajectory and hand movements were not aligned, the model could deny that the same object appeared after the opening, the program could judge whether the cup had been seized, affecting subsequent learning and execution. This release provided several specific test methods. The researchers could compare the model effects of using different pre-training data, add controls to the controls before the hand demonstration, and observe how tools and scene memory affect judgement while keeping the skeleton intact. Different controls allowed the team to distinguish which improvements came from the experience itself and which organizations depended on the use of information.

Ropedia Super Week brings us the latest in a series of studies that tell us what high-quality ego data and labels are, and what such high-quality data can do to greatly advance research on these issues. For Ropedia, these results can in turn guide subsequent data collection, labeling and model research. The value of human experience will ultimately reflect the performance of robots in specific tasks.

Homepage of papers and projects

Ropedia W0