Based on the pre-trained Cosmos-Predict 2.5-2B, SkeLAM used a total of about 10 hours of robot interaction data to train. In human-to-robot migration experiments, inputs were converted into human demonstrations, and the mechanical arms in the video generated still showed the corresponding movement direction.
The team used latent action to monitor potential movements in video with the robot skeleton, allowing the motion vector to more accurately represent the robot's movement, while visual changes such as object response are retained by another vector.
In DROID's evaluation, compared to DreamDojo, SkeLAM's endpoint trajectory error was reduced by 19.5%, and the perceived LPIPS distance was reduced by 11.1%.
The title of the paper is: Skeleton-Grounded Latent Action Model for Robot World Models
Link to the paper:https://aopolin-lv.github.io/SkeLAM/assets/SkeLAM.pdf
The project's homepage and video: aopolin-lv.github.io/SkeLAM
Human videos as a tip:
Zero-sample cross-formal movement migration
The input of this group of experiments was a video of a UniSkill human demonstration, as well as an image of a robot scene. The RGB encoder extracted potential actions from the demonstration, and the world model generated subsequent videos in a given robot scene.
The human hand pushes down the towel, and the resulting mechanical arm is searched; the adult hand pulls the paper upwards, and the mechanical arm is turned upwards. The human hand and the mechanical arm have different appearances, structures, and locations, and the direction of motion in the demonstration is still reflected in the resulting output. In contrast, the DreamDojo moves smaller endpoints in the two sets of examples.
Figure 1 shows two sets of zero-sample migration examples: downward pushing towels and upward pulling paper. Each set is followed by a human demonstration from top to bottom; DreamDojo and SkeLAM, with a circle point marking the end position.
Human demonstration here serves as a context for the reasoning movement: replacing demonstrations can change the generated motion in these examples without updating model parameters. This is a specific form of in-context human-to-robot motion transfer, which also provides qualitative evidence for zero-shot cross-embodiment motion transfer.
The skeletal oversight,
To make it easier for potential moves to align with real control
Learning movements from video is the first step in dealing with the problem of mixing mechanical arm movements with object responses. The mechanical arm moves forward, may push a cup, or it may not touch anything. The control instructions can be the same, but the changes in the video are different. If these changes are encoded into the same potential movement, the actual control instructions need to correspond to a goal of changing the scene.
SkeLAM uses a skeleton to restrict the amount of movement in the skeleton. The skeleton records the position of the robot's joints and connectivity relationships, providing a reference to the movement in the video; the remainder represents the remainder while retaining the complementary visual dynamics.
Figure 2 provides an overview of the SkeLAM method. The movement in the skeleton monitoring video shows that the movement and the ratio of residues respectively provide conditions for the world model, and the motion adapter then maps the robot instructions to the movement.
A clear movement reference to potential movements
The same mechanical arm movements, under different contact states, may result in different object responses. These changes all contribute to the reconstruction of the video, but when the potential moves simultaneously include the results of movement and interaction, the robot control instructions need to correspond to a scene-changing target.
SkeLAM uses a skeleton to restrict the amount of movement in the skeleton. The skeleton retains the location and connectivity of the robot's joints to describe more directly how the robot moves; the other part of the residual representation retains the complementary visual dynamics.
Specifically, the skeleton decoder monitors the amount of movement extracted from RGB video, enabling it to predict the corresponding changes in the skeleton. Transmode switching reconstruction requires further: replacing the amount of movement in the RGB reconstruction with the corresponding skeleton representation; retaining the original remnants, the model still needs to recover the same change in the image. Thus, the correspondence between the video and the skeleton representation directly affects the reconstruction results.
In order to use this set of representations for robotic control, SkeLAM trained a lightweight motion adapter to map the actual robot's motion to the motion ratio and then use the adapter's predicted conditions to fine-tune the world model. When reasoning, the model generates future video based on the current image and the robot's motion, with residual input zero.
Whether the results are closer to the real movement
In the DROID assessment, compared to DreamDojo and SkeLAM, the end trajectory error ADE decreased from 51.11 to 41.15 pixels, and the visual distance LPIPS decreased from 0.1473 to 0.1310. The former measures the deviation between the generated motion and the reference trajectory, which reflects the perceptual difference between the generated image and real video.
The following comparison shows the performance of these differences in the image. SkeLAM is closer to the curved trajectory in the reference video and better preserves the desktop layout.
Figure 3 shows the results of the same initial image and robot action. The left-hand row is real video, and the second row is SkeLAM; the last row is superimposed on the trajectory of the end motion, making it convenient to compare the deviations between the generated motion and the reference motion.
Making more human video a resource for robot learning
In this direction, large-scale human-operated video, especially first-person video, is expected to become a movement source for the model of the robotic world. If it can be steadily extracted from it, there will be an opportunity to use more human demonstrations for video generation of robotic scenes, to expand the interaction process available for model learning, and to further explore their value in robotic strategy training and evaluation.
From these generated results to real robots, more complex interactions with longer operational processes need to be tested, and the video generated needs to be evaluated for practical help with strategy learning and closed-loop execution.
For the full paper, video introduction and more visual results, see the project's homepage.
Author's note
Qi Lv: The first author of this article, from the Harbin Industrial University (Shenzhen) and the National University of Singapore, is involved in research on robot operation, motion representation and world models.
Jiayi Lin: From the Harbin University of Industry (Shenzhen), related research involves robot operation, motion representation and world models.
Yuxin Jiang: A*STAR PhD student at the National University of Singapore, whose research interests include model generation, video world models and their applications in physical intelligence.
Mike Shou Zheng: Communication author, assistant professor at the National University of Singapore, whose research areas include video understanding and multi-modal intelligence.
© THE