From physical video generation to movement migration to controlled world modeling.
If natural language helps AI to read human records, PhiZero's exploration of pseudo-physical language attempts to make AI read movement, interaction and change in the real world.
PhiZero: A World Model Built Around Physical Language
Website of the project:
The team included Shuyao Shang, Yuqi Wang, Ruopeng Gao, Xu Chen, Tieniu Tan, Liu Fan, Zhaoxiang Zhang.
It is affiliated to the Institute of Automation of the Chinese Academy of Sciences (NLPR, CASIA)
Language allows for the preservation of experience and the continuous transmission of knowledge across time and with individuals. Human beings use language to describe everyday life, summarize scientific laws, write history and procedures; large-scale language models master conceptual expression, knowledge organization, and digital information understanding and reasoning by learning these contents.
When AI begins to enter the real world, is natural language alone enough to support its understanding of the world? Furthermore, in addition to describing the text of things, can machines learn a set of symbols that directly represent the particle changes of the physical world?
From this question, PhiZero hopes to create a new learning, predictable representation of the evolution of the state of the world.
From the outcome: What capabilities has PhiZero demonstrated?
1. Producing videos with coherent physical processes
Whether sea water shocks rocks, fluid-filled containers, or objects plunge into hot oil, or metallic tubes explode, PhiZero can model the follow-up process. It seeks not only to ensure that each tube is realistic enough, but also to ensure that the effects of the action can be sustained and that the whole event is kept complete over time.
Although the scene spans waves, fluids, burns and explosions, the core task facing the model is the same: to predict how the current state will continue to develop.
2. Motion Transfer: Let the same change happen on different carriers
In the physical language, the focus is on the change itself, rather than the specific object of the execution of the change. Based on this characteristic, the corresponding state shift of a movement can leave the original video and be given a new appearance, physical structure, or visual style.
Human Body → G1 Humanoid
For real-life movements, PhiZero can first extract the status changes and then allow the Unitree G1 humanoid robot to present the corresponding process.
The graph contains the results of four groups of cross-body migrations; each group consists of the real input on the left side and the output on the right side of the G1.
Human Hand → Sharpa Dexterous Hand
Similar migrations can also occur in hand-to-hand operations: human-to-object contact, the process of grasping and turning the wrist, can be replayed on Sharpa's clever hand.
Simulation → Reality
This mechanism is also capable of connecting simulation domains to real domains. The model preserves the movements and interactions contained in the simulation footage and then generates corresponding follow-up videos based on the initial appearance of the real scene.
Each line shows the same interaction process: the simulation input is on the left, and the reconstruction results in the real field of view are on the right.
3. Starting from a single image, continuously explore interactive worlds
PhiZero is capable of receiving continuous input and updated control signals. Each time a change in direction or observation perspective occurs, the model generates the world forward, while striving to maintain consistency in environmental layout, key landmarks, and spatial relationships.
The four demonstrations were held from the outer horizon, a view of the lake below the moon, the village of Yangtze and the sunset plains.
4. The driving action determines the future picture
Different control tracks should lead to different outcomes when faced with the same path start-up. PhiZero combines the direction and width of the direction to generate the corresponding future; in the following four cases, multiple alternate control signals and their generated videos are displayed alongside each other.
5. Predicting the visual consequences of robot action
In robotic tasks, trajectories such as grabbing, extracting, and two-arm collision can be first encoded into a physical language and then restored to fine-grained interactive video.
The video looks real, so why not?
In recent years, the generation capacity of video world models has steadily improved. By entering an image or a text instruction, the model has been able to synthesize future footage with excellent visual effects. However, images are realistic and cannot guarantee that what happens in them conforms to physical laws.
How should a tennis ball move after hitting a toy bowl? What kind of fluid reaction is triggered by an object entering a rolling oil? When a metal container explodes, how do the light, fragments and projections change together over time? To answer these questions, the model needs to go beyond the quality of the pixels on the surface, to understand the effects of movement between states, and the causal relationship between events that unfold backwards.
Traditional methods often generate the future directly in extremely high-dimensional pixel spaces, and the physical world's state transfers are therefore enveloped in a huge visual prediction network. Such models are good at synthesizing the next image, but lack an intermediate layer specifically designed to express how the universe will change.
What is a physical language language language?
By performing self-monitored learning on large-scale real-time video, PhiZero acquired a set of compact, discrete representations called Physical Language. This representation does not repeatedly record static visual details such as color, texture, or cover all the pixels in the image; it specifically encodes another type of information:
How objects move, how they interact with each other, and how the whole scene moves from one moment to the next.
For example, the shape and material of the container is already contained in the headstock; dynamic processes such as inclination, flow, diffusion are carried out by physical language. Thus, information describing appearance is clearly separated from information describing evolution.
This division of labor makes it possible for state changes to be replicated. Even with the change of scenery, material or object form, the original laws of motion and interaction can be preserved and further engaged in combination and migration.
The objects in the picture have changed with the environment, and the status shift across the events remains consistent.
PhiZero's method: rolling before and rolling after
With physical language as the middle layer, PhiZero builds Reason-then-R.