After robots leave the lab, training data is no longer just a question of whether the environment can cover the scale, shading, contact and movement of the real world, and directly determine whether the strategy can land. Real2Sim is therefore refocused.
The difficulty is that the world seen in the video is not the same as the world available to the physical engine. Existing 4D reconstructions often result in hidden fields, Gaussian primitives or dot clouds, suitable for new perspective rendering, but difficult to directly dismantle independent objects, and lack closed mesh, real scale and stable timing movements.
The answer given by OVOW (One Video, One World) is straightforward: starting with a simple single-part video, dismantling scenes into instances, completing geometry, restoring scale and movement, then reassembling them according to gravity, contact and support relationships, and finally getting a 4D Mesh world that can be accessed by a physical engine.
OVOW: Start from a single-shot video, restore the 4D Mesh world to an instance level and send it to a physical simulation
One video, one world: turning monocular video into physical 4D scenes
The main page of the project:https://onevideooneworld.github.io/
The paper:
The code is:https://github.com/SparcAI-Inc/OVOW
From hydrogen sulfide dyeing to hydrogen simulation
OVOW's breakthrough was not only to rebuild cylinders like cylinders, but to produce editable, colliding instance-levels of Mesh: solids record true scales with 6-DoF bits in the shape of cylinders, and non-combusts retain topographically consistent sequence changes. Objects can therefore be moved, removed or replaced separately.
The system also estimates ground and gravity, corrects suspension, penetration and faulty support; a single image can also be processed as a video.
Image-to-3D Reconstruction: On the left side of each group is an input image, and on the right side is an instance-level 3D Mesh scene where OVOW is restored. The system can restore independent objects, complete geometry and spatial layout under single diagram conditions.
A flow of water from the video to the physical world
The entire flow line is not a dedicated large model, but a series of visual base models, which then complete the scene understanding, geometry and motion recovery, and physical assembly.
OVOW Method Overview (Phase 1x3): Objects from the video are detected starting with motion classification, respectively restored /static/concave Mesh, variable mesh sequence, true scale and stepwise motion.
The system first identifies the instance with the type of motion and generates a translucent mask, then completes the full shading area, restoring the mesh of static, rigid and variable objects; it then estimates the true scale with the step-by-step motion, and finally corrects the ground, gravity, suspension, penetration, and support relationships.
Critical design does not rely on pre-defined skeletal or category templates, and directly describes static, rigid, and non-rigid movements using vertex transformation.
A set that covers static, rigid and non-rigid movements
OVOW uses the example of Mesh as a base, separating the overall trajectory from the local shape, so it can handle cones such as vehicles, aircraft, and non-cones such as fleas and polar bears at the same time.
This is the case of a tabletop.
Dynamic GIF: On the left is an input video consisting of a black pickup truck, pumpkin and flowerpot, and on the right is an example-level 4D Mesh scene of OVOW recovery.
Case two: Multiple indoor objects
Dynamic GIF: On the left is an input video consisting of a black pickup truck, pumpkin and flowerpot, and on the right is an example-level 4D Mesh scene of OVOW recovery.
Cases of three-dimensional flying patterns
Dynamic GIF: Left side for wild flying video, right side for OVOW restored topographically consistent non-rigid 4D Mesh.
The number of airports in Hong Kong
Dynamic GIF: Multi-object video of the airport on the left, and an example-level mesh scene of the OVOW recovery on the right. The aircraft, the ferry and the terminal were rebuilt separately.
Cases 5 and 6 of polar bear shapeshift
Dynamic GIF: Video of polar bears and multiple objects on the left, and an example-level 4D Mesh scene of OVOW recovery on the right.
From the results of reconstruction to the last kilometre of physical simulation
The key to simulation is that the scene can still produce new results after changing the initial conditions.
Simulation and editing
OVOW can be repositioned or exerted, and new collisions, rolls, and displacements can be calculated by the physical engine.
Case One: The original layout
Dynamic GIF:OVOW Project Homepage Simulation and Editing Toy-Car Tabletop I in the Simulation and Editing Kit. Rebuilt black SUVs, trucks, flower pots and desktop assets collided, collapsed and moved in a new initial state.
Case 2 - Changing the layout
Dynamic GIF:Toy-Car Tabletop II. The flower bed is located at the front of the vehicle, with the brush and book taking on a new starting position; gravity and contact give rise to different collisions and displacements.
Cases of re-incorporation
Dynamic GIF:Toy-Car Tabletop III. Foam, flower pots, books and black pickup trucks were reassembled to produce a third set of rolling, sliding and collision processes by the same group of rebuilding assets.
Three animations repeat the same set of assets, only changing the initial layout produces a different trajectory; this is not video repositioning, but the result of a physical engine recalculation.
Rebuilding quality and reliability of pipelines
The team constructed static and dynamic scales for 120 scenes, each with 3 to 5 objects, and provided mesh, trajectory, camera, depth, and split real values.
On a dynamic scale, the Scene-IoU-OBB is 0.440; the Object-IoU is 0.210, and the speed is 3.35 seconds/h. Movement type identification, posture recovery, effective scene and gravity simulation stability is 95.4%, 92.4%, 86.8% and 82.7%, respectively.
When ordinary video becomes a data input for physical AI
For Physical AI, RGB pixels are just inputs. Strategic training really needs to be executed in states such as object boundaries, geometry, scale, trajectory, and contact and support. OVOW turns video into an instance-level 4D Mesh, and then simulators change layout, objectives, and external forces to extend a multitude of counterfactual trajectories from the same observation.
This also allows OVOW to be the upstream of the D4RT data pipeline. D4RT uses a unified time and space query interface to learn depth, camera and 3D point trajectory, while training time and space query from real-valued trajectory data; OVOW provides object-level mesh, camera, depth and motion and generates longer, controllable, real-valued tracking sequences in simulation. The two are not the same, but can be interconnected in data production: OVOW is responsible for converting real or generating video into real worlds, and then outputting the D4RT reconstruction and tracking tasks required for training samples.
The data links common video to instance-level 4D Mesh, physical simulation, controllable long-range trajectories, and 4D reconstruction, tracking of models and robotic strategies.
Thus, the value of OVOW is not just to reproduce a video, but to turn a single observation into a data source that can be intercepted repeatedly: repositioning objects, making collisions, shadows and long tail states, and then handing these previously difficult-to-collect processes over to world models and robot strategies.
ONE VIDEO, a video recording of ONE WORLD, should not only be about the appearance of the world; it can also be a physical world that can be understood, edited and simulated.
Introduction by the author
The team of authors from Qinghai University, Chinese University of Science and Technology, SparcAI, Hong Kong University of Technology, University of Electronics and Technology, South Korea University of Technology, focused on 3D vision, dynamic landscape understanding and physical intelligence.
