Recently, a team from Oxford University & Monash University & Bristol University has proposed the interactive video world model Oneira, which moves from open generation to open world interaction.
Existing video world models have been able to continuously generate new scenes as the lens moves, but the intelligent body can usually only navigate or perform some simple interactions in them. Oneira maintains a World State Table that records the objects in the scene and their state through Coding Agent. Whether the objects already in the input image, or the newly generated objects as the lens moves, Oneira can change them accurately and allow the changed state to persist in subsequent videos.
The first author of this paper was Dundee, now a Ph.D. student at Monash University in Australia, and a Ph.D. lecturer at Prof. Huang Tzin and Prof. Feng Shui, whose main areas of research are video generation models and world models.
Project Page:
Paper:
Code:
References
Video world models have made great strides in recent years, supporting navigation and simple interaction. However, it is believed that when using these models, you will find that when a specific object in a scene needs to be changed, the model often works on the wrong object.
The team believes that this limitation stems from the fact that existing models do not record the state of the world separately. Which objects in the scene, which objects have changed, exist only in the image generated and the context above. Therefore, the model is difficult to distinguish between the individual instances of objects in the scene, relying only on the text description to infer the objects to be interacted. The results of the interaction also exist only in the image generated, and the progress of the video is easily lost. Game engines and traditional simulators can accurately record the state of each object, but both the scene and the object need to be preconstructed and cannot expand with the generated content.
To overcome this limitation, the research team proposed the interactive video world model Oneira, which shows what is happening in the world of photography and how photography is treated separately from photography.
The former is responsible for the Coding Agent, which records the location and state of each object in a World State Table scene. After assigning an action or goal, the Coding Agent first reads the World State Table, determines the objects to be interacted with and plans the interactions, and then rewrites the results of the interactions into the table. For new objects that appear when the camera moves, the Coding Agent will call a 3D sensing tool on the image generated, get its 3D BBox, and then add it to the table.
The latter was completed by the Video Generation Model. Engine first rendered the World State Table as a simplified conditional video, real-life appearance, action and interaction details, and then completed by the Video Generation Model.
The results of the experiments showed that Oneira was able to accurately change the designated objects and maintain the state after the change in the long video. This proved the feasibility of a truly interactive world that was created by separately recording the state of the world through the World State Table.
The method
In this work, Oneira, proposed by the research team, consists of four sections: Coding Agent, World State Table, Engine and Video Generation Model.
As shown above, the Coding Agent understands the objectives and plans interactions, Engine updates the World State Table according to the plan and renders it as conditional video, Video Generation Model and then generates real images based on conditional video.
Specifically, Coding Agent uses an existing 3D perception model to construct a World State Table from input images. The table only records information related to interaction, including walkable terrain areas, and each object's semantic labels, 3D BBox and interaction states. The appearance of the object is not written in the table and is generated by the Video Generation Model.
Coding Agent combines the World State Table with the current image to determine which apple and which chunk is to be picked up. It also deduces that the other two apples will roll down, and the two apples and targets will be written down together. The Engine then updates the World State Table according to the planned action, marking the three apples as leaving the original location, with the rest of the object remaining unchanged. Each time, the results of the interaction are written down in the table, how Apple moves, where it rolls, depending on the validation of the Video Generation Model.
In a conditional video, each object is represented by a colored box. A change in the color of the box indicates a change in the object's state and the disappearance of the box indicates that the object has left its original location. Each time a video is generated, the Coding Agent calls the same set of 3D sensing tools on a new image, adding the newly emerging object to the World State Table, so that the interactive object will increase as the exploration progresses.
Finally, the Video Generation Model describes the movements of the camera in the grid and color background in the conditional video. The 3D BBox indicates the object and time of interaction, while the text description indicates the type of interaction. When the camera returns to its previous position, the model also refers to the image as a memory pad. The 3D BBox only represents the state of the object in color, unrelated to the object category, so the model can also interact with the object category not seen in the training.
The results of the experiment
Interaction at the instance level
Long-range status maintained
Quantitative results
In the experimental phase, the team constructed the InteractionBench assessment set, containing 100 interaction tests and 50 memory tests, and made qualitative and quantitative comparisons with MiniMax-H3 Ref2VA, LingBot-World-V2, YUME 1.5 and AlayaWorld.
As shown in graphs and tables, in interaction measurements, Oneira scored the best results on whether interactions occurred, whether the object was accurate at the time, and whether the object was correct on all three indicators, 29,19 and 29 percentage points above the strongest baseline respectively. In scenarios involving multiple identical objects, the baseline model often works on the wrong object or changes multiple objects at the same time, while Oneira changes only designated objects.
In memory measurements, Oneira's PSNR and SSIM improved by 93% and 83%, respectively, after the camera left and returned to the same position, and the status retention rate improved by 18%. In VBench's video quality indicator, Oneira also remained at or near the best level.
The results of more experiments are shown below.
Interacting with open content generation
Multi-step interaction
More examples of interaction
Conclusions
While video world models have been able to generate scenarios for navigation, they lack a separate record of the state of the world, and interactions often work on the wrong objects, and the altered state is easily lost. The research team proposed the interactive video world model Oneira, which records the objects in the scene and their state through a World State Table maintained by the Coding Agent, so that it can accurately change the designated object and maintain the altered state. The experimental results confirmed that Oneira was clearly superior to existing methods in terms of interaction accuracy and long-term consistency.
