As video diffusion modeling capabilities continue to improve, starting from a single image and generating new perspective video along user-defined camera trajectories, it is becoming an important technological route for World Modeling.
Camera-Controlled Video Diffusion Models have been able to use camera parameters or geometric conditions to control changes in perspective and generate videos of higher visual quality. However, this method relies largely on implicit time-space representation, lacking explicit 3D geometric grounding. Especially with large perspective changes, models are prone to unreasonable geometrical structures, object shapeshifting and cross-angle spatial inconsistencies.
In response to this, HiDream.ai proposed DreamWorld: Geometry-Grounded Video Diffusion for 3D-Consistent World Modeling and was adopted by ECCV 2026.
DreamWorld's core idea is to combine the spatial architectural precedents of the 3D Foundation Model with the high-quality generation capabilities of the Video Diffusion Model and explicitly represent Geometry as an intermediate structure in the world generation process.
Unlike completing all generating tasks directly in RGB space, DreamWorld uses a two-stage framework called Geometry-then-Appearance:
First, the geometrical structural characteristics of the target perspective are generated, and then the final RGB video is generated with these characteristics as conditions.
Figure 1: DreamWorld contrasts concepts with existing Camera-Controlled Video Diffusion Models. In contrast to the lack of explicit 3D geometric grounding, DreamWorld represents geometry as an intermediate structure and introduces the spatial structural precedents of the 3D Foundation Model to enhance cross-sectional consistency and geometric stability.
The title of the paper is DreamWorld: Geometry-Grounded Video Diffusion for 3D-Consistent World Modeling.
The main page of the paper:
Link to the paper:https://yanghb22-fdu.github.io/DreamWorld/asserts/DreamWorld.pdf
The main highlights of this article
Geometry Drive: Introducing the spatial architecture pioneers of the 3D Foundation Model into Video Diffusion, which provides explicit 3D geometric grounding for the Camera-Controlled World Generation.
Structural intermediate mode: Geometry is elevated from the implicit results in video generation to an explicit intermediate structural pivot, representing the geometrical structure from the master to the target perspective, and then guiding the final video synthesis.
Conclusion: Using the Geometry-then-Appearance two-stage framework, 3D structural reasoning is modeled separately from high-quality appearance generation, while geometric consistency is combined with visual quality.
Effects lead: DreamWorld leads on benchmarks such as RealEstate 10K, Tanks-and-Temples and WorldScore.
01 Camera-Controlled Video Generation，
What else is missing?
In recent years, video scattering models have expanded the boundaries of the model's ability to generate models. In Camera-Controlled Video Generation, users can assign an initial scene image and specify a camera trajectory, and the model then generates corresponding new perspective videos based on the camera's motion. This method allows the video generation model to gradually have the ability to explore scenes and become one of the important technical routes of World Modeling.
Existing methods typically use camera parameters, Plücker embedding, or point-cloud projection views reconstructed from single images as generating conditions. These methods can already achieve better camera control, but the video scattering model itself still relies mainly on implicit time-space representations to infer the scenic structure of the unobserved region.
Under the larger viewpoint change, this mode of modeling with no apparent 3D grounding could lead to:
geometrically implausible structures；
distorted object shapes；
cross-view spatial inconsistencies。
In other words, as the camera continues to move, the model will not only need to generate new image content, but will also need to continuously deduce an unseen three-dimensional structure.
02 generated from video,
To Geometry-Grounded Video Diffusion
DreamWorld's starting point was to rethink the role of Geometry in Video Diffusion.
In many existing ways, geometry is more like an auxiliary information: camera parameters can tell the model how to move the camera, and geometric projections can provide some of the visible content from the target perspective, but the final structural complement and visual generation still need to be done by the same video model.
DreamWorld chose a different path: no longer using Geometry as an ancillary condition, but modeling it explicitly as an intermediate structural pivot in the production process.
In other words, DreamWorld does not directly generate the final target video from the input image step, but divides the generating process into two consecutive stages: given Input Image and Camera Trajectory, the model first performs Geometry Generation, generating the corresponding structure representation of the target perspective; subsequently, Appearance Generation uses these geometry features as a condition to further synthesize the final RGB Video.
The first stage represents the structure responsible for generating the corresponding objective viewpoints. The second stage represents the completion of the final visual synthesis based on these structures. Thus, Geometry is no longer just an implicit by-product of RGB production, but becomes an intermediate bridge connecting 3D spatial structures and high-quality video production.
Figure 2: When directly using warped partial images as a condition, the output results are vulnerable to structural deficiencies and projection artifacts; DreamWorld first completes structural reasoning in geometry feature space and then performs appearance synthesis to obtain more stable cross-sectional results.
03 Structural precursors with 3D Foundation Model
If geometry is to be modeled explicitly, a natural question is: What is the geometric structure that represents co-operation in video generation?
DreamWorld chose the intermediate characteristic of the pre-trained 3D Foundation Model as a geometry representation. This characteristic is derived from large-scale 3D/multi-view data pre-training, which can encode geometry-aware information related to scene-space structures, rather than directly in RGB pixel space.
Specifically, DreamWorld extracted multidimensional geometric features from the 3D Foundation Model and compressed them into representational spaces suitable for the Video Diffusion Model using a lightweight projection module. These geometry features were then used as explicit structural conditions to guide subsequent video production.
Therefore, the key to DreamWorld is not to directly predict the outcome of a particular apparent geometry, but to convert the spatial structural advance learned from the 3D Foundation Model into intermediate structural representations that can be used by the Video Diffusion Model, connecting 3D geometry modeling with high-quality video production.
04 Appearance Video Diffusion：
Let generations be built on geometric structures
DreamWorld first trained an Appearance Video Diffusion Model. Its goal was not to independently complete the entire scenario, but to produce high-quality RGB video on the premise of a given geometry-aware representation.
Specifically, the input video first gets latent representation through the pre-trained VAE Encoder. At the same time, the video is entered into the pre-trained 3D Foundation Model, extracting multi-scale geometric features. However, the feature space of the 3D Foundation Model is not naturally consistent with the latent space of the Video Diffusion Model.
As a result, DreamWorld introduced a lightweight MLP-based compression module that mapped high-dimensional geometry features into representations better suited to use by the Video Diffusion Model. These geometry features were then spelled with the noisy video latent on the channel dimension and entered together into the Appearance Video Diffusion Model.
In this way, the task of the Appearance Model becomes clearer: to focus on producing high-quality visual content under given structural conditions.
Instead of assuming two goals at the same time: structural reasoning and appearance generation.
Figure 5: DreamWorld uses the two-stage generation method of Geometry-then-Appearance: first to generate geometry representations corresponding to the target perspective based on the input image and the target camera trajectory, and then to input them as structural conditions into the Appearance Video Diffusion Model, ultimately generating RGB video.
05 Geometry Video Diffusion：
Structural representations of the objective view produced
The training phase can extract complete geometry features from real videos. However, in real reasoning, only: an initial image; a user-defined camera trajectory is entered. Therefore, a key question is: where did the geometry features from the target new perspective come from?
DreamWorld further designed the Geometry Video Diffusion Model for this purpose. First, the model represents the initial scene from the input image and projects it to a new perspective according to the target camera trajectory. These warped observations are inherently incomplete as only a single image is input: new views expose areas that were previously invisible, while also showing shadows and lack of structure.
DreamWorld entered these incomplete warped observations into the 3D Foundation Model, and received the corresponding incomplete geometry features.
Subsequently, the Geometry Video Diffusion Model generated conditions in the geometry feature space: Specifically, the Geometry Video Diffusion Model used incomplete geometry features as conditions, generated conditions in the geometry feature space, and predicted the corresponding complete geometry features from the target perspective.
In training, the geometry features extracted from real-world target video are used as learning objectives. Through feature-level flow matching, Geometry Video Diffusion Model learns to recover complete geometry representations corresponding to new perspectives of the target in incomplete structural conditions.
Therefore, DreamWorld did not complete all the missing areas directly in RGB space, but rather: first complete the structure generation in Geometry Space and then enter the Appearance Space to complete the visual generation.
06 Geometry-then-Appearance：
Disconnect the two missions.
DreamWorld's overall reasoning process eventually formed a clear two-stage pipeline.
Phase one: Geometry Video Diffusion Model: input the initial observation and target camera trajectory to generate geometry representations corresponding to the target perspective.
Stage Two: Appearance Video Diffusion Model: The geometry representations generated in Stage One are used as structural conditions to make noise in VAE latent space, and are eventually decoded into RGB frames.
Thus, the entire process can be summarized as: in the process of complete reasoning, DreamWorld first performs point-cloud-based warping based on initial observation and camera trajectory, and uses the 3D Foundation Model to extract geometry features from these incomplete observations.
Subsequently, the Geometry Video Diffusion Model generated complete geometry representations of these structural characteristics that corresponded to the target perspective; finally, the Appearance Video Diffusion Model represented these structures as conditions to produce high-quality novel-view RGB frames.
At the heart of this design is Geometry-Appearance Decoupling.
Geometry Stage focuses on structural completion; cross-view consistency; camera adherence.
Appearance Stage focuses on: texture; visual details; photorealistic r