The main authors of the paper include Ph.D. students Sun Yang Tian, Tianjiajia, and VAST Huang Jie; communication authors include VAST's chief scientist Cao Quang Bai and Hong Kong University's adjunct professor Xi Xiaoming.
Large models, such as the GPT-6 Astra, have become increasingly clumsy with 3D scenery. However, when the results are actually used, the geometry of the prospective object, its relative size and layout are still biased against the input image. The University of Hong Kong and VAST have introduced the Mira-Scene: a pixel-aligned layout ensures that the reconstructed 3D scenery and prototype are strictly consistent.
From left to right, Condition images, Mirascene + GPT-6 Astra and GPT-6 Astra From Scratch, respectively.
Recently, 3D modeling has made remarkable progress in object-level generation. Given an object image, models like Tripo have been able to recover a fairly fine-grained three-dimensional geometry. However, when the problem extends further from generating an object cube to reconstructing an entire scene cube, a new dilemma needs to be solved: where should the object be placed after it is generated?
To address this problem, the paper Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction proposes a new generative 3D scene reconstruction framework. It no longer directly returns the rotation, displacement and scale of objects, but instead transforms the scene layout into a compact, pixel-aligned 3D relationship recovery problem. Using only 80k open source data, Mira-Scene achieved a 39.8% improvement in 3D IoU accuracy and 16.5% improvement in projection 2D IoU compared to SAM3D.
The paper:
The main page of the project:
The code is:https://github.com/VAST-AI-Research/Mira-Scene
The new layout shows
The core idea behind Mira-Scene is to come up with a layout that is easier to learn and more generalized.
Although it looks very simple, this representation actually has two problems: 1) sparse and unbounded. All complex spatial information of an object is finally compressed into a few pose parameters; while parameters such as translation are directly defined in an open scene coordinate system, there is a wide range of numerical changes. It is not easy for the network to accurately return the entire spatial parameter directly from an image when there are ambiguities and complex spatial relationships. 2) Such representations rely heavily on scene-level 3D data with precise Layout annotation, whereas in real-world 3D object scenes high quality 3D data is far from having a level 3D asset.
Mira-Scene shows the layout of the object in 3D space using Canonical Coordinate Map (CCM) and Point Cloud Map (PCM), as shown in Figure (a):
Top left: ((a) Different layouts show contrast; right: ((b) Geometry & Layout Expert structure
In this case, the CCM records the coordinates (x, y, z) of each pixel corresponding to the canonical 3D model of the chair, while the PCM gives the pixel the corresponding 3D coordinates in the current camera/scene coordinate system, which naturally form a dense 3D-3D correspondence.
Geometry & Layout co-generated
To ensure that the generated 3D shapes and CCMs are better aligned, Mira-Scene further proposed a Geometry-Layout Co-Generation framework. As shown in Figure (b) above, Geometry Expert is responsible for generating the geometry of objects in 3D voxel latent space, while Layout Expert is responsible for generating CCMs in 2D pixel space. The two modules retain their own network parameters and representation formats, respectively, and interact through shared multimodal self-attention.
(c) Mira-Scene Training Pipeline
The two-stage training model
Another advantage of CCM given its definition in standardized canonical object space is that the training of the model does not require having to have a large number of full 3D scenes with precise layout labels, and a typical object-level 3D asset can produce CCM supervision. Thus, the paper adopted a two-stage training program: the first stage was Object-level Pre-training, where the authors used 60K 3D objects to render one million object-centric views, allowing the model to first learn object geometry and co-generation of CCM. The second phase is Scene-level Fine-tuning, which uses about 20K 3D-FRONT scene views to further adapt the model to real-world shadows, visual changes, and amodal object reconstruction. The training strategy can be summarized as:
(d) 2-Stage Training Paradigm
Scene reconstruction results
Mira-Scene is able to start from a single image, generate 3D objects, and restore the corresponding scene layout, as follows:
Thanks to the CCM-PCM layout of Mira-Scene, the 3D scenes generated can be directly aligned with the point cloud estimates of a single image, resulting in:
Mira-Scene has demonstrated significant advantages over existing methods. Significant improvements have been made in both the BlendSwap and 3D-Front benchmarks:
The following is a visualization of the results compared to the SOTA method:
Applications: From reconstruction of scenes to editing and interaction
As a combined scene generation method, the results of Mira-Scene can be used directly for object-level editing, animation and simulation. For example, removing objects, adjusting the position of objects, or reassembling scene assets, and then linking the assets to the animation production process. The following results show the effects of scene editing, mechanical arm grabbing, gravity simulation, and collision in the blender:
In addition, the results of Mira-Scene can also be used directly for physical interaction. Below are the results of simulating interactions in Mira-Scene scenarios using the action policy created by Robot using GPT6:
In addition to the training/reasoning of the Mira-Scene, the code library also contains a pipeline that restores a complete 3D scene from a single image, supporting the automation of the entire process, and welcome attention to experimentation:
Prospects
Mira-Scene hopes not only to provide a more accurate way to reconstruct 3D scenes, but also to further encourage the community to rethink compositional scene generation from a more scalable perspective: Rather than simply relying on ever-expanding scene-level data, more scalable object-level 3D can effectively improve the quality of scene reconstruction through rational representation design. To facilitate further research, the team has opened up full code, pre-trained model weights and training data samples and evaluation benchmarks, hoping to provide intelligent 3D scene reconstruction, editable 3D content generation, and an open, replicable research base, and to explore more widely available 3D scene creation models with the community.