In order for robots to truly enter the ordinary home, training and evaluation cannot be constrained by the efficiency of data collection in the real world. The simulation environment has thus become an integral part. However, most of the existing simulation environments have scant furniture and monotony, while the real rooms are cluttered and cluttered, with objects supporting and overshadowing each other.
Currently, there are two main lines that can generate the quantity of this room, one of which is:Intelligent bodiesOne way to generate a scene through text is throughParametrically, images are used to generate scenes.But the two main routes hit the walls.
Smart body language 3DIt takes 7.56 hours to produce a single scene in an experiment, which is slower than a manual swing.
Parametric 3D graphicsConstructing a scene to be broken down into an "objects-by-objects +6D position estimate", with the help of real-world image-bearing layout pioneering, it takes ten minutes to restore the entire scene, a cost that is not acceptable: direct-return positions are often physically unstable, often produce chairs hanging, books carrying a modular divider, and the collision rate is up to 20%-26%.
There is also a problem that both routes have not really been solved.It's not "living".Existing methods only give a definite layout of an input. But the real room will evolve as people's activities change, such as a chair being pulled off, a book being moved to another location, and overall function and physical relationships remain.Structured rearrangement to maintain consistency in the sceneRather than the existing methods, which are often only obtained after repeated sampling, small changes near the original layout.
For this purpose, a team of Deeb teachers from institutions such as the University of Hong Kong put forward a proposal.SceneMosaicMoses (Massaic) is the central metaphor:A scene is not a stationary whole.It's been a while.One pieceIt's a "local piece" that can be substituted independently.
Scene Mosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution
Written by Xingjian Ran*, Xiaoye Mo*, Sihao Liu, Jianyu Zhang, Li Luo, Bo Dai
Address of the paper:https://arxiv.org/abs/2609.05594
The main page of the project:https://rxjfighting.github.io/SceneMosaic
The code link:https://github.com/rxjfighting/SceneMosaic
How did SceneMosaic come about?
The overall process of SceneMosaic is divided into three phases:Scenes rebuilt and structured → Scenario combinations driven by diversity。
Step 1: Turn a picture into a "scene tree".Perceived intelligence first completes instance-level object registration, then SAM3 splitting, SAM3D independently reconstructing each object's grid and initial position; then using a DAG-scheduled dedicated functional intelligence body to infer the boundaries of the room and the space between objects.attach(Support) andcontain(capacity) Dependence, organizing the scene into a layered scene tree.
What's really important is the definition of a screw local unit cluster: a non-leaf node as a focal point, which together with the sub-nodes it carries directly constitutes a local unit, such as a book on a shelf and a book on a shelf, a display on a table and a keyboard is another. This is the core of SceneMosaic:They are essentially independent of each other and can evolve independently.A small number of functional relationships between local units (the chair should be facing the desk) are explicitly drawn as constraints so that the constraint itself does not in turn restrict the diversity of the scene.
Step two: Let the laws of physics speak, then let the mind move.Because for these scenarios, the most important characteristic is to satisfy the physical laws of the real world. Before the intelligent body intervenes, it does accommodative corrections and gravity simulations, allowing the object to unravel the collision itself, eliminating the gaps.Depending on the locationGround sliding is downward gravity, ceiling sliding is reverse gravity, wall sliding is along the outer line of the wall, so that the attached objects will naturally "sink" to the supporting surface.
This is followed by the Critic-Actor cycle:CriticRead a corrective rendering chart, report on collisions with historical memory, and issue qualitative suggestions that do not contain coordinates ("move the chair in the direction of the table"), to avoid being biased by unreliable numerical estimates;ActorTranslate the suggestions into EnglishSymbolic position expressions referring to the current layout stateIn order to prevent the intelligent body from falling into a deadlock, the research team also made three layers of protection: the strict boundary of the context, the reduction of 3D position adjustment and the convergence of 2D displacement and rotation, and the recording of repeated fail patterns in explicit layout memory.
Step 3: One evolution, one combinationOut of the seaThe candidate scenario.Because the local layout is expressed under the coordinate system of their respective focal points, the arbitrary variants of the subunits can be combined under the arbitrary variants of the parent unit, and a large pool of candidates can be obtained along the scenery, which is the cost of changing a large number of candidate scenarios at a time, which is precisely why the variants are cheaper than the base scenarios. Finally, the algorithm combines relativity, absolute distance, and the neoliberal distance of rotational differences, combined with dynamic Max-Min greedy search, to generate a set of compact and diverse representative scenarios from the large pool of candidates, and these variation scenarios accurately reflect real-world layout changes.
The results of the experiment
The research team is working onSceneEval-100It is used to measure physical effectiveness using NAV, COL, OOB, and navigation.
In terms of semantic quality, SceneMosaic is on par with the strongest baseline (POS 84.3 versus 83.5, even slightly better).Significantly reduced physical violationsAt the same time, it is 24 times faster than SceneSmith. It takes only 0.03 hours to generate a variant scene, faster than all baseline methods.
Qualitative contrast is more intuitive: SAM3D is rapidly initialized with visual precognition, but the position of objects is often inconsistent with physical conventions. Scene Mosaic preserves the overall spatial structure of the image and then evolves through layout to improve the relationship between objects, so that the chair and the book are properly supported on the floor.
Dissolving experiments showed that these designs were not impossible: removing the image precursor, which took from 0.14 hours to 1.25 hours; removing physical processing, which increased the collision rate from 0.0 to 18.9%; and using 3D visualization to show doubling of reasoning time and decreased quality.Once converted to global evolution, variable generation almost loses its original efficiency advantage.The time it takes to generate a variable is 21 times that of SceneMosaic.
In terms of diversity, the selection layout is much more similar after the greedy search has been replaced by random sampling, while repeat sampling is less frequent and more time-consuming.
So is this diversity just messing with the original layout?
The disruptive intensity increases and the differences between layouts do increase, but the semantic quality and structural coherence also decrease significantly.Diversity is considerable.SceneMosaic has a semantic positioning score of 84.5 to 71.6 and a correlation rate of 99.1% to 71.5%.This diversity is not at the expense of the effectiveness of the layout.
A user study of 48 participants also came to the same conclusion: SceneMosaic scored the highest score on both semantic rationality (4.33) and physical rationality (4.47) with the lowest standard deviation.
Limitations and Prospects
SceneMosaic relies on single image initialization and therefore inherits the natural limitations of 3D:An image can usually only paint a room.As a result, it remains on the room scale and cannot be directly extended to the entire home. Combining the results of room levels into a coherent home-scale environment is a very valuable direction.
The team believes that "picture pioneers are responsible for fast, intelligent evolution is responsible for multifaceted, and the department is partly responsible for multiple solutions", and that this is a path worth taking for building large-scale simulation environments. SceneMosaic is open source and welcome to discuss and improve.