From 2D pixel to 3D scene Query, ImageTrace is trying to move 3D content production to a new stage of scalable calling.
A few seconds later, an interesting scene took place.
In fact, they've risen from two-dimensional images to 3D content that can be played.
Without scanning, almost without waiting, it can be generated in a few seconds.
I also took six photos of coffee shops downstairs using my phone from different angles.
Even the bedroom at home, after a few clicks, becomes a 3D space that can be freely explored.
Meals, tables, graduation, moving, weddings, everyday life... It's really fun to have a cell phone that records important moments in a moment and sends a scene to a friend who can access it.
This app is calledThe Crystal Space CameraInSpatio has developed the latest research into a free 3D capture tool that makes it easy for anyone to experience the feeling of taking 3D space photos.
The technology behind this is the forward-looking, open-source 3D Gaussian Splatting, which produces the QuerySplat technology. With only a small number of different viewpoints entered without providing a camera-like image, 3D scenes can be generated in a single, forward-thinking, second-scale, freely interchangeable viewpoint.
In large-scale DL3DV-Evaluation benchmarks, Topos-Lite scored the best results on both the PSNR and SSIM indicators in the 2, 4, and 12 view settings; it also ranked first in the 2 view and 4 view settings in the LPIPS indicator, which focused more on perceptual similarity.
A more intuitive change is that the edges of the objects generated by the model are smoother, the texture is clearer, and the cross-sectional structure is more stable.
Similarly, after entering five images, Topos-Lite generates more complete outlines, clearer textures, and cleaner stability than other methods.
This time, Retro opted to opensource QuerySplat, not just to disclose a technological achievement, but to push forward-thinking 3D generated from the forefront of exploration by a few teams, moving to a technological base that can be collectively verified and continuously generated across the industry, accelerating the real scale-up of 3D content production, and creating more fun 3D applications.
The main page of the project:
GitHub and Model Weight:
Technical report:
From pixel to scene:
Let's make Transformer work more than just two-dimensional images
Topos-Lite's focus is not only on making 3D scenes clearer, but also on making them more accessible.Changes the way models organize 3D spaces。
Topos-Lite architectural map
In the past, 3DGS has been used in a number of ways.It's all about pixel alignment.It can be understood as: the model starts with each pixel of each image, predicts what the pixel corresponds to in three-dimensional space, and then combines a large number of predictions into a single scene.
But it has aThe problem of yesterdayThe Gaussian point is still tightly bound to the pixel and camera radius of the two-dimensional image. A slight error in the pixel from different angles may result in conflict in the three-dimensional space, resulting in gravity, floating, structural deviation, or a large number of small, inefficient Gaussian spheres.
It's a bit like a couple of people giving the same drawing outline to the same bottle from different angles and then folding several transparent drawing outlines together. Ideally, the bottle mouth, bottle body and bottom should fit together exactly.
Topos-Lite is used for this purpose.Query-basedIt's a thought.
It no longer requires that each Gaussian point belongs first to a particular pixel, but rather to a single pixel.Let a set of learning queries serve as "scenario-level positions".• Each query can span different images, aggregate information from different regions, and generate a set of Gaussian Primitives in a continuous three-dimensional space.
This makes Transformer familiar with the Query and Attention mechanisms, which really take on the role of organizing three-dimensional scenes in this work: Query no longer works around two-dimensional tokens, but becomes a scene slot in continuous three-dimensional space.
A small number of images, without manual labeling, generated on a secondary basis
With the input of a large number of unmarked hand-to-hand photographs, the model is able to produce 3D Gaussian scenes that can be freely switched quickly by reasoning in front of and without camera positions. Compared to other methods, the material outlines are more complete, the texture is clearer, and the scenery structure is cleaner and more stable.
Traditional high-quality 3DGS often requires large quantities of images to be collected around a scene, obtained or calculated camera parameters, and then optimized for a longer period of time for that scene alone. It can get good results, but every time you encounter a new scene, you have to do it again.
The forefront model is changing the cost structureAfter model training is completed, a new scene is no longer optimized from scratch, but directly predicted. Just as image generation models are not retrained for each new image, 3D scene generation also begins to be produced separately from each project, becoming a universal capability that can be repeatedly called upon.
For the average user, this bringsThe three most immediate changesFewer images needed to be taken, no need to manually provide professional camera labeling information, waiting time from traditional process compression to second level.
As a result, Topos-Lite was achieved in multiple rare view settings of DL3DV-Evaluation.Currently the strongestThe overall performance.
This is especially true in the 4 view setting.Ranked first in all three indicators of PSNR, SSIM and LPIPSEven if only two images are imported, it retains its leading structure and rendering quality.
The quality comparison.
In addition to the SOTA effect, Topos-Lite also provides clearCapacity expanded interface。
During the training, the number of queries can be gradually expanded from 1,024 to 8,192, with each query decoding a set of Gaussian Primitives. Stronger geometric skeletons, larger decoders, more queries, and higher-resolution appearance features can be expanded separately.
This can't simply be called the "proven 3D Scaling Law", but it does at least illustrate one important thing: Forward-facing solves the marginal generation costs of new scenarios, while query-facing leaves a continually expanding input for scenarios. Speed and quality no longer have to be optional.
It is worth mentioning that the performance of Topos-Lite is still in the early stages of research, far from touching the ceiling. With the continuous drive of data-scale effects, the quality and precision of its 3D production will continue to leap; more importantly, in the future, the model will be capable of running efficiently on the edge, achieving rapid localization of 3D content and unlocking wider application scenarios.
Why does the next generation of AI need to understand space?
In retrospect, in the last decade, AI has been primarily studying the world that humans have already expressed, and in the next stage, it needs to understand the world itself further. Language is the gateway to intelligent understanding of human civilization and space is the foundation of intelligent access to the real world.
Images and videos, of course, contain a wealth of spatial clues, but in essence they are still a two-dimensional projection of the three-dimensional world at a specific time, from a specific perspective. The same room can take many different images, but the three-dimensional structure of the room itself remains unchanged.
This is why we are here.2D is more like Observation and 3D is closer to State.。
If the model only learns from two-dimensional pixels, it must not only understand the scene, but also repeatedly process the changes caused by perspective, projection, shading, and lighting. The large amount of noise in the video can only be recorded repeatedly at different locations, at different times, on the same wall, on the same table.
The introduction of 3D means separating "what the world is" and "how we see the world". The model can learn a more compact, more stable spatial representation and understand how this state generates different observations from different perspectives and conditions. It does not have to repeatedly remember a large number of similar images, but also has the opportunity to continuously track the environment, objects and their relationships.
This also explains why spatial intelligence is not a marginal branch of AI. Linguistic models can answer "what is it" and "what does it mean", but entering the real world machines must also answer "where am I", "what is the relationship between objects", "what happens next", and "how should I act".
It's time to move from digital intelligence to physical intelligence.Space is not an option, it is a basic skill that needs to be mastered, and it is a threshold that cannot be overcome.。
A technological breakthrough behind nearly two decades of accumulation
Space intelligence is a path that is difficult to follow in the short term. From camera modeling, labeling, SLAM, to 3D reconstruction and geometric optimization, to neural network training, data systems and product deployments, each link has a large amount of engineering detail that is not easily seen by the outside world.
With nearly two decades of accumulation in the fields of SLAM, 3D vision and spatial intelligence, ShotTrace's core team has developed a comprehensive system of capabilities for basic research, engineering implementation and industrial deployment.
For more than two decades, the founder of PhotoTrace, Professor Chao Kang, has been asking a question:How can machines understand a real three-dimensional world?
At the undergraduate and graduate level, he studied computer science. At the graduate level, he joined the team of Professors at the University of Hong Kong, where he began to work on the problem of camera tracking in large-scale scientific research projects. First, he solved the problem of how to restore the movement of the camera from a video.
These technological directions point to the same thing: a machine cannot only see a single image, it must also know where it is, how the surrounding objects are distributed, and why what it sees from different points of view still belongs to the same space.
Since then, Chongqing has gone through three-dimensional technology, from camera tracking, mobile terminal SLAM, real-time space reconstruction, to the development of NeRF, 3D Gaussian Splatting, generative 3D and world models. Technical expressions are changing, but the core problem remains the same: how to restore, represent and unfold a relatively stable three-dimensional world from limited two-dimensional observation.
If university studies have brought him closer to the technical nature of the problem, his experience with digital space related technologies and business in business has given him a different perspective:A three-dimensional technology established in a paper and whether it can become a basic capability in the real world are two different things.
In order for 3D technology to enter the product, not only in the pursuit of precision, but also in the face of speed, stability, equipment costs, system efficiency and customer delivery; moving from a beautiful demo to a large-scale application, it is necessary to optimize the project capacity of each scenario individually into the capacity of a system that can be continuously replicated and scale calls.
Over the years, this dual-identity exercise has enabled him to transform the cutting-edge algorithms of spatial intelligence into the real driving force behind industrial change. These theoretical breakthroughs from the academic end to the industrial end, further amplified by the engineering capabilities of co-founder Dr. Min Min and his team, have completed model training, efficiency optimization and system verification at the industry level, enabling the cutting-edge algorithms to respond to extreme real-time penetration under limited computing power.
The value of long-term collaboration is far more than mutually familiar. In underlying technological research, many problems do not have standard answers, and technical routes often cannot be proven beforehand. Teams need to be able to quickly expose problems, question each other's results, and still form common judgments when information is inadequate, direction uncertain.
In the interaction with Shot Retriever, Machine Heart discovered that Shot Retriever's team structure was also exceptional:Both long-term accumulation and keeping the young team dynamic。
Several core members have been deeply involved in the field of 3D vision for over a decade, building a long-term infrastructure around 3D research; at the same time, many young researchers can participate directly in core topics and undertake critical work.Topos-Lite's key breakthrough was led by a team of "post-00" researchers.The long-term accumulation with the rapid growth of young researchers is not contradictory here, but together constitutes the research efficiency of the team.
In this way of organizing, there is a back-and-forth.A judgement of research effectivenessResearchers with real potential should approach the most important issues as much as possible and grow rapidly in real-world tasks. Age, qualifications and titles can serve as a reference, but should not be the boundary of talent for core innovation.
The first step is to get into the real world of humans, and then become the data that machines understand the world.
Of course, the endpoints of spatial intelligence are great, but the value of 3D technology doesn't have to begin to emerge until robots have actually reached thousands of millions of homes. The more realistic path is to lower the production threshold for 3D content, allowing it to enter people's existing records, exhibitions and creative needs.
The first is the spatial record.When three-dimensional generation is simple enough, fast enough, and cheap enough, people can turn several photos into "spatial photos" that can be freely switched angles, re-entered. The first home to be moved, the room where children grow up, the pre-graduation classroom, or the wedding hall can all go from a set of flat images to a more complete spatial memory. Past two-dimensional images have become mainstream, largely due to their lower cost of capture, storage, and dissemination.
The second is space exhibitions and remote experiences.In real estate, hotels, trade shows, retail stores, and interior design, users are often genuinely concerned not only with a few well-chosen graphic images, but also with the space layout, the relationships between objects, and the changes that people see as they move around. Through a small number of on-site photos, three-dimensional scenarios can be quickly generated that can be browsed, allowing rooms, ballrooms, galleries to be visited, decorated communication, and product presentations to be expressed more directly, while reducing the cost of traditional three-dimensional collection and content production.
There are also generated content productions.For creators of games, video and interactive content, this means that a more direct connection can be established between two-dimensional production, video production, and three-dimensional scenes. In the future, a piece of video or a few conceptual graphics can be further transformed into three-dimensional assets that can be freely browsed, edited and used for virtual photography, thus entering the workflow of games, video, e-commerce and interactive content.
For robots, these three-dimensional worlds are even more important. When the scenery has further object semantics, dynamic changes, and physical laws, they are no longer just content for viewing, but can also be a training ground for machines to simulate, validate, and learn. Before entering the real world, robots can experience the environment, understand space, and learn action in the countless generated, controllable, and reproducible virtual worlds.
First, allowing 3D technology into the real world, and then allowing the real use of data to be trained with greater spatial intelligence, could be a real path to scale up 3D data.
Endgame: Let's really get into space
Today, the "world model" has become a widely used concept. From video-generation, interactive game environments, to robotic control systems, different technological routes have been incorporated into the discussion of world models.
That's right.Back to the FutureIn other words,The world model is the technological path to achieving goals, and space intelligence is the long-term direction.
It wants to build an intelligence that can test its ability to understand space in the real world: to know where it is, to understand the relationship between objects, the environment and itself, to predict how the world will change, and to judge the outcome of actions.
Obviously, Topos-Lite is not the end. But it solves one of the basic and realistic problems: allowing Transformers to quickly organize stable, clear, and reproducible three-dimensional scenes from a small number of two-dimensional observations using clear three-dimensional structures.
More importantly, it also reflects long-term judgments about spatial intelligence, implementing technological outcomes that can be made public, quantifiable and continuously iterative.
It remains to be seen whether the future of space intelligence will continue to evolve along the Transformer, or shift to JEPA or new architectures that have not yet emerged. However, no matter how the architecture changes, space understanding cannot be bypassed as long as machines need to enter the real world; this ability cannot be separated from large-scale, high-quality 3D and 4D data, and the infrastructure that continually reduces the cost of data production.
In this enormous narrative, it is this set of models, data and engineering infrastructure for physical intelligence that we want to be part of. The future of robots, smart cars, smart glasses and new forms of terminals will require a master brain to understand their space and to perceive, predict and act accordingly.
Topos-Lite is a step-by-step proof of this long journey.: 3D scenes can be generated faster, clearer, and more openly invoked; Transformer's processing objects have also begun to extend from language and 2D tokens to continuous 3D space.
When space can be recorded, generated and shared at a low cost, like images and videos, a new network of 3D content may just open its door...
RetroTech's long-term commitment is to invite researchers and engineers from 3D Vision, AIGC, World Models, Multimodal Large Models, AI Infra, AI Systems Engineers, AI products and cutting-edge AI research to join us in exploring more possibilities for space intelligence and 3D world models.