How much do you need to prepare to hand over an ice cream shop to a robot?
Mechanical arms, dexterous hands, cameras, and models responsible for perception and decision-making make up the visible system. However, according to Heisei co-founder and CEO Li Seong, to make robots really work, there are also a number of problems related to "space".
Speaking at the conference on Sharpa's collaboration with DQ, he outlined three goals that the team hopes to achieve: not changing the original environment as much as possible, completing the complete loop of tasks, and undertaking real production work. He explained that making ice cream involves more than 50 steps, and the team hopes robots can take over the entire process.
In order for the robot to take over the process, the team first prepared a "digital store" for it: moving the layout, equipment and operating space of a real store into a virtual environment for training.
He said that in the past, it took about 300 hours for a team of professionals to create an artificial modeling to restore a store, and their equipment was able to complete the scan in about 15 minutes and generate a highly restored 3D training environment; Sharpa's failure rate has dropped to a tenth of the original, with a 99% success rate.
This is also the issue that is being discussed in space intelligence. Li Fyifei generalizes space intelligence as AI's ability to perceive, reason, and act in three-dimensional space. Compared to a visual model that only answers "what you see", it further requires AI to understand where objects are, the geometry of space, and how the world changes after an action occurs.
So how does a real store turn into a place where robots can learn repeatedly?
On 14 October, Heisei released the three-dimensional world data engine Kosmo, including the space camera Kosmo One and the cloud platform Kosmo World. The former is responsible for collecting real space, the latter for rebuilding, understanding and editing, and delivering data to robotic simulations, world models, and 3D creations.
What information is left after a shop is scanned, and how does this information turn into a training trajectory that robots can use?
Why build a space camera specifically?
What kind of digital shops do robots need?
The color and texture of the surface of the counter, which affects visual recognition; the position along the table, the distance between the devices, is related to the approach of the path to the operating space; changing the angle of observation will change the observation of whether the cup is held by the device.
Therefore, a digital environment that can be used for training needs to maintain appearance, geometry and spatial relationships at the same time.
Of course, professional teams can do this by using traditional CG modeling, but the problem is that shops, factories and hotels in the real world are different. If you start modeling from scratch every time you enter a new scene, the cost will increase rapidly with the complexity of the real world.
On-site scanning offers another path: no longer assuming a standard space, but directly recording a specific location.
Kosmo One integrates laser radar, imaging systems and high-precision IMU into one device. The laser radar provides space measurement, camera records color and texture, IMU provides information for device movement estimates, and synchronizes, labels, and converges data from different sources into a unified reference space.
What is the meaning of millimeter level accuracy?
Cosmo's custom laser radar is able to clearly show these fine-grained structures. This demonstration gives the abstract precision indicator an intuitive reference: when scanning objects from the outline of the room become fine-grained structures, the measurement capability determines which details can be left behind.
The imaging system uses one-inch CMOS, which supports up to 8K shooting, and ultimately exports the highest resolution of the file to 6K. But in 3D reconstruction, there is another problem besides the resolution: Is the same object captured by multiple cameras consistent?
The scanner walks around the room and the same floor enters images repeatedly on different cameras. As the camera's direction and background change, processing such as auto-exposure, white balance can be adjusted differently to give the same surface different brightness and color.
These features can improve the perception of a single photo, but 3D reconstruction requires organizing multiple moments, multiple perspectives. If the appearance of the same surface changes continuously, the continuity of texture restoration and perspective switching will be affected.
Therefore, Kosmo has customized the three-dimensional reconstruction of the ISP, a system that processes the original signal of the image sensor into an image. It needs to take into account the consistency between cameras, between moments, so that different images can describe the same space together.
The geometry and the image also need to match precisely. When the scanner passes through the counter, the camera takes an image, the laser radar takes a set of measurements, and the device itself is still moving. The system needs to know when and where the information comes from, respectively, in order to align the edge of the counter to the edge in the space measurements.
Therefore, the completeness of a space camera needs to be reflected in the entire collection link: both to obtain sufficiently fine geometric and visual information, but also to allow these information to work together.
When introducing Kosmo One, Low emphasized a physical button and a screen button: the user can complete operations such as opening, scanning, etc. He set the goal for this product to make space modeling as easy as photography.
He said that under normal indoor conditions, scanning of 100 square meters of space can be completed in about 15 minutes, with specific times changing depending on the scene structure and collection requirements.
However, after the scan is completed, only a spatial record is obtained. The next key is how to process it into a training asset that can be used repeatedly.
How does one scan turn into a massive training track?
In the interview, Li Yan posed a question to the media: Will people understand 3D reconstruction as coloring a dot cloud, or putting photos on a wall?
He immediately gave an example: look at the clock on the wall.
People look at the same clock in different locations, and the shape, reflection, and detail they see changes. It is difficult to fully express the effect of the side observation if only a positive photo is retained. This kind of perspective change occurs as the robot moves in the environment.
Kosmo World's underlying system, Kosmo Spatial Core, can generate 3DGS, Mesh and Depth in a single reconstruction. Three expressions serve the needs of appearance, geometric surface and distance information respectively.
3DGS, or 3D Gaussian, uses a large number of Gaussian representations with parameters such as location, shape, transparency, and color to reconstruct scenes and support rendering from a new perspective. When the virtual camera moves along the counter, the system can generate a relationship between objects, textures, and shadows seen in the corresponding location.
Mesh uses vertices, edges, and faces to describe the surface of an object, providing a geometrical basis for collision and interaction modeling.
The three expressions maintain spatial alignment, meaning that the counter that the robot sees in the image is able to correspond with the counter in the geometric model, the distance in the depth of information, and the training system can therefore obtain different forms of spatial information in the same environment.
Next, it is necessary to place the virtual camera where the robot actually observes.
A person stands at a table that can be seen, which may be obscured by a table in a lower camera view. Changes in the height of the camera's installation, direction and angle of view, change the size of the object in the image, the range of view and the relation of obscuration. According to Haysay, the data generated by Kosmo can be visually adapted to the camera characteristics of the target robot, bringing the observation in the digital scene closer to the actual conditions of use.
The scenery itself can change.
Kosmo World supports AI-driven semantic partitioning, which can identify and extract objects such as chairs, sofas, cameras, and move and edit them.
Suppose a chair in the store is moved from the side of the table to the side of the aisle, the passage area will shrink, and some items may be blocked. Even if the camera moves along the original route, the images will be different. These adjustments can constitute new training or test conditions to observe the model's adaptability to changes in layout and shading.
The editors also expose areas not seen during the collection. Where did the carpet below come from after the couch was removed?
In the conference presentation, Kosmo uses AI inference to fill in the areas that were previously covered by the sofa. Such processing makes it easier for objects to move and scenes to be more complete. At the same time, the complementary part includes model inference, which the training team needs to determine based on the task requirements: whether it is suitable for use as a background, as opposed to the data requirements for use as a precise touch and operation surface.
With an environment that can be observed and edited from different points of view, trajectory generation has a basis.
In a virtual store, capturing a video requires someone or a device to move along the route; in a digital store, the location, direction, height and path of the virtual camera can be reset, generating continuous images along the route, and the corresponding depth and position.
More importantly, the same real space does not only correspond to a set of fixed data. By adjusting the position of objects, changing the observation conditions of virtual cameras, new scenarios and training tracks can be continuously generated. Heisey says that based on the same three-dimensional space, tens of thousands of training tracks can be automatically generated.
A spatial record can continuously generate data around different observation conditions, and teams can also add samples for easy-to-mistake locations. For example, a device that is often stuck to a target can set more observation paths around the relevant area to check for visibility from different angles.
But the information carried by different tracks is not the same.
The sequence of observations produced by virtual camera movements can be used for multi-point learning, spatial perception and related world model training. Operational trajectories for robots to complete tasks such as grabbing, moving, and other tasks involve joint action, contact status and mission outcomes, requiring collaboration between robotic models, physical simulations, and strategy systems.
So when a scene can be edited continuously and generate new training conditions, why keep going into more real stores?
In an interview, Lee said that the essence of robot training data solving is a problem of generalization: after a room is digitized, it can become a training sample of similar scenarios, but space gaps in the real world will continue to increase from dining to retail, hotels, and homes.
A room can change constantly, while more real rooms provide new structures, objects and spatial relationships. The former is responsible for continuing to magnify the value of a data, while the latter is responsible for continuing to provide a new "world".
In Lee's judgment, the more scenarios AI can handle, the greater the demand for real-world data.
The value of a single scan extends to the end of the scan.
How far does he want to go from a space camera to a data engine?
In the past decade, Heisei's most familiar business has been laser radar, with applications ranging from autonomous driving to robotics. In addition, Heisei's laser radar has also been used to obscure the myth: the ancient scanning of lighthouses.
In the interview, Li Yan answered one question: Why Kosmo, a company that makes laser radars?
The starting point of the product comes from customers. Customers in the fields of mapping, construction and others are constantly demanding higher-precision laser radars, advancing from the centimeter to the millimeter level. These customers will also have to configure cameras, guidance and computing platforms to build a complete space acquisition system after getting the radar.
Similar integration work was repeatedly carried out by different teams, allowing Heisei to see product opportunities: designing sensors, collection systems and subsequent data processing together. The ability to accumulate in the past in complex optoelectronic systems, customized chips, labeling testing and productization can also be extended to space collection.
This extension has a clear engineering connection. Cosmo needs to simultaneously process geometric measurements, image quality, sensor synchronization and on-site operations, and the performance of individual components is ultimately transformed into the availability of the entire system.
After entering the field of training assets, feedback on the application side is equally important.
While Hays and Sharpa operate independently, the co-founding team provides closer communication conditions for both sides. The robotics team can put forward specific requirements: which viewing angles are easy to lose targets, whether objects can be easily simulated after splitting, and how much artificial processing is required to rebuild assets.
This gives data quality specific discussion metrics. Whether a three-dimensional scenario looks realistic can only answer partial questions; whether it fits a robot sensor, whether it can conveniently configure tasks, depends on when the development team can start training.
Cosmo has connected platforms such as Isaac Sim, Isaac Lab, Omniverse, etc. The convenience of tool compatibility, object editing and asset processing will all affect final delivery efficiency. As customers expand to different core and training frameworks, these links will also become a part of the product that needs to be continuously refined.
Some of the needs were beyond the team's initial expectations.
In an interview, Lee described the use of video as "unexpected harvests and unexpected surprises". Dylan Sisson, creative director of Pixar RenderMan, took Kosmo and scanned Pixar Park Square, imported the data into Houdini in USD format, and then rendered it to RenderMan.
He also scanned a handmade cardboard TV frame. The texture of the cardboard, the edges and the watermarks were preserved to be the details that could be invoked in subsequent works. For the creator, the value of a single scan also reflected how much editing and expression space remained after these details entered professional software.
Gode's application brings three-dimensional assets to space browsing. It is known that Kosmo, in collaboration with Gode, has launched the first reconstruction of heavy art galleries and façade galleries online to scan the list of "flying streets", allowing users to see the inside of the site before visiting.
The expansion of use has also led to changes in delivery methods.
Some customers need equipment in order to continue the acquisition; some customers just want to complete a project and get a rebuilt model; and some customers want to directly use the space assets that have already been made.
Accordingly, he proposed two types of business models: customers who are continuously accumulating demand through software-hardware integration product services, and large-scale spatial data production services for global model and physical AI enterprises.
The latter involves scene organization, quality control and continuous delivery. After the equipment is sold, whether the data can be used smoothly, how many times it can be repeatedly called, will further affect the customer's judgement on the value of the product.
This is also part of Kosmo's ongoing exploration. When asked who has the largest customer base, Li Yan did not give a definitive answer in the interview.
For Hay Race, extending from the sensor to the data engine means that the delivery object has changed. Laser radar's measurement, accuracy and reliability can be assessed by relatively clear indicators; its value is also related to mission, model and simulation configuration once 3D assets enter training.
Kosmo is trying to change the way it has done in the past: "take one, model one, use one". After a real space is digitized, it can continue to edit objects, change the conditions of observation, generate new training scenarios, and take one can also be the starting point for subsequent rounds of training.
In short, give AI a "real-world bottom line".
Over the years, AI's capabilities have grown rapidly in the eyes of everyone: from chess, text and images, to the exploration of real physical space today.
There is no doubt that world models and spatial intelligence are a hot topic at the moment. The future applications to which they correspond are also fascinating, ranging from autonomous driving, physical intelligence, and digital transformation. Many of the scenes that have existed in science fiction in the past are gradually gaining technological possibilities.
But as the heat rises, discussions about a "physical intelligence bubble" also emerge. Perhaps the way to address this concern is not to cool down the industry, but to build up the foundation a bit.
One of the key issues is data. Lesser Lord summarized the problem of data with physical intelligence as an "impossible triangle" at the conference: real-machine remote data is real enough, but it is expensive, low-efficiency, and difficult to scale; simulation data is easy to scale, but it is difficult to fully restore the complex physical laws of the real world; and 2D images and videos lack depth, location, and real-space relationships.
If a significant portion of AI training data in the past came from text, images, and video on the Internet, physical AI will need more data from the real world and an infrastructure to collect, process, and transfer it into training processes to grow further.
In an interview, Li Yan identified Hefei as "the builder of physical AI industry infrastructure", and stressed that this is not a single product line, but one that revolves around multiple fundamentals.
Kosmo is trying to turn the real world into a resource that AI can continue to learn and use.
© THE