Robots have been collecting data for the past few years and have been trying to do one thing: get rid of the robot itself, make it lighter, cheaper, and easier to scale.
Initially, robots relied primarily on remote learning. Operators controlled the robot to complete tasks, the system recorded joint status, end-to-end activity, and environmental information. The data obtained in this way could be used directly for robot strategy training, but the cost was also obvious: every hour of data required real robots, operating equipment, and corresponding human resources.
Later, portable instructional programs such as UMI began to break free from the constraints of the body, and Ego first perspective collection further simplified the matter. People with cameras could walk into homes, offices, shops and do all kinds of operations in a real environment.
Data collection doesn't always have to revolve around robots.
But as the equipment became lighter and lighter, another problem began to emerge: We photographed how people did the job, but we didn't necessarily fully record their physical movements and contact conditions when they did the job.
For example, laying the bed. People need to walk to the bed, grab the corner, lift the arms to be stretched, and then move along the bed, a little flattening. The position of the body determines where the hands can touch, holding the position and strength affect how the fabric unfolds.
A first-person video can record what is happening in front of the eye, but it is impossible to directly measure how dry, leg movements, and pressure changes when hands and hands are in contact with fabrics. These are important training information for robots that learn to move their entire body with agility.
This pushes robot data collection into a new choice: continue to access more video with lighter devices, or replenish interactive information that has been discarded in the lightweight process?
Breaking intelligence chose the latter, but did not intend to return to heavy professional hunting.
On October 14, PHANES AI, a joint venture of PHANES AI Mobile, released the full-body model data collection system SeePhys Hub to attempt to explore a new collection model between high-precision but difficult-to-scale remote operations, moving data capture, and light but relatively limited Ego data.
The system consists of a fourth head ring, two wrist rings, three waist leg movement trackers, and a pair of touch gloves that can simultaneously record first-point vision (360° peripheral vision coverage), hand full-mode data (hand position tracking, high-density touch) and body motion information (position and force).
What it wants to do, it can simply be understood as a complete complement to the ego: both maintaining the human ability to operate freely in real situations, and recording how the body moves, how the hands touch objects, so that the data can more fully describe a real interaction.
For the breakaway, it's not just a collection of equipment, it's a re-option of the robot's data scaling path.
One, the ego solves the problem of scale.
Whole-body model wants to make up for missing information
How much data does a person with physical intelligence need? There are even estimates in the industry of 100 million hours of Ego data measurement.
The path of the Ego has quickly gained attention, and the reason for this is not complicated. It no longer requires every hour of training data to be generated by the robot itself, and human daily operations themselves can be a learning material.
However, the growth in data size does not necessarily lead to the same degree of learning benefits, especially when robots move from a fixed position with their hands to a full-body locomotion. For example, vision can record how their hands approach objects, how objects move, but it is difficult to directly judge how much force is applied when touching. The same hand trajectory, under different contact conditions, can produce completely different results.
Whole-body operation adds another layer of complexity: robots must know where the body stands, when to move, how to adjust posture, and how body movements interact with their hands.
In the absence of these states, the model can only infer from limited visual clues. More videos, of course, are valuable, but not necessarily able to solve the incomplete problem of input information.
At the bottom are Internet open data and ordinary Ego video, large in size, rich in scenery, but lacking in direct measurement of body posture and contact information; at the top are high-precision data such as machine remote operation, professional motion capture, which can provide reliable motion monitoring, but the cost is high and the collection conditions are more limited.
SeePhys Hub hopes to fill the middle layer: using a cost-near-Ego collection method to obtain human operational data with full body posture, hand movement and tactile information.
In his view, the middle layer is not a replacement for motion capture or machine data. On the contrary, the team's full-body gesture estimate model still needs high-precision motion capture data to provide supervision.
For model pre-training, not every human joint requires a millimeter-level positioning result. More realistically, the goal is to accurately meet model training needs, allowing data collection to access enough real-world scenarios.
He also hopes to connect different levels of data through algorithms: using high-precision motion capture data training gesture estimation models, and then trying to add gesture, touch, and other information to a mass of Ego video.
Real data collected by mid-range devices could provide more oversight and validation for this type of model. This ultimately leads to a question: Can the learning value of hourly data be improved through more complete interactions?
He hopes to explore whether million-hour full-mode data can support the expected generalization capabilities. However, this is still the direction the team is verifying. There are currently no publicly available model training results to prove that millions of hours of full-mode data can replace 100 million hours of Ego data.
Second, the hardware is cutting down, giving more work to the algorithm.
If you're going to have a full body, why don't you put more sensors directly into the human body? When it comes to designing the SeePhys Hub, you've actually dropped the question: How much equipment does a collector need to keep to make the data truly scalable?
He said he hoped the device would be as unpretentious as possible, cost low enough, and be able to scale up free collection in real environments.
This differs from the traditional point of departure for the pursuit of maximum accuracy. SeePhys Hub does not choose full-node inertial motion capture, but places three motion trackers on the back of the waist and both legs, and combines the two-wrist device with head vision to restore complete human posture through the model.
The team compared pure-visual gesture estimates to whole- node inertial motion capture. The former was light enough, but invisible body parts could only be guessed by models; the latter could provide more direct observations, but increased the number of devices, difficulty in wearing and cost.
SeePhys Hub ultimately opted to merge visual, IMU and UWB.
IMU provides additional constraints such as node-to-node distance measurement by recording movement changes such as acceleration, angular velocity, and UWB, both combined with vision, to mitigate the shading problems of pure vision and cumulative drift when merely relying on inertial information.
Based on data provided by the team, the total body posture estimate model shows that the median body node positioning error is less than 1.5 cm.
The head vision is similarly built around the concept of completeness. The SeePhys Hub features four wide-angle fish-eye cameras, a 193° horizontal field of view and a 128° vertical field of view. The front camera records the main operational images, while the rest of the camera expands the scene coverage, reducing movement away from the field of view.
For example, after the collector has removed items from the refrigerator, the refrigerator door is closed laterally, and it is not usually possible for a front-facing camera to fully record the process, while a multi-directional camera can retain more scene information.
The headset also integrates a 500Hz high-frequency IMU, which is used to supplement the movement between adjacent vision valves when rapidly turning. In terms of multi-mode data synchronization, the UWB also assumes clock-to-clock synchronization functions between devices. The team introduced that its underlying synchronization program has a theoretical accuracy of 10 nanoseconds; the headset's internal camera, IMU and UWB synchronization error, however, does not exceed 0.5 milliseconds.
These technical details ultimately go back to the experience. SeePhys Hub headsets weigh no more than 220 g, single wrist rings no more than 75 g, and both types of devices support continuous 24-hour collection.
The design of the touch gloves is even more interesting. When it comes to contacting the data collection operations, there has been a very direct demand: don't let our people wear gloves.
The reason for this is realistic. Collectors may need to wash dishes, arrange items, or complete operations involving frequent friction. Touch gloves may affect hand activity, as well as water resistance, wear and data stability issues.
Therefore, SeePhys Hub will make the touch gloves and wrist rings a detachable structure. When real touch supervision is required, the gloves can be worn; in situations where gloves are not appropriate, only wrist equipment can be used.
But it also raises a question: where did the pressure come from without the touch gloves?
Another technological route to breakthrough was attempting to replenish full-touch with models. The team's previous EgoTouch established a corresponding relationship between first-person video, hand gestures and real stress distribution; TouchAnything further learned from visual estimates of contact positions and stress distribution.
Currently, the team is also advancing a purely visual bare-handed touch estimate, hoping that in the future it will be possible to supplement contact information for normal operating video without the use of touch gloves. This direction cannot replace all real touch measurements, but it offers an idea: first gain control through high-quality sensors, and then allow the model to recover some of the information from lighter inputs.
If future models can handle more gestural and tactile estimation work, the acquisition of equipment will have the opportunity to continue to simplify.
In addition to the hardware, SeePhys Hub also supports a self-developed data processing platform for analyzing, cleaning, cutting, semantic markup, and scene reconstruction, full-body gesture estimation, and hand movement tracking.
Motion Retargeting is responsible for mapping human movements to different robot bodies. Since the human body structure is not exactly the same as the robot, it still needs to be converted after harvesting to complete human body movements in order to further serve robot strategy training.
This also explains another layer of the apparatus industry's judgment that hardware structure itself isn't the hardest part to replicate, and that the real long-term investments are multi-modal fusion algorithms, gesture recovery capabilities, and the engineering process of converting human data into robot-training data.
This time in partnership with Tianjin Mobile, the company provides mainly product design and algorithmic capabilities, while Tianjin plays the supply chain, manufacturing and cost management advantages in the field of consumer electronics, promoting the commercialization and scale delivery of hardware products through joint ventures.
In the long run, the focus remains on data processing, algorithms and model development. Whether the devices can be deployed at scale is just the first step, and more importantly, these add to the body-wide movement and touch data that ultimately can bring benefits to robot training.
I've been teaching entrepreneurship for three, 98 years.
Data collection is just the beginning of a world model.
The team's long-term goal is to build a model of a world where humanoid robots can move their entire body flexibly.
He is a professor of computer science and technology at the University of Technology, Shenzhen. He is the youngest professor in the field of artificial intelligence in the country.
Since its inception, it has gradually formed a team with both model research and engineering experience. More than 90% of its core members have graduated from universities such as Tsinghua University, Beijing University, Hong Kong University, Harbin Industrial University, with a high academic talent density, research and engineering practices covering all models of large models, world models, large-scale data processing and hardware research and development.
The team believes that the current data collection, model architecture and application scenarios are not yet fully aligned, and that premature binding of specific nationalities may limit the direction of model development. By contrast, building data and model capabilities across nationalities is better suited to the current stage.
This route has already accumulated a number of studies. From collecting real touch from EgoTouch, to TouchAnything using visual estimation of touch, to TouchWorld introducing touch to world models, predicting physical states such as contact, force and posture, and understanding how touch will gradually shift from perceptual signals to models understanding physical interactions.
SeePhys Hub extends the scope of this research from the hands to the whole body. For the agile operation of moving the whole body, robots need to understand not only how the hands touch objects, but also how the body moves, how gestures are adjusted, and what the physical results of these movements are.
This is the next step in what we want to learn through a full-fledged model of the world.
According to the plan, the team expects to launch a full-model world model by 2027 and prioritize verification in vertical scenarios such as industrial, catering and retail.
Understanding model competition also has its own judgments. Robots need to perceive and execute in real time in real environments, and many scenarios require end-to-end deployments. How to make a finite-scale model truly understand physical interaction and complete tasks steadily is a more practical challenge than expanding parameters.
Of course, moving from more complete human data to stronger robotic capabilities will still need to address issues such as cross-body migration, tactile estimate reliability.
But the SeePhys Hub has given this technology route an engineered start.
Over the past few years, robotic data collection has continued to break free from the clutches of intrinsic and complex devices. On this basis, the company hopes to replenish body movement and touch, so that large-scale data collection can truly contain the information that robots need to understand the physical world.
Learning from real interactions, acting autonomously in the physical world. From tactile research to full-body model data, to future world models, this is exactly the path we want to take.
© THE