Over the past year, the data path of physical intelligence has shifted rapidly.
Remote processing of data is expensive, slow, difficult to scale, and more and more teams are starting to focus on human video. Teams such as Ingrid and EgoScale use the 20854 hour first-view human video pre-training robot strategy to fit a paired linear curve; HumanScale works even further, arguing that at the same amount of data, first-view video as pre-training data can be superior to real remote processing data.
Scaling behaviour with pre-trained human data, arXiv:2602.16710
The logic of this route is relatively smooth for claw-grabbing. However, the problem becomes more complicated when the executor is replaced by a hand with just over 20 degrees of freedom, because the video records how the hand moves, but not how much force is used. For tasks that require precise handling such as bottle caps, plugging connectors, cutting apples, success depends precisely on the movement of a few millimeters after contact occurs.
Sharpa Wave is a skilled hand.
The problem of data manipulation is essentially a problem of contact.
Sharpa, a robotics company focused on agility, has identified three main lines of exploration around this issue:
Software and hardware collaboration design will shift from robot-centric data collection to human-centric data collection.
Define "motion" with "interaction" and transfer contact to high-frequency, touch-based underlying controls.
Learn the general operational characteristics of inter-robot bodies using touch as a tool
Along these lines, Sharpa's hardware platform has also become an important vehicle for a number of cutting-edge research. For some time now, teams from Google, Inc. and others have been conducting a series of physically intelligent experiments and cutting-edge explorations based on Sharpa hardware.
At the same time, Sharpa is collaborating with teams from the University of Hong Kong, the University of California, Berkeley, Tsinghua University, and Shanghai Transportation University to advance data collection, model learning, and transnational generalization research in the areas of agility.
The heart of the machine recently interacted with Sharpa's academic director Zhang Kaifeng to try to figure out what questions the work was answering and how they were spelled out into a complete picture.
One apple, hard to live with VLA
Let's look at a specific task: cutting an apple with two clever hands.
In this task, one hand holds the apple and continues to rotate, the other hand controls the cutter, moving along the surface of the apple. The curvature of the apple changes, the thickness of the fruit skin changes, the strength of the grip and the angle of the cutting must be adjusted accordingly. The two hands do not perform asymmetrical movements, one hand is responsible for "steady" and "turn", the other hand is responsible for "cutting", and the other hand is responsible for rhythm.
Hand cut apple sequence chart in the MoDE-VLA paper
In Sharpa's MoDE-VLA paper in collaboration with institutions such as the Shanghai Transport University and the Shanghai AI Laboratory, the researchers assigned this task to π0 as a baseline for the mainstream VLA model.
Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA
Address of the paper:https://arxiv.org/pdf/2603.08122
This is not surprising. The paper points out in the introduction that VLA models are quite mature in using parallel claw-grabbing and placement tasks, but are clearly inadequate when faced with complex tasks that require extensive contact, hand-to-hand coordination and hand-to-hand operation.
To answer this question, we must first clarify the difference between hands and claws.
According to Zhang Kaifeng, the difference between the two is not just the degree of freedom. He said: "Traditional mechanical arms and claws simplify all the problems of operation into problems that can be solved in kinesiology. And the hands are dynamic. "The movement space of the claws is basically only open and consistent, and as long as it is moved to the right place and closed at the right time, the task is largely completed. The hands are called to use various underlying atomic skills (motion primitive), and Zhang Kaifeng estimates that human hands have nearly a dozen such skills.
This means that subtle operations cannot be performed solely by "looking at the position again". The size of the force, the state of contact, the movement of the object in the hand, all affect the next action in real time.
At the hardware level, high freedoms and tactile dexterity have gradually matured. Zhang Kaifeng believes that the biggest bottleneck after hardware lies in data, "especially data that is manipulated with such dexterity by humans".
Two data, one hard and two hard
Currently, there are two main sources of sophisticated data.
The first is remote operation (teleoperation): humans complete tasks by controlling robots remotely via visual devices or external skeletons, recording the robot's own movements. In recent years, academics have proposed a series of remote operating systems such as AnyTeleop, Open-TeleVision, Bunny-VisionPro, HOMIE, which are constantly improved in immersion and ease of use. These data interact with the robot's body and can be used directly for training.
Hand-cut Apple remote operation demo from the MoDE-VLA paper
But remote operation has a difficult problem to overcome: the operator does not feel the robot's touch. "It is difficult for the operator to get high security, the robot's sensation when operating", Zhang Kaifeng said, because the power and touch of the robot's fingertips are difficult to feedback to the operator in real time. In fine operation, this means that the operator can only be affected by the power, efficiency and data quality of visual guesswork.
The second is human-centered data collection: people directly complete tasks with their own hands, recording them by moving devices, data gloves or headphones. DexCap, DexUMI, DEXOP, OSMO are all in this direction. The human hands themselves have full touch feedback, are natural, and data is easier to scale.
The problem is that human and robot hands are not the same entity. Zhang Kaifeng divided the gap into two layers.
The first layer is visual: the images are human hands, not machine hands.
The second layer is on motion, and more fundamental. "Whether you're using the most advanced motion capture devices, motion capture fields, or any other means, what you can get is hand state, not motion", he explains, "what's wrong with motion and state? what's wrong with power and touch signals".
Thus, the researchers were faced with two difficulties: the closer the method of data collection to the natural movements of humans, the more comfortable the operation, the greater the physical difference between the robot and its body; the closer the method of data collection to the robot's structure, the higher the requirements for the operator, and the worse the experience.
Around the long-term goal of "Towards Human-Like Dexterous Manipulation", Sharpa's academic team has established three main lines for these two challenges:
Addressing the tangible differences between human-centered data and remote-operated data, answering "How data can be transformed into robotic capabilities".
Unlocking "motion" and "interaction" through simulation and reconstructing contact control with atomic skills;
Cross-bodied immutable representation learning responds to "how models migrate between different robots and sensors".
According to Zhang Kaifeng, the three main lines are "a parallel and mutually supportive state".
Main Line One: Breaking the "body gap" to make human data the fuel for robots.
To escape the dilemma of "the more convenient the collection, the more difficult the data", we must first answer: can we both preserve natural and efficient human-centered collection and replenish the missing power and touch information?
For this, it is necessary to collect non-essential data.
The so-called "nobody" refers to the fact that when data is collected, robots are not needed, and humans are directly operated by sensors. The key to this type of program is to preserve the nature and touch of human operation and to allow the collected data to be used by robots.
UVTA (Unified Visual-Tactile-Action Modeling/Unified Vision-Tactile-Action Modeling) is a representative work in this direction, completed by the Shanghai University of Transportation, Beijing General Institute of Artificial Intelligence, Sharpa and Beijing University of Technology. According to Zhang Kaifeng, the research team installed fingerprint sensing sensors on the Sharpa AE01 outer bone data gloves: 22 degrees of free passive linking rods to record finger movements, with each fingerprint's thermostatic array recording touch, and the wrist machine recording the first visual image.
The paper is titled: Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation.
Address of the paper:https://arxiv.org/pdf/2609.34182
The practice of UVTA is to combine human demonstration and a small amount of robotic remote control data into a joint training, mapping the two to a shared vision - touch - action space, allowing the model to simultaneously predict future movements and touch signals. Human data is always performed by robots as a training supervision and deployment. In Zhang Kaifeng's words, it is "to combine the relationship between vision and touch prediction, to test future state and touch signals, and finally grounding to the action signal".
UVTA Overview. VTPM converts the visual-touch input to a 20-dimensional pressure-resistant touch unit token, while Transformer extracts a 384-dimensional visual token. The two tokens are spelled as a unified 404 dimensional representation and passed to the scattered action head and touchback head.
UVTA has an average success rate of 70% on five high-contact tasks such as scrolling, light bulbs, dial switches, classifying small balls, and liquid transfer. The T-Rex as a control is 29%. More remarkable is a set of Scaling experiments: fixing 150 robot demonstrations per task, gradually increasing the human demonstration from 0 to 1000, and increasing the average success rate of three missions from 0% to 84%, which the paper says is still not observed in 1000 tasks.
Scaling curve of human demonstration numbers and success rates
Another work by DexEMG attempts to further reduce the capture barrier. It uses a surface electromagnetic signal in the forearm to infer the gesture of the hand, and then re-directs it in real time to the sharpa wave of 22 degrees of freedom, replacing the heavier outer bone and visual motion capture.
The paper is titled: DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization.
Address of the paper:https://arxiv.org/pdf/2603.05861
But at the same time, in the real world, there is another category of assets that is much larger: tens of thousands of hours of untouchable human video on the Internet, where there is no record of touch at all, how can man-made action bridge the gap?
In Sharpa's Dex-X program in collaboration with Xu Mondi's team at Tsinghua University, the researchers opened up another path: from a single RGB human video to accurately extract three-dimensional spatial motion trajectories of human hands and objects, and then import this trajectory into a physical simulation environment, where the simulation engine, through collision with kinematics, reverses the information about the kinematics of hands and objects that is invisible in real video, thus achieving effective migration from pure human video to machine touch strategies.
Title: Learning Visual-Tactile Dexterous Manipulation from Human Videos with Simulated Interaction
Address of the paper:https://arxiv.org/pdf/2609.07747
It can be seen that from UVTA's real-machine touch enhancement, to DexEMG's lowering of the collection threshold, to Dex-X's simulation mechanics completion, the mainstream is shifting the focus of data collection from "forcing humans to adapt to machines" to "making systems understand human touch".
But there is always no power in the video. The action can be redirected, but contact cannot be generated in space.
Main Line 2: Untangling "motion" with "interaction", reconstructing the common underpinnings of contact control
Even after receiving the data, the difficulty of the delicate operation is still concentrated in the moment of contact. Zhang Kaifeng said that the most difficult part of remote operation is that the sensation of the robot cannot be transmitted to the operator with high confidence when it comes into contact with objects.
Since contact is a bottleneck, a natural way of thinking is to separate movement and interaction: the movement part, which can be learned from human videos or even Internet videos; the interaction part, that is, the part that is rich in contact, is given to simulation and reinforcement learning.
This separation has its physical rationality. The wide range of movements before contact is closer to the problem of motion, and human video can provide a rich precedent; after contact, squeezing, sliding and force control are dynamic problems, highly sensitive to error, which is precisely the most missing information in video.
This idea is reflected in Sharpa's CraftNet sublayer model at CES 2026.
CraftNet three-tiered architectural layout
CraftNet divides robotic decision-making into three layers:
System 2 is a "reasoning brain" with a visual language model responsible for understanding instructions and dismantling tasks, running at a frequency of about 1 Hz.
System 1 is the "motion brain", responsible for movement planning and rough control before contact, about 10 Hz;
System 0 is the "interactive brain" that uses real-time tactile feedback to continuously adjust the fingers at about 100 Hz during the exposure phase.
After the demolition, new questions arose: what interface is the connection between movement and interaction, what information is the upper layer to convey to the lower layer, and how should the lower layer respond?
Sharpa and the team explored several different answers to this question.
The first interface is a pedal. This goes back to the original Apple. In the MoDE-VLA paper, the researchers first trained a set of in-hand operation skills, known as IMCopilot, with reinforcement learning, including maintaining a steady grip, allowing objects to rotate around a specified axis, etc. When collecting data, the operator uses the outer bone to control arm movements, and when it is necessary to turn the Apple, the foot on the pedal, and the rotation in the hand is given to IMCopilot to complete automatically.
System Overview: (a) Remote operating systems, including external skeletons, VR headsets, and pedals; (b) VR views, which integrate robotic camera images with real-time and touch feedback superimposed layers; (c) robotic platforms used to perform high-contact tasks.
The success rate of IMCopilot-assisted in-hand operation trials was 89% and direct remote operation was only 34%. High-quality cutting Apple data was thus collected. At the autonomous execution stage, the VLA model could also call on this underlying skill of IMCopilot, which was combined with a specialized processing power and tactile signal path, to form the MoDE-VLA. In the four tasks of cutting Apple test tubes, turbine assembly, charger plugging, the average success rate of the MoDE-VLA was increased from 15% to 34% from the baseline, and the success rate of cutting Apple was increased from 0% to 30% and the cutting skin was increased from 73% to 8%.[3]
Zhang Kaifeng believes that this is only the first step. "In the future, I will be able to drive the controller to perform more sophisticated human operations as long as I give some targets, some operational intentions", he says, "to hand over these contacts to the controller to the controller at the bottom, the top only needs to give some intentions and rough trajectory movements".
The second interface is the point-cloud. A typical example is the Dex-X, which runs a top-down route. It rebuilds the hand-to-object interaction process from a single human video and then allows the reinforcement learning strategy to track the reference action in the simulation. The missing contact information in the video is replenished by a physical engine in the simulation. The paper calls the simulation a "tactile completion engine". Finally, the strategy in the simulation is steamed into a student strategy with a point-cloud, intrinsic body perception and point-touch input, deployed to the real Sharpa Wave, with a 93% success rate in capturing real-machine blocks.
DEX-X's training framework. After migrating human representations to simulation environments, reinforcement learning is used to train an expert based on privilege states and combines demonstration references, object states, and touch contact information. Subsequently, the expert is distilled into a multitasking vision - touch strategy, running on a unified touchpoint cloud representation that embeds point-to-point touch feedback into scene geometry and integrates visual power and intuition to conduct strategic learning.
The same top-down thinking has also appeared in studies by other teams based on Sharpa hardware.