A series of technological reports published by the newly-established University of Applied Sciences and Technology have been released: 30,000 hours of tactile data have been used to replenish the senses of the body and the hands.
Here's a video of a real-life scene:
There's a lot of stuff in the video that robots can do: cook eggs, water these homemade dishes, and organize the dishes, it's handy; even factory flow line assembly, fine parts assembly, wire wrapping around grinding these fine things aren't mentioned.
I don't know if you've ever seen the phenomenon of robots actually driving over cars: the insert edges are hard to insert, the clutch of plastic bottles is uncontrollable, and the bottle gets stuck, and the overlapping towel scatters in the blink of an eye.
These failures point to an industry consensus: the key to determining whether robots can survive in the real physical world often lies not only in the accuracy of visual perception but also in the instantaneous touch feedback. The camera is able to determine the spatial position, but cannot detect whether the coil is in contact with the coil, reaching the edge of the coil, overloading the coil or slipping the coil, etc. The lack of this layer of feedback is the core pain point that prevents the robot from crossing the last centimeter of the coil.
Recently, Shanghai's New Intelligent Body and Smart Technology Limited (NeoteAI) jointly with the Institute of Credible Body and Smart Research of Ren Tan University, has officially released a series of three technical reports that jumped touch from the silicon-assisted model to the silicon core architecture.
What is even more industry-valuable is that the project data and models are open-source and full links to the project can be viewed directly:
-Foundation：
Unified Touch Dialogues builds 30,000 hours of interactive content
The large-scale application of touch data has long been subject to incompatibility between the data formats produced by various types of touch devices. It is difficult to integrate in the same model the conjugate graphs, capacitance matrices or six-dimensional vectors produced by different sensors, as in the absence of a uniform syntax.
To this end, the NeoData data set built by NeoteAI aims to break down this barrier:
Contains more than 30,000 hours of interactive video.
Approximately 1.4 million operating clips and 3.3 billion time steps;
8 billion tonnes of RGB images and 10 billion tonnes of tactile images;
Using six types of robot bodies, such as Franka, Piper, UR5e;
It covers 450 real-life long-range tasks (such as coating, plugging out connectors, pouring liquids, etc.) collected by 90 operators.
More than 5,000 hours of visual-touch interactive videos have been released.
In addition, the team proposed the NeoForce Unified Characterization Model. Regardless of the underlying sensor hardware, it is possible to learn the dynamic, time-structured touch characterization to use downstream intelligence strategies: where to press? with how much force?
For a more intuitive understanding, the following video provides a complete overview of the Foundation's core functions, including hardware, data sampling, comprehensive evaluation, and more.
-VTLA：
With lightweight VLA, it can be used to pre-determine vibration.
After the data base was confirmed, the team thought about how to really integrate touch into the path of VLA technology.
In existing multi-mode fusion practices, simply spelling out a tactile image directly with an RGB visual signal is often challenging: since the tactile signal only produces a high-frequency response at the moment of physical contact, most of the time it is in a state of still silence, it is extremely vulnerable to being overwhelmed by mass-visual tokens. More importantly, even if the model successfully extracts the current tactile features, it is still in essence a delayed feedback mechanism on the actions that have taken place.
- VTLA proposes a completely new scheme: it does not rely on lagging current touch but predicts tactile evolution within the next 50 steps. The model will encode the current touch as a compact potential token, combined with trends such as visual context-predetermining slippage, force, and so on, to guide the pre-adjustment of the action modules.
In addition, VTLA breaks the boundaries of traditional simulation learning that relies solely on expert success trajectories, allowing robots to autonomously evolve from failing data. The model combines tactile information with post-deployment data and human in the loop, which trains a progression assessment model to identify task execution phases and even complex behaviors such as step-by-step regression and rescue pads. By combining offline reinforcement learning, the success rate of robots on towel folding, book picking, carton folding, and so on, has significantly jumped from 50%, 35% and 20% to 95%, 80% and 75%, respectively.
The following video intuitively demonstrates VTLA's powerful ability to fine-tune operations.
-TWAM：
Re-constructed in the world model of touch and imagination
If VTLA is a pre-determination radar, TWAM is a reconstruction of the brain's cognitive centerpiece: it puts the touch into the world model, allowing the robot to fully simulate its operational perspective and senses in its brain's imagination before moving.
The current world model is largely confined to the production of future visual images and faces severe physical alignment challenges when driving a real robot: cups are caught in the picture, but fingers are not fixed; inserts are aligned in the picture, but the next second won't stick?
TWAM uses a hybrid expert architecture, built-in video, touch, action, three asymmetrical expert networks, video experts heat-start from pre-trained models, and touch and action experts use narrower structures with a final total parameter of 7.2 billion, which is only half the total width of the program.
In simulation and real-machine testing, TWAM demonstrated significant advantages: an average success rate of 84.5% compared to the world's strongest model baseline of 36%; an average success rate of 46.3% for real-machine 8 missions; a diluted experiment further confirmed that removing the future touch prediction valve or current touch condition valve would lead to a significant decrease in performance, establishing that touch is essential in the world model.
The following video shows the actual operating capabilities of the TWAM model.
Touch: the next Scaling Law of physical intelligence?
Throughout the three reports in the series, NeoteAI released clear industry signals: Foundation validates the scaling potential of touch modules with visual similarities; VTLA demonstrates that touch can naturally access existing VLA architectures; and TWAM establishes touch's central position in the design of the world's top models, making it truly aligned with visual modules.
This means that the cognitive paradigm of robots is evolving from a single pupil-driven pupil to a pupil-focused pupil. They no longer understand the world simply by looking at pupils, but they begin, like babies, to validate assumptions and plan their next action through active tactile exploration.
While there's still a long way to go, and the cost of collecting data in real-world scenarios, the ability to transnationalise and reason in real-time are all engineering challenges, the work of NeoteAI has essentially pushed touch from the research direction of Silicon Valley to the core architecture of artificial intelligence.
Perhaps the next chapter of being physically intelligent is not just about letting robots see the world around them, but actually letting robots touch the world around them.
The company's official website: