GPT-6 Astra's front-end general intelligence model has demonstrated excellent zero-sample capability in some operational tasks. But deployment at landing cannot only look at zero-sample performance, and model strategy reliability is more critical. Even the strongest general model in the industry, robots can easily repeat errors in interaction, alignment and execution of long process tasks.
The problem is that if every capacity upgrade requires artificial demonstration, the expertise to provide the demonstration will not keep up with the expansion demand when robots are deployed on a large scale.
On the other side of the ocean, Physical Intelligence is also looking into this issue. In a recent interview, co-founder Sergey Levine pointed out that robots initially need high-quality demonstrations from humans to build basic capabilities; once they are mature, they can test themselves in the real world and continue to learn from the data they collect.
In similar thinking, Qingdao University, together with Chiang Kai-shek, developed the Collective Experience Learning (CEL) model for group learning and completed prototyping verification of the model on 96 robots.
CEL paper cover
In the multi-stage, long-term industrial scenario represented by mobile phone packaging, the training strategy completed 120 tests on two robots of the same model that did not participate in the training, increasing the success rate from 79.2% to 98.3%.
It is also worth noting the benefits of scale: in the same period of physical training, the number of robots participating in learning increased from 16 to 96, and the test success rate increased from 86.7% to 98.3%.
Overview of the CEL paper
Technical report:
Link to the website:
Group learning experience: Robot companions can also be a source of experience
The existing path of robotic evolution relies heavily on human adoption. Whether it is simulation learning or reinforcement learning in humans, it requires human demonstration to complete tasks, or intervention when robots make mistakes. This requires not only fluid action by the operator, but also familiarity with the adoption timing, balancing the proportion of self-exploration and human demonstration data. However, as the number of trained robots increases, experienced human experts find it difficult to expand synchronously.
CEL has changed the source of action oversight in follow-up learning.
Assuming one robot encounters difficulties when placing a mobile phone, while another robot successfully completes operations under similar circumstances, the experience of the latter can help share strategies to make up for the shortcomings of the former, so that one robot's practice benefits the entire group.
In this paradigm, the role of human beings changes with it: from constantly providing action, assisting in exploration, and turning to determining results.
Differences between CEL and previous learning paradigms
To turn CEL's thinking into a viable learning process, two intersections are needed.
First, how do you find the experience that is really worth learning from a lot of execution experiences with only 0/1 feedback?
Secondly, how do you make scalable robots run online and accumulate experience steadily in the face of communication constraints and local failures?
What is it about robots, with only scarce feedback?
The duration of a task is 3 minutes or more, over 4,000 steps, but the final feedback of the task is only one bit: success is marked with 1, failure is marked with 0. Such feedback can distinguish whether the task is completed, but cannot directly indicate: where the strategy is inadequate and what successful experiences are worth studying. For this reason, the team proposed the failure-guided experiment detection micro-modulation algorithm FERT. It distinguishes between the role of success and failure samples: the success trajectory provides learnable actions, while the failure scenario indicates the focus of learning.
FERT algorithm framework chart
In the framework of FERT, initial strategies deploy asynchronous execution tasks on various robots. Successful trajectories are entered into shared experience pools as learning materials for subsequent strategy tweaks.
The role of the failed trajectory is to locate areas where the model needs to focus on learning. The algorithm intercepts a sequence of observations before the end of the failed task, co-encodes the camera image with the robot's associated state, and uses this characteristic to retrieve similar operating segments in the successful experience pool.
FERT will synchronize success trajectories with failure trajectories. Success sequences matched to current failure scenarios will be given higher sampling priorities, guiding training focuses on these error-prone links.
Under the updated completed strategy, robots are sent back, continuously generating new successes and failures. The value of the group experience comes from continuous data accumulation, on the one hand, and on the other hand, depending on whether the adapted experience can be accurately delivered to where the model needs to learn the most.
96 robots train at the same time, and the system must keep up.
In order for 96 robots to continue accumulating experience, they face the following real-world problems: some execute fast, some restart slowly; some have network fluctuations, some need to be charged, and a few robots may also have hardware failures.
If local failures were to reboot the entire training, large-scale collaborative training would never be mentioned.
Faced with this problem, the team designed the fault-tolerant asynchronous training system FATS, consistently dynamically organizing 100 robots, 50 edge nodes, and 6 cloud servers to support large-scale real-world online learning.
(1) SwarmLink: Making data flow efficient
SwarmLink is responsible for connecting environments, reasoning and training nodes. Each robot submits data asynchronously after completing a round of tasks, with the training side able to update data in real time based on arrivals of experience data, without waiting for all robot tasks to be completed. The system supports flexible editing of cloud resources, training, reasoning, and other components that can be deployed on demand in the cloud or on the end.
SwarmLink design
(2) HotSwarm: Highly faulty heat pump system
Control the dynamic inputs and outputs of the HotSwarm management node and workflow. When the robot is suspended, offline or re-joined, the system retains the registration information and resource allocation of the node, updates the workflow, and repairs affected connections.
In a 96-robot-scale system experiment, the cluster accumulated 9834 rounds of execution experience; a total of 22 robots suffered 30 failures during that time, and the loss of robot time caused by failures accounted for only 1.3% of the total operating time of the machine. Even if the hardware fails, the cluster's experience accumulation and training learning process continues sustainably and steadily.
HotSwarm design
How much improvement can the group experience bring to the five types of real tasks?
Demonstration of 5 tasks in experimental testing and accelerated processing of GIFs
The team covers five tasks across the household and industrial scenes, including plowing, folding clothes, charging cell phones, assembling hoods and packing cell phones, involving flexible objects, precise operation, tool use and long process execution.
Five successful test missions
On five real-world missions, two untrained robots, the team completed a total of 600 tests, cumulatively testing nearly 30 robot hours, with only 15 missions failing, with an overall success rate of 97.5%, a failure rate of only 2.5%.
Traditional off-line modulation (SFT) has an average success rate of only 36.3% in the above missions; the HG-DAgger multi-machine scheme based on the human demonstration, the success rate can be increased to 81.0%; on this basis, the FERT algorithm we adopted under the CEL paradigm further breaks the upper limit, with the final model performing above the average of 6 human remote controllers on all missions.
Scaling trends for training robots online are starting to show
If the group itself is able to provide learning experiences, a natural question is: Will the increase in robots continue to improve strategies?
True machine scaling curve
The team gradually expanded the size of the clusters of robots involved in the training. During the same physical time, the FERT algorithm showed a significant scaling trend on all five tasks: the more robots there were, the higher the success rate of the task. In contrast, the HG-DAgger program, which relies on human demonstration, exhibits only limited scalability and human demonstration collection efficiency becomes a bottleneck for its further expansion, breaking the maximum success rate.
In the most difficult mobile packaging tasks, robots involved in the FERT training expanded from 16 to 96, with the test success rate rising from 86.7% to 98.3%. Within the same real-world interaction time, larger groups of robots directly transitioned to higher task success rates.
This provides a scalable direction for physical intelligence that deserves further exploration: in addition to the accumulation of human sample data, the expansion of a group of robots with the ability to explore independently, making physical interaction in the real world a central source of ongoing learning.
After deployment, robots can grow together
From a successful demonstration to repetitive, reliable execution, robots need to continually improve their capabilities in real interaction. CEL's path to exploration is to allow robots that already have basic capabilities to contribute autonomously to the experience of converting individual practices into collective replicability through shared strategies.
In the process, deployment on the ground becomes a new source of experience. Successes and failures in the execution of tasks by robots can enter the next learning cycle.
96 Machine experimentation connects experience acquisition, experience utilization, and continuous operation to demonstrate the learning benefits of expanding groups in real robots. When each robot contributes experience, the number of robots has the opportunity to become a new dimension of capacity growth. Let each robot practice become the starting point for more robots to do better the next time.
© THE