The COCO Matrix releases the robot model CO01 (read /kuːl/) so that robots can learn from these experiences.
A demonstration, a failed experience, a reminder, or even a target image can help CO01 decide the next step. The team calls this a "experience cue": experience can be put directly into the context of the model without having to be retrained for the current task.
This makes new tasks less dependent on specific sample collection and fine-tuning, reducing the cost of adaptation and deployment. Users can also teach robots how to do things through demonstration and guidance. Failures, retests and human corrections can both be used for reference by robots at the moment, and can be organized and used for further training, so that the experience accumulated in using them continuously helps the model improve.
Let's hope CO01 becomes a very COOL robot model! realise the vision of a co-matrix: Robots for all, shaped by all!
Here's what you need to know:
New tasks are faster: with the same pre-training model, one demonstration, about 1 minute, the average success rate of unseen tasks reaches 52%; moderation to a similar level takes about 180 demonstrations, about 9 hours.
Failure results in a quantifiable improvement: with the same five attempts, the cumulative success rate of preserving its own history is 84%, not preserving history is 56%.
Strong technical support: Conversational training maintains the link between demonstration, failure and correction, allowing the action model to use experience-adjusted behavior without updating weight. The interaction records accumulated in deployments can also be used for follow-up training after being organized, so that real use continues to drive the data flywheel.
Unified perceptual capabilities: The same set of weights supports multiple perceptual tasks such as depth, division, light flow, human posture, five of the eight benchmarks are better than dedicated models, depth estimates are basically constant and provide a perceptual basis for robots to read spatial relationships and movement changes in the samples.
Experience is the Prompt: Introducing CO01
Address of the blog:
Directory
1. From In-Context Learning to Experience Instruction
2. Learning from different experiences
2.1 Example 2.2 Self-Trying 2.3 Instant Guidance 2.4 Cross-body skills
3. How experience becomes the basis for action
3.1 System 1 and System 2-3.2 Conversational Training and 3.3 Unified Visual Basic
4. From instant adaptation to continuous learning
1. From In-Context Learning to Experience Instruction
In the past, teaching robots new tasks often required remotely human manipulation, gathering large quantities of samples and fine-tuning models.
GPT-3 demonstrates this ability in language tasks, and the field of robotics has begun to explore learning from demonstration. CO01 further allows demonstration, experimentation, and instant guidance to involve multiple experiences in action decisions.
The real-world mission is endless. CO01 explores how new missions can be taught on the ground, how to execute biases in a timely manner, how to reuse historical experience to address the challenges of generalization, reduce deployment costs and promote cross-scenario replication.
Figure 1: From task instructions, single demonstrations, to multi-source demonstrations, previous attempts, instant guidance and target images, CO01 expands the context in which the strategy model can be referenced.
2. Learning from different experiences
2.1 Learning from Examples: Adapting to New Tasks and Scenes
CO01 can understand the goals and steps from a human first point of view, from a side video, or from a demonstration of other robots, and find a method that suits its own body. When inserted, it follows the changing steps in the demonstration, while the grappling method and movement path adapt to the claw.
Plug-in: The lower left-hand corner is a human-sided demonstration, the main screen is automatically executed by a robot, and is twice as fast.
After viewing the CO01, the demolition was completed in the order in which it was installed.
In the lower left corner, the construction process is played in the original order; the robot receives the download and release version and automatically performs the demolition.
Examples can also combine existing movements. Exercising only one folding model, the person folds the towel twice and then completes two foldings in succession; the person points to the button, which is then pressed in order.
Continuous folding: The lower left-hand corner is a human demonstration, the main screen is a robot's autonomous execution, at twice the speed.
After pointing to the button, the robot automatically executes at 4 times the speed.
The team kept the scene and start state unchanged, only changing the way the towel folds or the order of the buttons.
The left is the opposite corner bend, the right is the opposite side bend; the lower left is the corresponding demonstration, equal to 2 times the speed.
In the upper right-hand corner, the demonstration sequence is shown, with each side of the robot executing 1.5 times the speed according to their respective demonstrations.
The results showed that 96% of button operations followed the demonstration sequence and 92% of towel operations followed the demonstration folding method.
The team further altered the layout of the scene, objects, tools, and tasks, comparing the effects of "give a sample" to "only give language instructions", and tested each condition 100 times.
When switching to similarly functional alternative tools, a demonstration increases the success rate from 14% to 52%; from 6% to 43% in the face of skills not seen in training.
Figure 2: Left from the original scene, followed by a test of new layouts, new object instances and functional alternatives (S0S3); right, a test of known tasks, new skill combinations and unseen skills (N0N2).
In another group of unseen missions, the team compared the ICL to a fine tuning using the same pre-training model, with preparation time including sample collection and training.
ICL takes a demonstration session, about 1 minute, to achieve an average success rate of 52%; mitigation to a similar level takes about 180 demonstrations, about 9 hours, and preparation time is about 460 times different. This means that new tasks can first be rapidly verified by a live demonstration and then trained as needed.
Figure 3: Preparation time and success rate for unseen missions. Time includes sample collection and training, the shadow is 95% confidence interval; about 460 times as long as the unseen time is calculated.
2.2 Learning from Self-Trying: Using Failure to Improve Your Next Action
Whether the test tube can be inserted into the hole is only known by moving hands. The robots that kept the previous attempts tried on the left and right sides, the third time putting the test tube into the blue shelf; the other side of the history cannot be seen, two attempts on the same shelf, and the test tube collapsed.
Left to retain previous attempts, right to not retain history. Both sides perform autonomously, at twice the speed.
The same difference occurred in the presentation of the mobile phone box: the robot that preserved the history covered the box for the fourth time; without the history side, the box cover was twisted once again.
Left is to retain previous attempts, right is not to retain history. Both sides perform autonomously, four times faster.
True Machine Assessment selects the most frequently failed first attempts, with a maximum of five attempts, with the tasks, scenarios, and instructions remaining the same. While retaining its own history, the cumulative success rate increased from 33% on the first to 84% on the first five attempts; the same five opportunities, without retaining history, were 56%, and the failure record for other tasks was 58%.
Figure 4: (a) The same model belt, without historical attempts to train, loses movement on the record of multiple attempts not participating in training; with historical losses about a quarter lower and reaches baseline optimum levels earlier. (b) The real machine has a cumulative success rate of 100 attempts per point, with a 95% confidence interval.
2.3 Learning from Instant Guidance: Adapting current actions to new requirements
After the robot moves, it is still able to receive the instructions of the person. The robot is sending the cup to the cup shelf, the person puts the cup in the middle, and it changes the position of the cup.
The robot automatically executes the instructions of the intermediate receiver.
A single point can also serve as the basis for subsequent operations. The man first uses gestures to specify the pair of cups and cups, then disrupts the position of the cups; without new hints, the robot is still positioned according to the relationship that has just been taught.
Left to gesture specified pairing, right to disrupt cup rolling, self-executing, all four times faster.
The instructions in execution are retained in the context of System 1 and continue to be used as a reference for subsequent actions; the preferences for cross-tasks are saved by System 2 and re-provided in subsequent tasks.
2.4 Learning from other robots: Cross-body skills in simulation
In the simulation, the two-armed robots first sweep the crumbs into the pot, the wheeled robots use this video to demonstrate the task, and the humanoid robot learns from the wheeled robot. The goals and steps can be followed, and the three robots use different tools.
Skills in simulation: The last two robots followed one or more videos as demonstration, autonomous execution, real speed.
3. How experience becomes the basis for action
3.1 System 1 and System 2: From Experience to Action
System 2 is responsible for planning, remembering, calling tools and checking progress, selecting samples, previous attempts or target images for the current subtasks, and submitting failure records to System 1 for the next round of reference.
System 1 generates movements. It combines reference experiences, real-time images and robot status to make decisions, and continues to receive instructions in execution. When trained, it simultaneously predicts subsequent images and movements; when deployed, it only output movements.
Figure 5: System 2 Experience required for sub-tasks organisers, System 1 combines real-time observation to generate movements. It transmits both task instructions and demonstration and interaction records between the two.
The technical focus of CO01 is to allow the action model to directly exploit demonstration and interaction records, preserving the difficult to translate operational details in language.
3.2 Conversational Training: The Causes and Consequences of Retaining Success
CO01 maintains demonstrations, failures, retests, and guidance in the same task in a timely manner, using previous experience to predict the next move. Single execution records are also involved in training to retain skills.
Meetings come from specially collected and pre-recorded videos. Records from different sources are coordinated by task and step, labelled with visual language model assistance, and audited manually; simulation and video production further extend physical conditions, scenes and perspectives.
In training, relevant examples are put together with previous attempts, sometimes mixed with unrelated experiences, so that the model learns to distinguish what is relevant.
Figure 6: Targeting a third wave heat stroke with a microwave oven, prior to earlier attempts, and similar demonstrations provided by humans, other robots, and video models; sometimes adding unrelated tasks.
3.3 Unified Vision: Understanding Space and Movement
The team has collected about 50,000 hours of data from the first-person video, multiple robots and hands-on claw recording, simulation and screened web video pre-training.
In addition to predicting subsequent images, the same mastermind also learns perceptual tasks such as depth, surface lines, splitting, light flow, and human body, hand, and camera movements to make information related to operations easier for models to use.
Visual mainstream perception capabilities: the depth of the real-machine image, surface lines, light flow and division, and arm bones, human skeletons, language positioning and spatial positioning. The first five frames are from the same real-machine video, with the remaining frames as separate cases; the hands and positioning use P10000 EMA selection results, presented separately from the benchmark ratings below.
Figure 7: Comparison of the same set of weights and special models, five of the eight perception benchmarks lead, depth is basically flat, and two are lagging behind. The initial error in camera position and light flow is about 19% and 18% respectively; the ratio of convergence scores in the figure is not equal to the change in the original error.
The team also assessed the role of data size and task diversity, respectively: ICL and mitigation performance continued to improve as pre-training data increased; when ICL training covered 3,543 tasks, the single-sample success rate on unseen tasks reached 53% and only 18% given language instruction.
Figure 8: (a) Increase pre-training data by comparing 800 demonstration thresholds and lessons on a single demonstration in a new task set; 87% and 53% for 50,000 hours of success respectively. (b) Increase the types of training tasks in the context, comparing single demonstration with language-only instruction.
4. From instant adaptation to continuous learning
At the next stage, the team focuses on reducing the accumulation of errors in long missions and improving the accuracy of the CO01 selection experience.
The team also plans to have CO01 actively teach lessons: take timely demonstration requests, schedule exercises when they fail repeatedly, and look for experience in the records of other robots.
Currently, new skills are adapted mainly through up-and-coming literacy learning, with cross-tasking preferences preserved by System 2. COCO's long-term goal is to continue to evolve by implementing experience-driven models, allowing robots to share skills, accelerate learning and adaptation, and ultimately serve hundreds of industries and reach thousands of families.
Cocoa Matrix is continuously looking for partners to further optimize CO01, and welcomes researchers and engineers interested in the direction of In-Context Learning, Continuous Learning, Multimodal Express Learning, and real-world experimentation.
Reference links
1. Brown, T. et al. Language Models are Few-Shot Learners. NeurIPS 2020.
2. Generalist AI. GEN-1.5: Embodied Foundation Models are One-Shot Learners. 2026.
3. Skild AI. Introducing S1: In-Context Learning for Robotics. 2026.
4. Duan, Y. et al. One-Shot Imitation Learning. NeurIPS 2017.
5. Polanyi, M. The Tacit Dimension. University of Chicago Press, 1966.
6. Chan, S. C. Y. et al. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. NeurIPS 2022.
7. Raventós, A. et al. Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression. NeurIPS 2023.
8. Wang, L. et al. Video Generation Models are General-Purpose Vision Learners. 2026.
9. Silver, D. and Sutton, R. S. Welcome to the Era of Experience. 2025. PDF
© THE