Robots Finally Gain Task-Progress Memory: Four Hours of Continuous Work at Over 95% Success

Yu Wen, a second-year student at Shanghai Transport University and Shanghai Institute of Creative Thought, and a mentor at Luteryu, Wenchuan, whose main research focuses on real machine reinforcement learning and human after-cycle training systems, has been the first author or co-author of publications such as RSS, CoRL, RAL and other robotic conferences.

When the robot folds the clothing, the claw moves near the corner of the garment, which does not mean that it has already grabbed the clothing.

For long-range operations, robots need to judge not only what's in front of them, but also whether the action has worked, whether the task is still progressing, and whether the next step should be to continue to execute or adjust the recovery.

That's what Value as Memory (VaM) is all about. We use the recent predictions of value models as a memory buffer for task progression, providing action strategies to help robots select actions using information about task progress, stagnation, and regression. At the same time, VaM enables robots to continually improve their strategies by leveraging new experiences in real deployments through reinforced learning processes from offline to online.

A mirror finally records the entire process of the robot continuously completing tasks autonomously, including long-range flexible object (deformable object) operation and contact-rich (contact-rich) tasks. In paper box assembly tasks, VaM continuously runs autonomously for 4 hours, with a mission success rate of more than 95%, demonstrating stable performance in long-range operation.

Why do robots need memory pumps to progress missions?

While the general robotics strategy has been able to learn multiple operational skills from demonstrations, deploying these skills steadily in the real world still faces two kinds of difficulties.

The first type of difficulty comes from partial observability. The camera can only see from a limited angle, and the history of observation used in strategy is also limited in length. Therefore, even if the input image is very similar, the actual state of the robot may be different, and the corresponding rational movements may change with it.

For example, when folding clothing, robots may need to re-open their claws after losing contact, adjust their position, or they may need to maintain tightness after retrieving clothing.

The second type of difficulty comes from the lack of timely deployment of experience feedback. When robots execute tasks, they encounter situations that cannot be covered by sample data. If data collection and training are done in a sequential manner, new deployment experiences need to wait for the next round of training to affect the strategy, causing a slowdown in iteration. On the other hand, if training relies mainly on the latest on-policy data collected, already existing sample and historical correction experiences may not be fully exploited, reducing sample efficiency.

VaM revolves around two issues: supplementing recent progress information for action strategies and establishing an online learning loop that repeatedly leverages new and old experiences.

Value Caching is a memory that can be used to turn value predictions into action strategies.

In reinforcement learning, the value function estimates future returns. In VaM's mission setting, this prediction also provides tight signals related to mission progress.

In the past, value prediction was commonly used to evaluate movements, construct training targets, or provide optimization weighting. VaM further incorporated recent value prediction sequences as a strategy input and participated in motion generation in conjunction with visual history and robot entity status. We call this mechanism value caching (value caching).

Individual forecast values provide estimates of current progress, while continuous forecasts can reflect changing trends: whether the task is advancing, stagnating, or reversing. This information helps strategies distinguish visually similar but requiring different actions.

VaM uses near-term value prediction as a strategy for input action into task progression memory and continuously updates strategies and value models through online reinforcement learning.

In implementation, VaM builds motion experts based on π0.5 and configures value models that predict return distributions. The motion experts use gate control loop units (GRUs) to encode historical values and physical states, and then combine visual information-generating motion.

In order to avoid predicting the value of the current action waiting for the current moment, value prediction runs asynchronously with action generation. The action strategy reads the historical value that has already been calculated, so caching can be used directly when generating the action.

From human demonstration to autonomous experience

Effective progression memory requires reliable value models, and value models also need to be exposed to the state encountered when robots actually execute. For this, VaM will train organizations in four stages.

Phase one: Offline demonstration initialization.

Using expert examples of initialized action experts and value experts, the robot gains basic operational capabilities and establishes initial task progress estimates.

Phase two: Offline reinforcement preheating.

Strategy execution trajectory and human intervention data are collected to allow value models to learn real execution stagnation, failure and recovery. Then, using updated value models to calculate advantage labels and train action strategies.

The third phase: Online DAgger.

Robots perform tasks under conditions that allow human correction, continuously collecting new states that current strategies will encounter. Training simultaneously utilizes offline demonstration, historical intervention experience with current turnaround trajectories, updating action experts and value experts. The entire training process and data collection are perfectly parallel, forming an online real machine RL framework.

The fourth stage: Learn from Experience (LfE).

Robots continue to perform autonomously, rewarding learning through scarce task results, and no longer recruiting new human corrections. Demonstration and intervention data previously accumulated are still involved in training.

The four training phases connect offline demonstration with autonomous experience learning. Value experts simultaneously provide information on the advantages of strategic training and provide recent progress memories for action generation.

True machine RL mission verification

We selected three long process operation tasks, which included deformable object handling and contact-intensive assembly tasks: paper box assembly, laundry folding, and magnetic bus assembly.

How much has value caching brought about?

The paper compares the complete VaM, the de-melting variable that removes value caches, the de-melting variable that removes off-line reinforcement preheating, and the online RL version of RECAP.

VaM is contrasted with the online reinforcement learning success rate curve on three missions, showing the training process from human-corrected Online DAgger to autonomous experiential learning of LfE.

According to statistics from the last three online reinforcement learning checkpoints, the final task success rate increased by 29.4% compared to VaM's version of removing value caches.

Can value caching mitigate some of the observability issues?

For long-range tasks, an equally important question besides the ultimate success rate is: can the robot continue to complete the current phase by correcting the action after the problem is encountered?

The paper therefore blindly evaluates the execution trajectory of the strategic checkpoints at each LfE stage. We define recovery rate as: the proportion of successful completion through corrective actions in all task stages that need to be restored.

The VaM recovery rate for the three missions was 54.5%, 86.4% and 33.3% respectively. After removing the value cache, the corresponding results were 50.0%, 58.8% and 7.4% respectively.

Qualitative cases also show a consistent phenomenon: Value forecasting and corresponding strategy representations of VaM are more likely to reflect the stagnation of the task and accompany subsequent corrective actions in the demonstrated mission execution trajectory.

In Box Assembly, the qualitative contrast shows the camera images, the main components of the value experts are analyzing projections (PCA) and value predictions.

Effective value caching also requires value estimation in line with current strategies

Does the effect of value caching come from simply adding a set of inputs?

To further test the importance of cache content, our fixed-trained action experts only replace value experts with earlier versions of the demonstration initialization phase or offline preheating phase. The cache mechanism remains unchanged, but the success rate for all three tasks has dropped significantly.

Fixed-motion experts compare success rates using different value sources.

In further analysis of clothing folding, we fixed motion experts and all observed inputs, replacing only the value history. After loss of contact, the original value historical corresponding motion prediction tends to open and adjust the right clavicle to prepare for retrieval. After retrieval of the clothing, the original value history makes the prediction to keep the clavicle closed, while earlier price models generate open instructions at the end of the action sequence.

After losing contact, the original value cache guiding strategy predicts the opening and adjustment of the right clavicle to prepare for retrieval.

After retrieving the jacket, the original value caching correspondent predicts that the right clavicle will remain closed, while the earlier value model predicts that the clavicle will open at the end.

Online RL significantly improves sample efficiency

We also compared online and offline reinforcement learning. Both were based on the same offline preheating model and matched gradient updates and data collection budgets. At the end of the LfE phase, the average success rate of online RL programs on the three tasks increased by 32.8%. This reflects the value of using on-policy data to form a fast learning loop.

Online reinforcement achieves a higher LfE phase success rate in all three missions when the number of training steps is matched with budgeting.

Simultaneous execution also improves the time-efficiency of experimental processes. Compared to offline processes where data is collected, value experts are trained, and action experts are trained, online programs run all three simultaneously, reducing the real-world time cost of the three tasks by 57.8% on average for each step of the ladder.

Summary

VaM demonstrates one use of the value model: to use recent value predictions as a task progression memory, directly participating in action generation. Combined with offline to online real-world RL training processes, robots can both leverage historical experience and sustain current strategies in real-world situations.