“ SimpleMemVLA: In the original visual context as a memory, allowing robots to remember the past.
What can a robot do without adding dedicated memory modules and directly using the visual history of time as a context?
SimpleMemVLA's answer is: Within the scope of paper comparisons, SOTA is obtained on all four memory benchmarks, and without updating the parameters, it demonstrates the ability to visually learn lower literature. It is also a supporting technology for context memory and flow memory in MiniCPM-RobotManip.
The core of this path is straightforward: allowing the robot to combine what has happened in the past and understand what to do now. Where the object is hidden, how many times an action has been done, what steps the task has taken can all be drawn from the original visual history.
Why now? According to the paper, 60 seconds of history is only about 5.6k tokens, while the mainstream model supports 262k tokens.
The work was carried out jointly by China University of Technology, China People's University, Qinghua University, Beijing University, Beijing Institute of Artificial Intelligence and Guangzhou Chinese Academy of Technology in Beijing.
The paper:
Open source code warehouse:https://github.com/OpenBMB/SimpleMemVLA
Open source model warehouse:
01.
In the context of native visuals, robots can remember more accurately in long missions
From shading, counting to long-range interaction, SimpleMemVLA scored the best overall score on four memory benchmarks while maintaining high performance in two general control ratings.
These task tests include not only the ability of the robot to see the current image, but also its ability to remember clues that have disappeared, distinguish recurring events, and continue to track the progress of the task.
The four memory benchmarks are compared with two general control assessments. Each benchmark is compared separately; the gray bars are the reference results and do not participate in the rankings.
In long-range interactions that require continuous tracking of task progress, this memory ability has brought significant benefits.
RoboMemArena consists of 26 missions with an average trajectory of more than 1,000 control steps. Using the 126-second history window, the SimpleMemVLA's full-stage mission success rate was 63.6%, up 17.4 percentage points from the previous strongest model. Of these, counting missions increased from 31.4% to 71.4%, and shadowing missions increased from 39.1% to 64.3%.
The advantages come not only from the mainstream model, but also from the model's ability to directly understand the history of native vision when making decisions.
Whether a detail is important is often not known until subsequent tasks. Pre-screening, compressing or covering history may lose information needed in the future. The paper calls this limitation a promise when writing a note: when storing memory, it is necessary to decide what to keep.
SimpleMemVLA preserves historical visual information, time sequences, and timelines in the context above, so that the model understands the current task and understands the past, rather than processing history as a fixed summary and then acting on it.
Differences between the context and the dedicated memory mechanism in real-life vision.
Specifically, historical images are organized in chronological order along the video format of pre-training visual language models, and are attached to a timestamp; the current wrist image is entered separately. The model thus generates a current subtask, such as unlocking the cap on the red blocks.
The hidden state of the context that is formed when the sentence is generated is transmitted to the action head, along with the word embedding, the robot's state of being, to generate the next movement. Meltdown experiments show that historical information influences behavior primarily through this hidden state, rather than just through the literal text of the submission.
SimpleMemVLA architecture and streamlined reasoning processes.
In order to distinguish between mainstream capabilities and the effects of memory, researchers fixed training data, mainstream models, subtask monitoring, motion heads and optimizers, replacing only memory mechanisms. In RoboMME, the results are as follows:
In the original context 88.3% | Retrieved from 31.5% | Token compression 22.6% | Circular state 20.6% |
Success rates at task levels for different memory mechanisms with the same model and training setup
This group of results supports a critical judgment: that in the task being tested, the model combined with the current task to understand the still-readable history is more effective than deciding in advance which information should be retained.
Further dilution illustrations, key images and time sequences are indeed indispensable. In RMBench's block-covering task, the 30-second window still contains the coverage process of about 25 seconds before the model can complete the task; the window shrinks to 15 seconds; the key image is removed and the task fails. The key counting task also gradually deteriorates as the history window shrinks. Both tasks fail only when the current image is retained or the order is disturbed.
Context-based fusion experiments. (a) (b) Check the history window, sequence and timeline; (c) (d) Check the hidden state of the subtasks to see how the historical information is conveyed to the action head.
02.
Visual literacy lessons: not updating parameters, following visual context adjustment strategies
SimpleMemVLA can not only leverage past events, but also adapt current behavior to the context based on newly provided visuals.
The researchers accessed the original history of another segment of the execution trajectory: changing the location of the blocks covered, the model moved to a new location; changing the color cue from red to red, the model selected the red object; replacing the initial movement of the demonstration sequence, the robot would change the next step as well.
The history of these spellings did not appear in the training, nor were the parameters updated when reasoning. The model can still judge the state of the task from the context of the new vision and act accordingly. This is the ability to learn literature visually as observed in the paper.
How to hide or replace historical fragments and test models based on past actions
Changes in behavior in models are also targeted: changing critical history affects decision-making, while changing unrelated history does not.
Covering the blocks in the position of the screen would cause the model to choose the wrong cover; covering an already completed grabbing and placement event, the model would calculate less and add execution. The output remains unchanged when unrelated segments of equal length are covered.
These experiments showed that in the test tasks, the native context can both carry memory and allow new visual information to directly affect model behavior.
03.
Streamlined reasoning optimizes minute-level memory with real-time control
Long history doesn't necessarily mean a long wait. By recycling history calculations, SimpleMemVLA decreases the decision delay of 60 seconds from 1.02 seconds to 0.68 seconds, lower than the real-time budget of 0.96 seconds for testing tasks.
This result comes from a single H100, bf16, batch size set to 1. The idea is also straightforward: two consecutive decisions share most of the history, the robot performs the current action, the system calculates in advance the historical forecast that will still be used the next time, and saves key values in cache. To the next decision, it processes new content and generates subtasks.
This set of precise streamlined reasoning processes generates the same subtasking text as a full-quantity recalculation, but overlaps the shared history of computation with action execution, reducing the wait time for decision making.
Overlapping the computation and execution of shared history, shortening the waiting time for the next decision.
In a longer context delay test, about 45 minutes of history corresponds to 245k tokens, and the total weighting takes 32.1 seconds, while the flow decision path is shortened to 1.18 seconds. This shows the benefits of computational replication; in a longer context, backlog pre-filing and caching expenses will continue to grow with history.
04.
The real show
In three separate layout autonomous executions, SimpleMemVLA can uncover the cover in the order of red → green → blue, based on the visual history of the shading.
The task is done on a real two-armed robot: the robot first covers the three colored blocks with the same black cover and then uncovers them in the specified color order. By this time the current image can no longer see the color of the blocks, and the robot must judge what is under each cover based on previous experience.
Layout one: Colored cover
Layout 2: Colored cover
Layout 3: Colored cover
In the third execution, the uncover path changes with the colour position. The third section uses the sequence in parentheses → far → near parentheses, indicating that the robot does not always scan along a fixed direction, but instead uses the historical colour-location relationship to complete the task.
05.
Future prospects: Moving from memory to learning from experience
SimpleMemVLA incorporates context as a memory and, in the process, develops the ability to learn literature visually.
Next, we will introduce more rich contexts, including human first point of view (ego) data, robot demonstration video and autonomous execution history, make full use of the upstream literacy learning capabilities of the model, explore how robots can learn from examples and past experiences, and apply those experiences to new tasks and scenarios.
Making context not only help robots remember the past, but also an important source for learning and adapting to it is a direction we want to continue to explore.