Physical-intelligent long-term memory: seeing is not the same as remembering

  It's the first time in the history of the world.

Imagine a day-to-day scene:

The home robot had just seen a book in the tea room, and after completing several tasks in the kitchen, it was instructed to pick up the book, but started searching the room by room. It had already tried an unopened drawer for a while, or tried again before returning to the same drawer.

For people, "remembering where things were put", "knowing that they were taken away", "remembering which ways didn't work" are basic skills in everyday life.

However, connecting past observations, actions, and feedback to be used in the next mission is still a challenge for physically intelligent individuals. There are already benchmarks to start focusing on visual history, spatial status updates, and other issues, but there is still a lack of consistent action assessment of multiple memory needs in long-term interactions.

WhenAIWhat kind of memory does it need to enter the ever-changing physical world, and how can it be judged whether it really remembers and can act on it?

Physical memory requires connecting visual observation, action and environmental feedback, preserving state changes and causal relationships in the process of interaction.

To answer this question, the OmniAI team at Zhejiang University has proposedEmbodiedMemory-Bench（EMem-Bench）Through four types of tasks, 2,554 executable interaction samples, the system evaluates whether the intelligent body can build memories from history and use it to complete subsequent tasks.

The work also introduces an external memory systemEmbodied-Memorizer（EMem）and learning how to manage and use memoryEMem-8BAround three weeks after its release, the benchmark has been downloaded nearly 30,000 times on Huggingface.

Link to the paper:https://arxiv.org/abs/2609.28236

The main page of the project:https://zju-omniai.github.io/Embodied-Omni/EmbodiedMemoryBench/

The code link:https://github.com/ZJU-OmniAI/Embodied-Omni/tree/main/embodied_memory

The reference link:

To find out the difficulty of long-range physical tasks, the research team first analyzed Gemini-3-Flash and Qwen3-VL-32B on EmbodiedBench's EB-ALF, EB-Hab.100 failed tracksAmong them, there are four categories of issues related to physical memory.91%：

Ignore the fine-grained visual cues (35%)The target was once shown in the image, but the model did not remember its location and could only be searched repeatedly.

Changes in environmental status (18%)Objects have been moved or processed, and models are still operating in the old state.

State of unrecorded interactive feedback revelations (31%)Capture: whether it is successful, the hidden state of the environment, the information that needs to be confirmed and remembered from the results of the action.

Unable to migrate past experience (7%)The previous demonstration or correction has given clues, and after replacing an object, the model still makes the same mistake.

The proportion in the graph comes from an analysis of the above 100 failed tracks, showing problems with visual omission, status tracking, feedback recording and experience migration.

These phenomena refer to the special requirements of physical memory: some information appears only briefly in a given set; some memories need to be updated as the environment changes; some states must be discovered through interaction; and some experiences need to be incorporated from specific events and then migrated to new objects.

For example, "the drawer cannot be opened" is not necessarily judged by an image, it requires an attempt and corresponding feedback; "the tableware should be placed in the drawer" is not a simple record of the position of the object, but a law of multiple corrections of common expression.

Each sample of the EMem-Bench provides a chronologically ordered history of multimodal interactions, followed by a follow-up task.

Intelligent entities need to retrieve critical evidence from history and perform navigation, opening containers, taking and placing actions in a simulated environment until they reach the endpoint required by the task. Assessment connects "what happened in the past" with "how to act now".

The benchmark sets four categories of tasks around four memory needs:

1. Passive observation is a form of passive observation.Examining whether an intelligent organism can retain microscopic visual information that it accidentally sees while performing other tasks and find the location of the relevant objects in subsequent tasks. Objects may be mixed in multiple objects, and observation does not yet know that it will be needed afterwards. For example, a laptop computer once appeared in a tea room and after some time, a new task requires a computer to be picked up and the intelligent organism needs to use that observation to find it.

2. This is a dynamic tracking system.It examines whether an intelligent organism can change with the location or state of an object, continuously updating the understanding of the environment. History may contain multiple records of the same object at different times, and models need to judge the current state of validity, avoiding obsolete information actions. For example, a book is first on the bed, then placed in a box, and finally moved to the desk, where the intelligent organism should look for it based on the latest location.

3. Interaction FailureExamining whether an intelligent organism can remember the environmental state and interaction constraints revealed by the results of the action, and adjusting subsequent selection accordingly. Information such as locked drawers, unusable containers needs to be acquired through an attempt, and the feedback must also correspond to specific objects. For example, once an open drawer fails to reveal "the upper drawer is locked", the intelligent organism needs to use this feedback to select the lower drawer that can be used.

4. Experience Generalization is a term used to refer to the generalization of experience.This requires the model to understand why past experiences are relevant and what situations are suitable for migration. For example, the past few corrections have required that knives and hooks be put in the drawer, and in the face of newly emerging forks, the intelligent body needs to migrate the family's receiving habits.

The four types of tasks are to examine visual detail retention, up-to-date status, interactive feedback memory, and learning repeatable experiences from past corrections.

The team first chose a navigable, interactive multi-room environment, followed by a memory clue around the four types of tasks that determine the follow-up. AI2-THOR's single-room scene is combined into a virtual home, with room status retained when the intelligent body leaves and returns to the room; ProcTHOR provides multi-room housing directly.

Subsequently, the planner inserts unrelated interactions with the target before and after the key clues, pulling the clues away from subsequent tasks. These interference tracks must be able to execute effectively without touching the key objects, leaking answers, or changing the state on which the target task depends. Thus, history has changed and the evidence needed to complete the task can still be clearly identified.

The average historical length of the first three classes of missions is78 and 83 stepsThe experience of broadening the mission to achieve111 stepsThere is still a middle ground between the key clues of each mission family and the target mission.52 and 56 stepsOn average, key images of passive observation tasks include:11 visible objectsIt is also a requirement for detailed memory.

Coverage of benchmarks1,118 scenes, 125 visible objects, 83 target objects and 33 containers and carriersAutomatic checks verify the environment, trajectory, evidence and feasibility, while artificial audits further verify whether the instructions are clear, historically valid, whether the clues are sufficient to support the expected action, and whether the mission end state is correct, with Article 2,554 being retained.

From multi-room environments, task-related memory clues and interference trajectories, via automated verification and artificial auditing, an executable assessment sample is formed.

In order to explore how these problems could be improved, the authors propose EMEM, which organizes the experiences into three complementary memories:

Scenes memory (scenes memory): Preserve selected visual images, describe and identify objects, and tie them to the scene, helping to distinguish "the cups of the kitchen" from "the cups of the bedroom". Similar images are screened to preserve new objects and useful visual details.

Space memory (Spatial Memory)A physical map maintains the location, relationships, and attributes of objects. After the book moves from the bed to the desk, new location relationships replace old conflicts while retaining other still valid attributes to form the newest known state of the world.

Events memory (Event Memory)Time records actions, outcomes and corrections, preserving interactive evidence such as "the drawer won't open". Multiple corrections can also be summarized as a short rule, and associated with the original events that support it, to be retrieved when faced with new objects.

Scenario memory preserves visual and scenic information, spatial memory maintains the latest world status, and event memory preserves the results of interactions and attributes them to experiences.

The whole process is made up of:Observation of fox writing fox retrieval fox action fox feedback updateThe model processes history in chronological order, with each step based on the current picture, the previous action and its feedback determining what to write. At this point, the subsequent task has not been given, and memory needs to be accumulated from the interactions that have already occurred.

After receiving the task, the model selects the memory that needs to be queried. Finding a book requires the latest location, avoiding locked drawers requires past failure feedback, and incorporating new dishes may require multiple corrections to the rules. The retrieval is first filtered based on structural conditions such as scenes, entities, and relationships, then supplemented with text matching with the necessary semantic matching, giving relevant evidence to the action stage.

New observations and feedback generated by execution continue to update memory. If the action fails or a significant state changes, the system disrupts plans, re-queries, and planning that have not yet been executed. Thus, the results of previous actions can affect the next decision, and memory is constantly updated with interaction.

With external memory, the model still needs to decide what information should be stored, what current tasks should query, and how to convert the retrieval results into actions.EMem-8BIt provides training around these three decisions to help the model learn how to perform memory operations.

The experiment compared general open-source models, physical models, closed-source multi-modal models, and representative multi-modal memory systems. The results showed that the performance of various models remained uneven in four capacities.

In the full-context setting of the paper, the model obtains a complete text history and receives current visual observations when executed.Gemini-3-Flash ranked first in the set with an average success rate of 64.2%.The Gemini-2.5-Pro is 60.8%, the open source model Qwen 3.6-27B is 58.2%, and the GPT-5.4 is 56.8%.

According to the results of some models drawn in Table 2, the average success rate is equal to the equality of all four types of tasks, and full-context uses text history and current visual observations.

Even with the same model, there may be a large gap between different memory abilities: Qwen3-VL-32B achieves success rates in experiential generalization.63.1%But dynamic tracking is only20.3%• Specialized training in body models has also not steadily translated into long-term memory advantages, suggesting that spatial perception and planning training also need to cover further state maintenance and experience use.

With fixed GPT-5.4-mini as the backbone, EMem will have an average success rate of35.3% increased to 58.9%increased23.6 pointsIn this comparison, the best results for other memory systems were 37.8% for TeleMem, and the overall advantage of EMem was 37.8%.21.1 points。

All systems use the same GPT-5.4-mini backbone. EMem has the highest average success rate among memory systems evaluated in the paper.

Improvements also appear on the open-source spine: no mission-specific training is required, and EMem is delivered on Qwen3-VL-8B, Qwen3-VL-32B and Mistral-Small-3.1-24B respectively.10.8, 14.1 and 16.4 points.Improved average success rate of EMEM-8B after memory operation training.46.0%More than the original Qwen3-VL-8B20.6 pointsIt's better to use EMEM directly than freezing bone.9.8 points。

Dissolving experiments also support the complementarity of three types of memory: the average success rate of the entire system is 58.9%, and the average success rate of removing a single memory decreases to 0.9%.46.9%–47.9%Among them, dynamic tracking relies more on spatial memory, interaction failure and experience generalization relies more on event memory, passive observation relies more on scene memory.

A natural thought is: Since the model forgets, can we directly insert more historical images into the context?

The author uses GPT-5.4-mini as a fixed800 samplesA control experiment was conducted, with 200 items per class of tasks. Both settings retained full text history and current images, expanded the setting and added as many historical images as possible to the context budget. The results were as follows:

Table 3 paper original results. Both settings use the same 800 samples and operating budget; input tokens have the same value for each sample.

In this group of control experiments, the average success rate was36.6% decreased to 26.2%It's coming down.10.4 pointsEach sample has an average input of tokens from41,770 increased to 206,489It's about the original.4.9 timesEach step of the model delays from4.1 seconds increased to 39.1 secondsIt's about the original.9.5 timesThis shows that the accumulation of primitive history cannot replace effective memory; the key is to retrieve evidence of tasks, update outdated status, and suppress irrelevant information.

In a further analysis of the failure, the authors examined the GPT-5.4-mini on the EMem-Bench.400 failed samplesThere are 100 items for each class of tasks.61.3%Memory use errors: The actions taken by the model conflict with historical evidence. Analysis has found that interaction failure tasks also contain more context-based reasoning errors to remember a failure, and the model still needs to understand this feedback and make new feasible decisions.

The EMem-Bench assesses long-term memory through four executable tasks and explores the organization, updating and use of memory with EMem and EMem-8B. Making past observations, actions and feedback the basis for subsequent decisions is the core focus of this work.

The current results come mainly from simulation environments. The future will require longer ranges, more complex environments and real robots to test these capabilities in the face of sensor noise, error execution and open world changes.