Over the past year, video generation has been moving rapidly towards long-horizon generation, streaming generation and interactive world models. But as the generation ages, a fundamental question becomes increasingly difficult to overcome: how should the model "remember" the past?
At the same time, video generation/world models are increasingly using memory as the core of their work: historical frame, VAE latent, KV cache, recurrent state, 3D world state, and even writing historical information directly into model parameters.
Researchers from top institutions such as the Hong Kong University of Science and Technology, the City University of Hong Kong, the University of Pusan, CMU, NYU, NTU, and others have recently completed a 110-page review: The Past Frames the Future: Memory for Autoregressive Video Generation, hoping to redistribute relevant research in different directions such as long video generation, streaming generation, interactive world model, and put it in a unified Memory framework.
Link to the paper:https://arxiv.org/pdf/2609.28466
GitHub：
Why is memory increasingly needed to generate long videos?
Autogressive, AR) video generation continues to generate the next token, frame, or video chunk according to its history. The problem is: the history is getting longer, but the context to which the model can access directly is always limited. Restricted by the context window, display, KV-cache storage and latency, the actual system usually retains only a limited local context. As the rollout progresses, the early character identity, scene layout, motion status and interaction results will gradually leave the current window.
The review defines Memory as a continuously maintained historical state across AR steps: even if the original evidence has left the current local context, it can still affect future generation. In other words, Memory is not the same as Longer Context; the key is how historical information is maintained, managed, and reused.
On this basis, the paper reorganizes the entire field from five questions:
Forms: In what form does memory exist?
Functions: What does memory need to be preserved?
Operations: How does memory work?
Learning: How is memory learned?
Evaluation: How to prove that the model actually uses Memory?
Forms: Where is memory?
The paper begins with a memory carrier, which will be divided into four categories: Visual Memory, Implicit State Memory, Explicit State Memory, Adaptive Parametric Memory.
Among them:
Visual Memory directly retains past frames, clips or VAE latent, with the advantage of high accuracy of visual information, but also higher storage and computing costs.
Implicit State Memory preserves the history of KV cache, recurrent state, SSM state, or compressed neural representation, which is more compact and closer to the internal calculation of the model, but less explainable.
Explicit State Memory explicitly maintains world-level information such as entities, poses, geometry, 3D maps, event states, etc. It not only saves "what the past has seen" but can also represent "what the present world should be in".
Adaptive Parametric Memory goes further by writing history into fast weights, LoRA, or other online updated parameter states, allowing past information to directly alter forward computation after the model.
These four types of memory do not have absolute advantages. Actual systems are also increasingly inclined to mix multiple carriers while taking into account visual detail, efficiency, structured representation and online adaptation.
Functions: What do models really need to remember?
From "Where to Exist" to "Why to Exist", the paper summarizes five core functions: Identity Preservation, Spatial Preservation, Dynamic Preservation, Semantic Preservation, Causal Preservation.
In a nutshell:
Identity: The character returns after leaving the picture and remains the same character;
Spatial: The camera leaves and returns, and the scene layout and geometric relationships remain intact;
Dynamic: An entity can continue to evolve in its motion and state even if it is temporarily invisible.
Semantic: Role relationships, goals, events and narrative commitment will not be forgotten;
Causal: Once an action has truly changed the world, the consequences need to persist.
Among them, Causal Preservation is particularly important for the interactive world model: if the user has already opened a door, then when the action itself is already out of context, "the door has been opened" should still be the current world state, rather than being re-generated by the model into a closed state.
Operations: Memory is not a library, it is a life cycle
It is not enough to save the right information. A memory system also needs to answer: what to write, what to read, when to update, what to delete when the capacity is limited, and how it really affects the generation after extraction.
This point of view reveals an important issue: Memory is preserved and does not represent that memory is actually effective.
Critical information may not have been written at all, or it may have been stored, but could not be retrieved correctly; it may even have been retrieved, but the integration was too weak to actually change the result.
Learning: How do models really learn to use Memory?
Memory in video generation has a natural closed-loop property: the current content generated by the model will become future memory. Therefore, a generation error may be written into the KV, latent, world state or parameter state, and then continue to affect subsequent rollout.
A central problem arises from the distribution gap between training and inference: when training, models tend to use ground-truth history, while when deploying, they are faced with their own step-by-step history. Thus, from history augmentation to self-rollout training, more and more methods are beginning to allow models to learn directly from their own generated memory states.
Evaluation: Video consistency doesn't mean the model actually has memory
The paper also emphasizes: Long-term Consistency ≠ Memory.
A character always appears consistent in a long video, and does not prove that the model actually used information from a long time ago. The current prompt may have re-described the character, and the model may have guessed a reasonable result just by prior. Therefore, true memory-revealing evaluation requires a man-made creation of an information gap: the correct future output must rely on information from the past, which is no longer visible in the current context.
Typical scenarios include entity reappearance, scene revisitation, out-of-view state evolution, and delayed action effects.
This also means that the future benchmark should not only ask "Does video look consistent?" but also answer: Is history really preserved? Can it be removed when needed?
Where will the next generation of video memory go?
The review concludes with a summary of six areas of interest:
Unified and Composable Memory: lets Visual, Implicit, Explicit and Parametric Memory no longer fight each other, but share interfaces such as entity, time, coordinate, confidence.
Resource-Aware Memory: It's not about storing history indefinitely, it's about learning what information is worth saving, compressing, or reactivating for the long term.
Trustworthy Memory Updating: Avoid writing generation errors as "facts" permanently into Memory and support confidence, revision, and rollback.
Self-Rollout Learning: Allows the model to truly learn its own generation of memory distribution at the deployment stage.
Interactive and Causal Memory: Moving from "remembering historical observations" to "remembering the ongoing changes that action causes in the world".
Comparable Memory Evaluation: from single memory