In recent years, the video world model has become one of the most sought-after directions for generative AI. With only a reference chart and a scenario description, the model can generate a free-to-use virtual world based on the user's input camera posture or control instructions. But when the interaction extends from a few seconds to several tens of seconds, an embarrassing problem arises: the model loses its memory.
Recently, a team of researchers from Meta Reality Labs, the University of Illinois at Chicago, and Brown University has proposed World-T3 ("Test-Time Training"). World-T3 is a camera-controlled video world model that trains during the production process by testing and sliding the historical observations in the context window into a set of dedicated speed weights (fast weights) written in camera posture; when the camera returns to its previous position, the model is repositioned as a key to read out the scene features of the time.
On a long-duration retrospective benchmark containing a 120-section "go-to + original-return" trajectory, World-T3 raised the PSNR from 13.52 dB at the strongest baseline to 18.31 dB, reducing the absolute spin error from 3.344° to 1.075°. At the same time, the 4-step model, distilled by two instructors, ran at 20.6 FPS at a resolution of 1280×704 on a single NVIDIA B200 GPU, and the memory was updated in real time during the production process. The 4-step World-T3 was ranked first in all six retrospective consistencies compared to video models such as Matrix-Game 3.0, HY-World 1.5, DreamX-AR, Lingot-Fast, and SANA-WM.
When the camera returns to its previous location, World-T3 reliably restores the scene; while the existing regression baseline (such as SANA-WM) forgets the contents outside the context window and synthesizes different scenes.
The first co-authors of the paper are Junyi Wu and Zekun Li, who completed the work during the first two co-authors' internships at Meta Reality Lab. The project is led by Xijie Huang, a senior researcher at Meta Reality Lab. Junyi Wu is now a fourth-year computer science student at the University of Illinois at Chicago, where his research focuses on computer vision and physical intelligence; and Zekun Li, a fourth-year computer science student at Brown University, taught by Professor Srinath Sridhar, whose research focuses on video content generation and modeling learning.
Title: World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
Link to the paper:https://arxiv.org/abs/2610.17434
The main page of the project:https://adreamwu.github.io/World-T3/
The World-T3 method
The World-T3 is based on the pre-trained graphics to video model Wan2.2-TI2V-5B, with three progressive training phases to achieve camera control, long-term memory and real-time generation capabilities.
How to train your memory by using camera posture as an indexing test
As opposed to the more common method of storing historical chips or tokens, the World-T3 has a lightweight TTT memory module mounted on each layer of the Transformer. In the production process, whenever a video block is squeezed out of the context window, its "camera position → image content" correspondence is written into the fast weight group; when new video blocks are generated, the model uses the target camera position to query the memory, retrieving the scene characteristics that have previously appeared at that location.
Returns to minimum 2 times writing: just write new information into memory
Camera movements are usually smooth, and adjacent video blocks are bitwise similar, almost synchronous. If the previous TTT method is used to write step-by-step updates each time, the memory is repeatedly reinforced by the repetitive content, instead of blurring the previous information.
Double-teacher distillation: 4 steps generated and memory retained
In order to achieve real-time interaction, the authors distilled the 50-step sampling of the self-return model into a 4-step generator. Two teachers were employed at the same time when distilling: the two-way attention teacher responsible for short-range graphics and motion coherence, and the self-return teacher with TTT memory responsible for long-range spatial consistency, as well as visual quality and durable memory.
World-T3 generates visibility
Let's take a look at the most intuitive "go + return" test: the camera moves forward for 5 seconds along a given path, then returns for 5 seconds. A model that truly remembers the world should be almost identical to the first path. All methods use the same path, text description and camera path.
To the left is real video, to the right is World-T3, to the bottom is SANA-WM and DreamX-AR; the white line in the bottom progression bar is the moment the camera turns its head.
In longer generations, the World-T3 is also capable of retaining memory. Below is a real camera trajectory of about 30 seconds, containing multiple approximate return visits, while the model generates a continuous update of memory:
Baseball pitches (three times faster), from top to bottom, followed by World-T3, DreamX-AR, SANA-WM.
More World-T3 results are as follows, the same 10-second "go + return" trajectory covering indoor, outdoor, urban and natural scenery:
World-T3 generates more results: 2 times the speed.
World-T3 method details
Stage one: Two-way camera conditions teacher
The goal of the first phase was to allow the pre-trained video model to learn to follow the control of the camera. The authors used the Plücker raymap to encode each of the cameras: for each position on the subspace grid, recording the direction and rectangle of the corresponding rays, forming a 6-way camera port aligned with the subspace variable.
Due to the use of two-way attention at this stage, the model generates the ability to see the entire video at each turn, thus making the given trajectory a coherent movement. A well-trained two-way teacher assumes two roles in the subsequent stage: initiating the self-return model for the second stage and providing reference scores in the distribution matching stew for the third stage. It is worth noting that the same viewing angle produces exactly the same ray chart, which also lays the foundation for posture-based memory detection.
Camera conditions for injecting: Plücker radiographs modulate each DiT module through the camera-related scales and planes.
Stage 2: Self-return teachers with TTT memories
The bi-directional model cannot interact in real-time, so the second step is to transform it into a block-causal self-return model and add durable memory.
In the above context design, when generating the second video block b, the model looks at only two parts: the first generation block serves as a continuous information sink, used to stabilize the overall appearance; the nearest K clean video block consists of a sliding window, ensuring local coherence between the block and the block.
Outside the window, the history is given to the TTT memory. Each DiT layer is equipped with a fast weighted memory consisting of two layers of MLP: the first layer W1 is fixed, responsible for mapping the position keys to non-linear character space; the second layer reads the W2 matrix continuously updated during the generation process, storing accumulated scenes. A key design is that the memory key only comes from the camera's beam, and the value comes from the clean and noise-depressed state, so reading and writing are dispersed with noise decomposition.
When a video block is squeezed out of the context window, its radiographic image and cache state are passed through the bit position encoder and the content encoder respectively, receiving a pair of key values, which are then "consolidated" into memory. The authors write this process as the following privileged regression problem, and introduce the corresponding recursive minimum binary update:
The goal of memory consolidation optimization (above) is to return the rule of least double updating (below).
where P is the inverse of the matrix of the cumulative positional characteristics (after the regular point). The update direction Pφ is equivalent to a whitening preconditioner, which weakens the distribution on the memory-covered direction, allowing only the remainder of the actual new wave to update W2. Because W1 is fixed, this update gives an accurate minimum resolution of the target function at each step, so the final memory state is unrelated to the previous sequence of observations.
When reading, the model performs a forward calculation using a camera beam diagram of the target block, and then injects the layer into the stealth state through the gate control residue in each channel. When training, the gradient passes through recursive updates, combined optimization key projections, basic matrix readings, encoders, sample weights and gate control parameters; when deployed, the main and encoders are not all frozen, and only W2 and P evolve online, requiring reverse propagation of each entry to the distributed Transformer.
Stage 2: Self-return instructor with TTT memory. (a) The node block constitutes a boundary context with the nearest clean block K, and the viewpoint of the retrieval may lie outside the window; (b) The key value of the blocks that have been deleted is reinforced in memory with the return of the minimum binary when written, and recovered memory characteristics in the target location when read; (c) recovered memory is injected into each DiT module through the gate control residue.
Stage 3: Dual teacher distribution matching distillation
The second stage of the self-return model requires 50 steps of sampling for each block, and it is difficult to meet real-time interaction. In the third stage, the consistency distillation separates the continuous flow matching into 4 steps, and then uses distributed matching distillation (DMD) to recover the paint quality and consistency loss caused by the multi-step direct jump.
Unlike the common single-teacher DMD, World-T3 breaks down the real data scores into two complementary sources: the first phase is for two-way teachers to evaluate images in a fixed time window, ensuring short-term realism and movement coherence; the second phase is for self-return teachers to evaluate TTT memory along a long sequence, ensuring long-term spatial consistency. Each teacher is equipped with the same structured critic network, trained on samples generated by students. In particular, prior to calculating memory scores, the teacher and the critic's memory finish writing on the students' own generating trajectory, so this directly punishes the students in their own rollout accumulated trajectory drift and backward inconsistency.
Engineering Optimization: 20.6 FPS generated in real time
Without changing the model structure and numerical accuracy, the authors improved the end-to-end speed by a series of reasoning optimizations by 3.12 times: between 4 de-noise steps, repeating the memory reading results and the upper context key after rotating the RoPE, repeating the unchanged upper context preview, then combining torch.compile, FlashAttention-4, cuDNN9 with channels-last layout VAE decoding, convoluted shape automatically selecting the fastest engine, back-end thread pre-noise, and so on. Producing a 30-second video's end-to-end time decreased from 70.48 seconds to 22.59 seconds, reducing the time spent on each video block from 1855 milliseconds to 589 milliseconds.
The time it takes to break down the video blocks on a single GPU: before optimizing:#1I'm not sure what I'm talking about.#10）。
The results of the experiment
Evaluation settings. The authors never participated in the training test session, which consisted of 120 "go + return" evaluation segments, each 10 seconds (160 frames, 16 FPS, 1280×704), the camera returned to the original route in the 80th. Retrospective indicators compared the same pairing of the camera positions on the journey and return, four groups of pairs of time intervals (24×36 sub-frames) all over the model's 21 sub-frames, and only the models that truly "remember" the scene could respond simultaneously. Evaluations reported the DINOv2 semantic similarity, the PSNR compared to real video, the camera's number of observations based on the VGT-Ω reconstructed trajectory, and the VBench image quality.
The main results of the review review review collection.
Retrospective consistency is significantly ahead. Compared with the strongest baseline of retrospective indicators, SANA-WM, the 4-step World-T3 halves the DINOv2 characteristic distance by almost half (0.0800 → 0.0390), and the PSNR increases by 4.79 dB (13.52 → 18.31 dB).
The camera follows more accurately. For accurate re-visits, it is first necessary to accurately go to a designated location. The absolute rotation error for World-T3 decreases from 3.344° at SANA-WM to 1.075°, and the camera matrix error decreases from 0.1200 to 0.0318.[6] The authors note that the adjacent arc error on some baselines is small, but the absolute error is high, indicating a noticeable cumulative drift; and the World-T3 error is lower on both local and global scales.
The graphics quality remains the same. On a seven-dimensional VBench average score, the World-T3 is 74.52, with the baselines 72.87 x 75.32) and real video 75.69 in the same range.
After 50 steps of distillation to 4 steps, the student model's return PSNR improved slightly (16.69 → 18.31 dB), outperforming the teacher in 91 of 120 paragraphs; there was no statistically significant difference in the four camera indicators compared to the teacher.
From "generating the next generation" to "maintaining a sustained world", video world models are crossing a critical threshold. World-T3's answer is: not to keep history as long as possible, but to train the world you've seen to write in the model parameters during testing, and to write new information every time with a minimum of doubling. World-T3 is capable of simultaneous long-term memory and real-time generation on a single GPU with fixed-sized parametric memory and fewer instructors.
The authors also pointed to the current limitations: the model relies on three-dimensional basic models such as VGGT-Ω to provide camera trajectories, and the bias estimation errors affect memory retrieval based on bias; details may still deteriorate when the length is generated far beyond the training length.
In a world where models are pursuing a larger scale today, the World-T3 offers a path that combines memory and efficiency and hopes to provide new inspiration for real-time and long-term interactive world generation.
© THE