Recently, the National University of Singapore Minghui Guo and the University of Oxford Shengqiong Wu, Hao Fei, in their paper V-RAE: Rethinking Video Latent Spaces for Generation, presented video representation from the V-RAE encoder.
As video generation models continue to grow, data and computing continue to grow, people tend to focus on stronger generation backbone and longer training, but they tend to ignore a more basic question: in what space do generation models learn?
The mainstream video generation system typically uses VAE to compress the original video into a latent representation, and then the distribution of these representations is studied by scattered or self-returning models. This latent space is like a video-generated crystal lattice base: it determines what the model needs to learn, which also affects the training difficulty, conversion speed and final quality. Traditional video VAE is largely reconstructed with pictograms at its core, and is good at preserving texture, color and local detail; however, crystal can accurately restore real video frames, which does not mean that crystal is easy to generate entirely new video frames.
Recently, the exploration of Representation Autoencoder (RAE) in image generation has begun to rethink how this subgroup should be constructed. Unlike traditional VAE, which learns latent from pixel rebuilding targets, RAE directly utilizes high-level representation that pre-trained visual models have learned as a generating space, allowing generating models to emerge from a latent space with a more rich semantic structure.
This naturally raises a new question: can this idea be extended further from image to video? can pre-trained visual representation also become an effective video generation latent?
In addition to spatial semantics, there is also the need to model movements, time dynamics and cross-channel dependency, while there is considerable time redundancy in step-by-step visual representation. Therefore, we verified the viability of representation-based latent on video in the V-RAE (Video Representation Autoencoder) system and further studied what kind of latent space is really suitable for video generation.
Expand RAE from Image to Video
V-RAE uses a frozen visual base model as an encoder to study DINOv3, SigLIP2, EUPE and V-JEPA 2.1, four visual representations with different pre-training paradigms.
The key here is not to use these representations as additional supervision, but to make them directly the latent space in which models are generated. The encoders remain frozen, learning only the subsequent time-compression modules and video decoders, so as to preserve as much of the semantic structure as possible that the underlying model has already formed.
But from image to video, a direct problem is that video representation is too long.
While sequential encoding can preserve a wealth of information, it also means that the number of latent tokens increases rapidly with the length of the video, making generating costs unacceptable; while simply compressing average or roughness in the time dimension, it is easy to lose motion information and cross-sectional relationships.
As a result, V-RAE introduced a lightweight temporal attention pooling module that adapts to the continuous pitch representation to aggregate content, while achieving 4x temporal compression to preserve semantic and dynamic information. The Transformer decoder then rebuilds the continuous video with the 3D RoPE.
This design allows us to really fairly compare, under a similar latent budget: the traditional reconstruction-oriented VAE latent, with the pretrained representation-based latent, which one is better suited for video generation?
Representation-based latent is also established in video.
The experiment first answers the most basic question: Can the core idea of RAE extend to video?
The answer is yes.
Although these representations were not learned around pixel reconstruction, V-RAE still achieved strong video reconstruction performance.
On the Kinetics-600, the V-RAE reached 2.13 rFVD, further lower than the best result in the mass-training video VAE evaluated by 3.58; on the UCF101, its rebuilding performance was also close to the strongest VAE baseline.
The most noticeable difference is the information contained in the latent itself.
In UCF101's semantic probing, the V-RAE latent has a maximum accuracy of 90.92%, compared to 30.83% for the traditional VAE latent.
This suggests that even after time compression, objects, movements and cross-linkages that have been formed in pretrained representation can be better preserved.
But the real question is not whether it contains more semantics, but whether those semantics will actually make video generation easier.
More structured latents are actually easier to generate.
To answer this question, we further fixed the number of generating skeletons, latent tokens, and training settings, changing only the latent space defined by the underlying autoencoder.
The results showed that the four V-RAE representations on both UCF101 and Kinetics-600 resulted in better downstream generation performance.
The best models reached 117.86 gFVD (UCF101) and 19.16 gFVD (K600) respectively. It was more worthy of training efficiency: on UCF101, the V-RAE required only about 30,000 updates to reach the production level of about 150,000 updates of the traditional VAE; on K600, the highest observed convergence was about 6x faster.
The results show that the value of representation-based latents is not just that they store more information.
More importantly, it actually changes the problems that generators need to solve.
In the traditional reconstruction-oriented latent, generating models still need to rediscover objects, movements, and scene structures from these pixel recovery-targeted codes; while in the pretrained representation space, these higher-level structures have been explicitly organized so that generating models can be learned more directly: how visual states change over time.
In this series of experiments, we also observed a more interesting phenomenon, and one that is particularly worth rethinking in video production.
Good Reconstruction ≠ Good Generation
We are used to using reconstruction quality to measure the good and bad of an autoencoder.
However, in terms of generation, this criterion may not be sufficient.
In experiments, the rFVD of different autoencoders was only weakly correlated with the final generation quality. On UCF101, the correlation coefficient for the two was only 0.200; on K600, it was only 0.473.
In other words, a latent space can be very good at reconstructing real video, but it can still be a difficult space to generate.
Why not?
Because reconstruction only tests the decoder to see if it can recover the latent output of the actual encoder.
However, it is not always possible to accurately predict the latent trajectory of the actual data in real predictions.
If the latent space itself is not smooth enough, even small deviations can be enlarged after decoding, resulting in ghost shadows, flashes, structural drift or unnatural motion in the video.
Therefore, a truly generation-oriented latent space should not only answer: Can real video be encoded and reconstructed accurately?
It should also be answered: Is the space still stable when the forecast results deviate slightly from the real latent?
tFVD: from reconstruction quality to generative utility
Based on this observation, we have further proposed tFVD.
The basic idea is simple: place local inputs between adjacent latents over time, actively construct some latents that deviate slightly from the actual encoding trajectory, and then observe whether these representations can be decoded into natural, continuous video.
So what we're really testing is not just autoencoder's reconstruction ability for real data points, but latent space: temporal smoothness, and decoder's robustness for generation error.
The results show that this indicator is significantly more closely related to actual downstream generation performance.
The coefficients for tFVD and generation quality at UCF101 and K600 are 0.621 and 0.919, respectively, significantly higher than the traditional reconstruction metric.
This result also gives a more general indication: for video generation, a good latent space should not only accurately represent what has already happened, but should also allow for a continuous, smooth, predictable structure between different visual states.
In other words, we should not only be concerned about whether the latent is reconstructable, but we should also be concerned about whether it is generatable.
Moving forward from video generation to future forecasting
If representation-based latency is really learning a more structured set of visual states, then its value should not be limited to unconditional video generation.
So we put it further into future video prediction for verification.
On Cityscapes, using exactly the same predictive architecture and training budget, the V-RAE lowered the gFID from 15.02 to 11.52 for the traditional VAE and lowered the gFVD from 144.47 to 111.36.
One very interesting phenomenon is that the reconstruction rFVD of the V-RAE is actually worse, but the future prediction is significantly better.
This reaffirms the previous discovery that good reconstruction is not the ultimate goal for predictive/generative modeling.
A more semantic, smooth, and predictable latent state space, rather than accurately restoring each pixel, may be the most appropriate interface for the model to understand and predict the dynamic world.
In such a representation space, the same latent interface can be connected simultaneously: visual understanding → state prediction → video generation → pixel decoding.
Model learning is no longer just about how the pixel changes over time, but more closely about how the transition between visual states occurs.
Re-thinking the video-generated silicon substrate
Therefore, the meaning of V-RAE is not simply to move image RAE to video.
We first verified the system: a pretrained representation-based latent paradigm can also be effectively extended to video.
More importantly, the more complex video setting further exposes some of the previously overlooked problems in latent space design:
Video representation needs to deal with unique temporal redundancy;
semantic representation can retain a rich object, action and temporal structure after compression;
A more structured latent can lead to better generating quality and up to about 6x faster convergence;
Reconstruction quality cannot adequately predict generative quality;
The temporal smoothness and prediction robustness of latent space are critical to video production.
The advantages of representation-based latent can also be further migrated to future video prediction.
These results all point to a more fundamental question: how should we evaluate and design a truly generative latent space?
In the past, autoencoder was seen more as a compression and rebuilding module.
But from the perspective of generating models, it actually defines the whole learning problem as a sequence of coordinates.
A good video latent space should not only seek to reconstruct the past more accurately, but also make the future easier to generate and predict.
The paper:
The main page of the project:https://v-rae.github.io
The code is:https://github.com/V-RAE/V-RAE
Introduction by the author
Minghui Guo, a graduate student at the National University of Singapore, is the first author of the paper V-RAE, which focuses on multimodal base modeling, visual representation learning and video generation.
Shengqiong Wu, Ph.D. from the Department of Computer Science at the University of Oxford, graduated from the National University of Singapore, where his research focused on multimodal basic models, intelligence, visual generation and spatial intelligence.
Hao Fei, Ph.D. from the Department of Computer Science and Big Data at the University of Oxford, studied focused modeling, video intelligence and world modeling. His work has been published in important conferences such as ICML, NeurIPS, ICLR, CVPR, has been awarded the WAIC Bright Star, the Rising Star, and has been consistently ranked in the top 2% of scientists worldwide.