The video world model (Video World Model) needs to generate continuous images and maintain the consistency of scenes under the control of user-specified actions. For video world models based on self-return generated, history images are stored in the KV Cache so that similar scenes can be retrieved and replicated when generated. However, as the generation continues, the cost of storing the KV Cache is very high. For example, LingBot-World-v2 produces about 5 seconds of video displaying 93 tons, and the KV Cache requires more than 21 GiB of storage.
Quantifying KV Cache to 2-bit is an important direction to mitigate this problem, and there are currently methods to apply 2-bit KV Cache quantification to auto-return video generation and achieve VBench rating performance close to BF16. However, the research team found that when applying these methods to a video world model, there may still be noticeable blinking, blurring and detail distortion in the images generated, even when the rating score remains almost unchanged.
Researchers from Harbin University of Industry (Shenzhen) iLearn-Lab and the LV-Lab team at the National University of Singapore have analysed this phenomenon in depth and proposed the untrained 2-bit KV Cache quantification framework QuantWM for video world models. The method is based on attention protection and significantly improves video quality after quantification on five models, while achieving a maximum of 6.20-fold KV Cache compression.
The paper:
Projects:
Near scores don't mean good results
The existing method Quant-VideoGen (QVG) has used 2-bit KV Cache quantification to generate retrogressive video, achieving performance close to that of the BF16 model on the video benchmark. However, the team found that these scores did not fully reflect the visual degradation brought about by quantization in the video world model. In HY-World 1.5, for example, the BF16 and QVG's VBench Temporal flickering scores were 95.62% and 95.85% respectively. However, the video generated can still be observed to produce severe image flickering and visual degradation at the input of the quantization, 1 showing this inverse. This shows that even the rating indicators associated with flashing have not been able to fully capture the visual world model's quantization performance.
Figure 1 2-bit KV Cache quantified sequential flashing and decreased graphics quality, while video rating indicators remained virtually unchanged.
The key to a smaller error is to make a bigger impact.
The team further analyzed the source of this phenomenon. The KV cache contains Key and Value. The Key can be understood as a crawler index for retrieving historical information, while the Value is the crawler content that is read after retrieval. After quantifying the two respectively, the team found that only compressing the Key to 2-bit results in more noticeable visual degradation than just quantifying Value. More noteworthyly, compared to Value, the Key's own quantification errors tend to be smaller, but still cause greater output errors. This means that the difference in video quality is not because the Key is more difficult to quantify, and the single quantified difference in size cannot be explained.
To explain this phenomenon, the research team further analyzed the attention calculation in the video world model. The current Query matching score with the Historical Key determines how the model allocates attention to historical information. Therefore, the quantization error introduced by the Quantitative Key may change the relative ordering of these points, causing the model to shift to focus on another key, another spatial location. As shown in Figure 2, for the same Query, the historical time-space location with the highest prior to quantification of attention scores changes, sometimes even across different channels. For continuous generation, this bias may disrupt the model's use of historical scenes, thus affecting image stability, causing sequence blinking and quality.
Figure 2 The same query may be selected from different historical columns or image blocks before and after key quantification
Step 1: Select a more appropriate concentration center after quantification
The first design of QuantWM was to quantify sensitivity perceptual clustering (QSAC). In QVG, each Key is assigned to a clustering center, with the center retaining high accuracy, and the residue between the two is quantified 2-bit. The research team believes that the distance from the nearest center is not necessarily the most appropriate result after quantification. The dynamic range of the residue, as well as the percentage of attention that errors in which channels will ultimately affect.
Step 2: Compensate the critical direction with a small amount of information
Even after QSAC 2-bit quantification, the most immediate solution is to compensate for errors in attention logits, but it is very expensive to preserve the entire amount of quantization errors. The team observes that the energy of historical queries tends to be concentrated in a few directions. For example, a layer of LingBot-World-v2 covers about 71% of Query's energy in just the first 8 main directions, allowing the compensation to focus on maintaining its portion in those main directions. Based on this, QuantWM further introduced sub-spatial attention compensation (PSAC), PSAC in a low-order format that preserves key quantization errors in those main directions, and compensates for key revisions when generating key, thereby modifying subsequent processes.
Five types of model validation: improved paint quality and significantly reduced cache
The team verified QuantWM on three video world models, Matrix-Game-2, LingBot-World-v2, HY-World 1.5, and further verified its generalization capabilities on two video generation models, LongCat-Video and Causal-Forcing. In the 480p, 93p video generation comparison, QuantWM's PSNR, SSIM and LPIPS outperformed QVG and KIVI, while visual content showed significant improvements in the quality of video produced by QuantWM, flashing, blurring and blurring. On five evaluation dimensions of VBench, QuantWM earned the best average ranking for comparison methods in each model. In addition, the paper further reported results of 720p and 1-minute video evaluations, which generated higher resolution and video quality ratings for video generations.
Figure 3 BF16 (left) ✓ QVG (middle) ✓ QuantWM (right) ✓ Generate video contrast (upper: HY-World-1.5, middle: Lingbot-World-v2, lower: Matrix-Game-2)
Attention analysis further indicates that QuantWM is able to reduce the selection bias in historical time and space caused by quantization. As shown in Table 1, in Matrix-Game-2, the ratio of selection variations in the highest scored historical token relative to BF16 decreased from 53.88% to 10.19%, while in HY-World 1.5 it decreased from 61.36% to 13.92%.
Table 1 Highest scoring historical token Select the difference contrast
In terms of system implementation, QuantWM designed the Triton kernel exclusively, with actual KV Cache compression reaching a maximum of 6.20 times after considering additional costs such as aggregation centers, quantification parameters and compensation information, as shown in Figure 4.
Figure 4 480p 93-inch video of the KV cache with maximum compression of 6.20 times
What is the meaning of this verse?
QuantWM reveals a problem that is easily overlooked in video world model low-bit KV Cache compression. Even if the video rating score is close, it does not mean that it is still possible to use historical scenario information steadily after quantization. Therefore, the compression study of video world models requires a combination of benchmarks, actual video performance, and a multi-faceted assessment of how the model uses historical information.
The author's note:
The study was carried out by a team of Prof. Zhang Liu from Harbin Industrial University (Shenzhen) iLearn-Lab in collaboration with a team of Professors from LV-Lab@NUS Yangtze, National University of Singapore.
iLearn-Lab Professor Zhang Chen's team has long focused on model compression, efficient training and reasoning.
The LV-Lab@NUS team has long focused on video generation, world models, and AGI-related research.
Jiaqi Zhao, Ph.D. and Xiaobin Hu, Ph.D.