With the rapid evolution of video generation models such as Sora, Runway and others, AI video generation is becoming increasingly popular. However, the slow pace of generation and high computational thresholds make it difficult for most developers and creators to locate it. For a 5-second 720P video, traditional methods take more than an hour to complete a widespread generation pipeline, which presents huge challenges for real-time creation, mass generation and consumer hardware deployment.
What's even more puzzling is that when researchers tried to break through the performance bottleneck through extreme scarcity (97% scarcity), they discovered an unusual phenomenon: training losses continued to decrease, video quality was dramatically deteriorating with the cracking of characters, background distortion, and time-order confusion.
To address this problem, researchers from institutions such as Beijing University, Tsinghua University, and Alibaba have proposed SparkDiffusion: the first video generation acceleration system to integrate sparse attention, low-step distillation, and low-precision quantification into a unified framework (with open-source weights and complete training codes), driving DiT video generation from "decentralized optimization of individual components" to a new paradigm of "end-to-end systematic acceleration".
SparkDiffusion: Mitigating the High-Sparsity Trap via Staged Sparse Warm-up, Trajectory-Mixed Distillation, and FP8 Quantization
The main page of the project:https://sparkdiffusion.github.io/
The code address is:https://github.com/AlibabaResearch/SparkDiffusion
Research institutions: Peking University, Tsinghua University, Alibaba University of Electronics and Technology and the Harbin University of Industry
Key findings
The longer the training, the worse the quality of the "high scarcity traps".
Extremely rare anomalies
The team found that when the scarcity rate was pushed from 90 percent to 97 percent (computation dropped from 10 percent to 3 percent), training losses continued to decrease, and video production began to break, distort, and lose sequence coherence.
The team called this phenomenon the "High-Sparsity Trap":
At extremely high scarcity rates, scarcity video DiT in a gradually supervised architecture will fall into a failure mode, and while single-step verification losses continue to be optimized, the quality of terminal generation stagnates or even declines, and it is impossible to substantially restore the quality of the terminal by extending gradual training.
Where is the problem? Mistakes accumulate at the high noise stage
The scattering model gradually generates video along the noise trajectory. The traditional method of step-by-step monitoring, such as Flow Matching, only requires the model to predict the correct speed at a particular point in time of noise, but does not directly restrict "what will be produced after this step is combined with subsequent trajectories".
Key Experiments: Oracle Intervention
The researchers performed an Oracle intervention experiment: at a 97% thinness rate, they replaced the predictions of thinner teachers with thinner students in different sampling ranges.
I found out:
Five Steps to Fixing the Highest Noise: Removing Most of the Terminal Errors
5 Steps to Fixing Minimum Noise: Improvement is Limited
Conclusion: Structural errors in high-noise phases are amplified by subsequent steps, ultimately leading to terminal quality collapse.
The solution
From diagnosis to treatment in stages
Core insights: terminal coordinated supervision
The problem with the traditional method: just overseeing "whether the current prediction of this step is accurate", regardless of where subsequent cumulative errors lead to the results.
The solution to SparkDiffusion: Terminal-Aligned Supervision, restricting "what this step will eventually generate" during training, avoids accumulation of errors from the root.
Theoretical verification: two-dimensional sequence distribution experiments
There are six two-dimensional sequence distributions:
Only gradually trained 95% rare models: the end of the trajectory clearly deviates from the data distribution
After integrating terminal alignment distillation: The final distance of the less advanced students is once again close to the tightest teacher
Method design
RoLA + CrossDistill + FP8
Step-by-step strategy: first adapt and then modify
Based on the above diagnosis, SparkDiffusion proposes a three-stage process:
RoLA: Keeping information globally for extreme scarcity
SparkDiffusion adopted by default the team's proposed RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers as a scarcity attention module. While the RoLA maintains high-energy query-key interaction through a block of scarcity branches, the low-order linear branches of rotational position information are used to replenish the scarcity-abandoned global context.
CrossDistill: 3 Steps to Quality and Diversity
SparkDiffusion's step-by-step distillation uses the team's CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation. CrossDistill sets intersections on the noise trajectory. Both PCM consistency objectives and DMD distribution matching objectives are not only suited to isolated moments, but use follow-up trajectory or final output structural oversight, and therefore belong to terminal alignment oversight:
The high-noise phase uses one PCM consistency step to follow the teacher's rough structure and movement trajectory while maintaining the diversity of different random seeds;
The low-noise phase uses two DMD distributed matching steps to directly correct endpoint visibility errors, enhancing detail and realism.
Finally, a three-step, CFG-free student model was obtained. It used a terminal alignment signal to cross the high-thinness trap, while balancing quality and diversity through high- and low-noise division.
It combines FP8's quantitative strategy with team-developed high-performance convergence algorithms to convert theoretical computational gains into actual delay reductions.
The results of the experiment
The single-card RTX 5090 achieves 265 times the acceleration
Key performance data
SparkDiffusion maintains high-quality fast generation in the following configurations:
At 90% scarcity: SparkDiffusion outperforms FastWan and TurboDiffusion across all indicators
Pushed to 97% scarcity: mass slightly decreased but still close to the density baseline, accelerating from 201× to 265×
Upper H100: Acceleration of 220x under the same configuration, absolute delay down to 8 seconds
Why is high-resolution acceleration more apparent?
The higher the resolution, the greater the scarcity gains:
The 720P single token number is 2.25 times 480P.
The calculation of attention is O (L2), so it is more than 5 times
Redundant computation of rare-leap RoLA also increased with it.
This explains why 720P-14B's 265x acceleration is 140x higher than 480P-1.3B's.
Instant experience
SparkDiffusion is fully open source:
• Weight
✅ Smaller moderation + distillation training code
RTX 5090/H100 reasoning script
• Unified Acceleration Framework (Rare + Distilled + Quantitative Integration)
• Intermodal support (Wan2.1/Wan2.2, T2V/I2V, 480P/720P)
SparkDiffusion: Mitigating the High-Sparsity Trap A Unified Framework for up to 265× Single-GPU Acceleration of Visual Generation
Address of the paper:https://arxiv.org/abs/2609.23153
The main page of the project:https://sparkdiffusion.github.io/
GitHub:
Weighted link:
RoLA's paper:
CrossDistill paper:
Summary and Prospects
SparkDiffusion is moving the acceleration of DiT video generation from "distributed optimization of individual components" to "end-to-end systematic acceleration":
Key contributions:
First identified and named "High Rarity Trap" phenomenon of gradual training failure under extreme rarity
Structural errors at high noise levels accumulate along the trajectory
Step-by-step solution: first dilute preheat to establish a rough precedent, then trace the mixed distillation correction terminal distribution
Unified Framework, the first complete open-source system to integrate scarcity + distillation + quantification
The results of the tests:
RTX 5090 single card: 265 times faster (720P-14B, 18 seconds to 5 second video)
H100: 220 times acceleration (configured, generated in 8 seconds)
VBench 2.0 decreased by only 0.5 points from 60.2→59.77.
The team hopes that this work will provide new ideas for video-generation landing, which will stimulate the production of consumer-grade graphics card video at high scarcity and allow consumer-grade graphics cards to produce high-quality video as well.
Plan for the next step
The team is expanding the framework to:
The accumulation of AR-backed video is the next breakthrough for terminal coordinate monitoring.
Full-mode generating, multi-mode, ultra-long sequence scenarios with scarcity of optimization space.
Follow-up progress on AR expansion, full-mode generation, please look forward.