A team of researchers from the National University of Singapore, Cornell AI and the Hong Kong University of Technology has developed DiffusionART, an online Diffusion RL algorithm that generates images and videos. The method supports ODE sampling, without requiring minimal reward models, and without additional input, and with extremely high sample efficiency in various forms of rewards, such as naturally-adaptable metric rewards, binary preference rewards, and more. In HunyuanVideo-13B's visual quality and sports quality experiments, DiffusionART reached DiffusionNFT and Flow-GRPO-Fast's late-term average reward levels of 12.02 and 11.72 times, respectively.
The first author of the work is Ph.D. in Computer Science from the Department of Computer Science, National University of Singapore (NUS), with a major research focus on multi-modal understanding and generation, reinforcement learning, instructional teacher as NUS Principal Young Professor Yuan Teacher.
The article links to:https://arxiv.org/abs/2610.12572
The code link:https://github.com/vanzll/DiffusionART
Hugging Face model links:https://huggingface.co/collections/vanzl/diffusionart
Link to the project:https://vanzll.github.io/diffusionart-project/
Figure 1 shows the training curve and GPU-hour efficiency of HunyuanVideo-13B under equal power VQ/MQ on the left; images and video generation cases of the same prompt on the right.
A: The shortcomings of existing routes: some updates are too slow, some updates are fast but unstable
Post-distributed model training technology is becoming increasingly important in the video generation industry and is mainly divided into two categories:
Reverse-process RL, represented by Flow-GRPO, is closer to the traditional RL paradigm. It models the reverse-noise process as MDP, converting the computational strategy gradient through ODE-to-SDE. However, the ultimate reward is allocated to each step in the trajectory, resulting in poor credit allocation and limited training efficiency.
Forward-process RL, represented by DiffusionNFT, is closer to the supervisory training paradigm of Diffusion. It reinforces the results of generation by returning positive-negative binary branches with reward-weighting, with no transfer probability, and can be followed by efficient sampling. But its stability depends on positive-branch co-training and EMA off-policy strategies that need to be modified by task: single-branch training or direct on-policy updates are easy to break down.
Here is a phenomenon that has not yet been explored in the original Diffusion NFT:
1. In ideal inference, the distribution toward the positive sample is closer to, and away from, the negative sample distribution, and should point in the same gradient direction.
What is the intrinsic difference between on-policy and off-policy in the continuum space RL of Diffusion?
Two, three mechanisms of insight
Starting from the essence and sources of instability of RL prior to understanding, the authors gained three key insights:
Insight 1: The gap between forward and reverse speeds creates a hidden force in the optimization process
The sample generated by the model can be defined by adding a linear noise to the ideal forward speed field, but it is not the same as the reverse speed field used when the model is actually generated.
In addition to reward-improving guidance, the paper finds that there is a forward-facing reverse consistency force produced by two speed field differences (forward-facing and reverse-facing consistency forces). This force does not directly improve the reward, but may produce greater parametric gradients, disrupt reward optimization, trigger gradient spike and even reward collapse. The problem is not the lack of the correct reward direction, but the mixing of additional consistency in the optimization process.
Insight 2: The role of the two branches is to counter this dangerous force against each other.
The reward guidance of the positive and negative branches is consistent, but carries the opposite direction of consistency. Therefore, joint training can partially offset the additional consistency, which explains why individual training branches are prone to collapse.
However, this compensation is not complete. Reward normalization can only ensure the coefficient balance within the sample group and cannot guarantee the consistency of each state. Large residual gradients may remain after different state models are mapped to the parametric space. Thus, the two branches can only mitigate the interference of consistency and cannot completely eliminate it from the target construct.
Figure 2: Left side: The positive or negative branches of separate training collapse, and the combined training performs better.
Insight 3: EMA's key stabilization mechanism is recovery in loss, not just off-policy data.
The EMA also changed two factors: the strategy to generate the rollout, and the old policy as a reference in the loss. Which of these factors brought stability? The dilution experiment found that even if the EMA rollout was maintained, the training would still collapse as long as the old-policy anchor in the loss was replaced by the current policy. Further gradient decomposition revealed: the key role of the EMA was to introduce a restoration force that would pull the current model back to the old policy, not just using off-policy data.
The team further found that even without using EMA, breaking down each round of rollout data into two conflicting minibatches, continuously updating it twice, would naturally generate recovery power, avoiding collapse within the experimental observation range. But the cost was a significant decrease in training efficiency: simply updating the number of model updates of a single rollout from 1 to 2, resulting in a 15-fold decrease in training efficiency for OCR tasks. More counterintuitively, the recovery power in the RMS in the output space was 9.4 times smaller than the guidance, and the gradient range after clustering into the parameter space was 4.9 times greater.
This reveals the extreme gradient conversion effect of recovery power: even if the recovery power of the output space is small, it can overwhelm the effective reward gradient. Therefore, stable training requires Restoration, but too strong recovery power will suppress Reward Guidance, causing training stability to be ineffective.
Figure 3 from left to right: Single NFT updates and double OCR updates; retaining EMA rollout; removing anchor collapse; output power versus parameter gradient scale contrast; how the second MSE update is broken down into guidance and restoration.
Third, DiffusionART: Focus on the rewards direction, and then let the recovery flow.
The previous analysis gave two design goals: eliminating consistency from the target construct rather than relying on cross-sample gradient compensation; and obtaining moderate restorative strength without EMA modification.
Based on this, the research team proposed DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which retains a simple MSE regression interface, but redesigns the way in which the reward direction is constructed and positioned.
1. Aggregation-First: First aggregate rewards and then guide real to noisy states
For the same prompt, ART first calculates the direction of improvement of shared rewards based on a set of generated results and their rewards:
From the point of view of reward tilting, increasing the weight of high reward samples to generate a one-step change in the direction of distribution averages is exactly what differentiates the coefficient.
The shared guidance is then applied to states that have been passed through the real ODE rollout, returning the MSE by freezing the target:
Unlike NFT, the target does not rely on the ideal speed field to be added to the pre-noise, thus eliminating additional consistency from the target structure.
This is the key to Aggregation-First: not introducing guidance with a consistent force in each state, then relying on gradient aggregation to offset interference; it's the first step.