How can we train big models without reverse propagation?
It's just that reverse propagation has been so successful: in recent years, reverse propagation has been the only mainstream algorithm for training neural networks. All of our architectures, optimizers, and even GPU hardware today, in some sense, have grown around it.
But some people have always believed in this evil: reverse propagation requires that the entire network be microscopic, which is actually a very strong restriction, inseparable modules, discrete computing, external processes, ununified training circuits.
So, if computing is cheap enough, you can simply not "calculate" the gradient and change it with the most primitive idea. You can change the network randomly to see if it gets better or worse, the changed direction is kept, the changed direction is thrown away.
A small research lab called Q Labs recently tried a similar method and named it "Dust".
They claim that this is the first zero-order optimization method to reverse-propagate positive competition on the Transformer pre-training mission.
Its general idea is: not to hit the weight, but to give the activation value of the output to each layer of the activation noise, and to add it independently to each token. Thus each token is equivalent to a "virtual experiment individual" and can simultaneously evaluate thousands of individuals at a time. This is what they call a "virtual population".
Q Labs claims that Dust can propagate in a large population (i.e., significantly larger computing volumes) with good approximation to reverse propagation and even exceed it in multiple settings. They think, "This suggests that we may be able to go beyond reverse propagation in abundance of computational resources".
But the bottleneck is also clear: scale. The author himself says it's a lot more expensive than reverse propagation, and it's cheaper to qualify for "alternatives" at a few quantitative levels. The largest experiments in the paper are only 243M parameters, a training budget of 20M tokens, and the pre-training of 100 billion tokens driven by the industry.
Third-party reproductions from SOTAAZ Lab also provide a calmer perspective for Dust: Dust's results are basically reproducible in comparison to a round of SGD set up in the paper, but it has a small gap with reverse propagation and does not show the obvious advantages emphasized in the paper; if the reverse propagation is replaced by a more suitable optimizer, or if it is allowed to train several rounds on the same data, reverse propagation takes a significant lead, and the computational cost is much lower than Dust. In other words, Dust means that a zero-stage method can be used for Transformer training rather than an alternative reverse propagation at the moment.
Why challenge reverse transmission?
In the technical blogs published by the team, they outlined their research motivations, claiming that this is in essence another tribute to "The Bitter Lesson": As more and more computing is available in the world, those who rely on subtle attribution biases (such as microscopy, reverse propagation, high-level approximation) tend to end up losing out to "common, calcified, violent searches".
Their example is AlphaGo Zero: a version guided by human chess is faster to play, but a version with pure self-control overrides it with sufficient computing power. The authors' analogy is that microalgorithm and inverse propagation may be a "good bias in the low-calculation era": it makes learning more efficient, but in the high-calculation era, it limits the architectural space to be trained.
They further hypothesize that this may explain why neural networks now require massive amounts of data to be generalized: the gradient method does not fully explore the loss landscape, and a search-based, more flexible credit allocation algorithm may be a key step toward better generalization.
So Dust's goal is "crazy": to replace reverse transmission with an algorithm that relies more on violent computation and less on parsing structures.
How does Dust get around reverse propagation?
Dust's core idea is to shift the "weight disruption" of traditional evolutionary strategies to the "activation space" of the Transformer and use the Token dimension to achieve large-scale parallels.
When training ES models, it is necessary to generate multiple models with slightly different model weights, allowing them to carry out forward and forward propagation respectively, and then judge which disturbances should be retained based on loss changes.
Although this method does not rely on reverse propagation, the cost is high. Each member of the population needs to perform the pre-computation separately, and the larger the population, the higher the calculation costs.
Previous EGGROLL attempts have been made to reduce costs through disruptive disruption, but a separate sequence location still needs to be assigned to each population member.
Dust changed its mind: instead of directly modifying the model weight, it added random noise to the output of each linear layer and generated disturbance for each token independently.
Specifically, for the linear layer yt = Wxt, Dust modifies the output to:
yₜ → yₜ + σaₜ
In this case, σ controls the amplitude of the disturbance, and at is the random noise that obeys the standard normal distribution.
In this way, each token is equivalent to experiencing a slightly different model parameter without actually having to copy or modify the weight. The research team calls it "Virtual Population".
It has the advantage of being able to simultaneously evaluate thousands of members of virtual populations along a single forward propagation along the Token dimension, whereas traditional strategies for weight space evolution usually require forward computation for different members respectively.
This is the key to Dust's ability to scale the population and reduce the cost of sampling.
It is not necessary to spread it backwards.
How do you know where to update the parameters?
With disruption, the next step is to determine which disruptions actually improve the performance of the model.
The Dust method is to observe how the loss of each token changes after adding noise. If a disturbance in one direction reduces the loss, a higher reward is given; on the contrary, a lower reward is given.
However, there is a dependency relationship between the tokens in Transformer. Particularly under the Causal Attention Mechanism, changes in a previous Token may affect predictions of multiple subsequent Tokens.
As a result, Dust introduced a credit allocation mechanism with declining coefficients:
In this case, rt represents the disruptive reward of the second Token, cs represents the degree to which the loss of the second Token has improved, andγ controls the contribution weight of the future Token.
Intuitively, a disturbance of a Token can be rewarded with an additional reward if subsequent forecasts become more accurate.
Subsequently, Dust sampled the random noise multiple times with a corresponding reward weighted average, estimated the error signal at the output end of the linear layer, and exaggerated the input with that layer to obtain a gradient estimate for updating the weight:
In this case, jt is an output-end error signal estimated by a disruptive reward, and xt is a linear-layer input.
This process differs significantly from traditional reverse propagation:
Reverse propagation is by chain rule, layer-by-layer computing gradients; Dust estimates the updated direction in forward computation based on changes in the loss caused by the disturbance.
Thus, Dust can obtain the approximate gradient used to train Transformers without performing standard reverse propagation.
It's just that, at the same time, there's so much noise.
How to avoid interfering with one another?
The parallels on a large scale have also created new problems.
If multiple layers, multiple tokens are disrupted at the same time, the loss changes of a particular token can be mixed with other disruptive effects, making it difficult to determine exactly which disruption played a role.
To this end, Dust has designed several targeted optimizations.
First, different types of layers perform perturbation calculations separately and cache uninterrupted intermediate results to reduce repetitive forward propagation.
Second, for components such as Query, Key, Value, and Gate within the attention module, Dust does not directly use Token loss to judge the disruption effect, but instead uses the disruption allocation reward based on changes in the attention output caused by the disruption, as well as an estimated output error signal.
Finally, for the language model output heads, Dust disrupts directly on the cached Logits, only recalculating the related cross-sectional losses, thereby expanding the mass at a lower cost.
The team also used hyperparametric searches to adjust different layers of disruption intensity, sampling ratios, and credit allocation scope.
Ultimately, Dust converted expensive model weight samples from traditional evolutionary strategies into easier parallel Token-level activation disruptions, allowing Transformer to rely on the reward signals in the calculation to update the parameters.
Its key value is to provide a training pathway for big model pre-training that does not rely on standard reverse propagation.
However, whether to gain computational efficiency advantages in larger models, longer training cycles and real hardware environments still needs to be further determined by combining experimental results.
It is not necessary to spread it backwards.
What kind of model can Dust train?
The method sounds ingenious, but the real question is: can it compete positively with mature reverse propagation by disrupting estimated gradients at random?
The research team trained a series of Transformers on the FineWeb data set and compared Dust to the traditional reverse propagation and evolutionary strategy EGGROLL.
Experimental results show that as the population expands, Dust's training effects continue to improve, even exceeding reverse spread under some training budgets.
Dust's advantage over EGGROLL is more apparent. Even though EGGROLL uses 16,384 population members, the effect is less than 64 disruptive Dust samples, 256 times the difference between the two groups.
This suggests that transferring interference from the weight space to the activation space can indeed improve sampling efficiency.
More interestingly, Dust is still able to train effectively when the model parameters are expanded by about 120 times, from 2 million to 243 million.
Test losses of 38 million and 243 million parametric models were lower than the minimum of 2 million parametric models on a larger population scale. Compared to smaller models, larger models could benefit more from increased sampling frequency.
This challenges the traditional impression that zero-level optimization is difficult to extend to large neural networks, and suggests that increasing the size of the parameters does not necessarily result in the failure of random disruption methods.
So how reliable is Dust based on random sampling updates?
The team further compared the gradient of Dust's estimates with the gradient of reverse propagation, and found that the two directions were getting closer as the sample size increased.
Even after the model has been trained for a longer period of time, this trend still exists, suggesting that Dust can continue to obtain useful gradient signals through population expansion.
However, there are still limits to these results.
Currently, the Dust Maximum is validated only on 243 million parametric models, and the maximum data budget for actual pre-training is 20 million Tokens.
As a result, Dust has demonstrated the feasibility of not relying on standard reverse propagation training of Transformers, but whether it can achieve the dual advantage of performance and computational efficiency on a larger scale still needs further verification.
What's up with Q Labs?
Q Labs' research has a clear headline: Will it learn better if it no longer gives the model more data and only gives it more computation? The lab's website places data efficiency and generalization capabilities at its core, and reveals investors including Jeff Dean.
Dust follows this line. The team introduced NanoGPT Slowrun, which fixed the data budget to allow researchers to use the limited data more fully than anyone else; and then proposed q0, which allowed multiple models to explore, learn from each other, and combine their predictions on the same data. At Slowrun, q0 reported data efficiency about 12.9 times that of the reference baseline.
When they got to Dust, they took the question a step further: can more computation replace new gains if the training methods themselves are changed?
The experience of the two co-founders provides a clue to understanding the team. Samip Dahal, a co-signer of Replit's Ghostwriter, published articles and was involved in early AI programming tools; Akshay Vegesna attended the California Institute of Technology and worked with the sensory team at autonomous driving company Nuro to rebuild tracking systems using neural networks. Both have experience in machine learning using real systems and now focus their research on learning algorithms themselves.
It is also worth noting how the team advanced the research: after the paper, following the code and open competition. After the release of Dust, they launched a Noise speedrun that does not allow reverse propagation, allowing the community to continue improving the algorithm around the same benchmark.
From fixed data budgets, to exploring model populations, to challenging reverse dissemination, Q Labs has always asked the same question: Is the way we use computing now far from good enough?
The next step is to see how far it can go.
At the time of Dust's release, Samip acknowledged that it was still far more expensive than reverse propagation. He believes that as computation increases, zero-step methods that rely more on search may continue to improve, even eventually replacing reverse propagation.
Subsequent community optimization did run fast. Recently, Samip announced that Noise speedrun's training time has dropped from about 1.76 hours to 1.6 seconds in 10 days, which is about 4,000 times faster.
This result had a clear threshold: at the time, the 10 million token data set, the verification loss of 4.2-1.6 seconds was 4,186, which represents the actual loss of 4,186 seconds when the specified score is used, rather than the model being fully trained to converge. The starting point of the comparison was the early Dust+Adam program, not reverse propagation.
Samip also pointed out that some acceleration techniques cannot be extended to a larger scale. After running a small scale benchmark, real testing has only just begun.
When the models are bigger, trained more fully, and the target losses are lower, can these updates from random trials still work?
Dust has produced results that are worth continuing to do. The next experiment will tell us exactly how wide this path is.
The link is as follows:
© THE