The work presented in this paper was carried out jointly by the team of Prof. Li Chongqing and Prof. Wang Rejun of the Chinese People's University of Guangzhou Institute of Artificial Intelligence.
The rapid development of widespread language models is becoming a technical direction of great interest in the field of big language models: thanks to the unique advantages of dual upstream context modeling and multi-word parallel decoding, this emerging model has grown from a pioneering academic exploration into one of the important technological routes of big language models, and is constantly receiving attention from academia and industry. As the scale of models and training algorithms continues to increase, the question of how to scale efficiently becomes a pending answer to the question of how best superparameters can be adjusted, how limited computing power is allocated between models and data, and how sparse expert architectures evolve, there is a need to extend the laws to science.
In the regression model, the law of expansion is quite mature, supporting the training of generations of larger models; however, discrete scattering language models use masking to noise targets, monitor signals only in the masked position, and each prediction is based on a damaged sequence rather than a causal forecast, and the experience of regression expansion cannot be directly illuminated. Mixed expert architecture is the mainstream choice for large model expansion, with the scattering rules combined with scattering models still lacking systemic engraving, and the training of large-scale scattering language models still relies heavily on experience and trial errors.
For this purpose, the team of Li Chongqing, Wen Reun and Xiang Group for the first time systematically sketched the expansion rules of the hybrid expert dissemination language model and trained the new generation of 30B-A3B's dissemination language model LLaDA MoE v2 from scratch. The work will extend the rules through superparameter configuration, algebraic allocation and architectural design, as well as the dissemination language model from the simulation experiment drive to the key step of the simulation index: LLaDA MoE v2 activates about 3B input, and returns about 65% of the pre-training numbers of the same-scale autogenous model Q3 30B-A3B, i.e. in coding, coding, etc.
The title of the paper is: LLADA MOE V2: SCALING MIXTURE-OF-EXPERTS DIFFUSION LANGUAGE MODELS
Link to the paper:https://arxiv.org/pdf/2608.03457
The team expects to weigh the open-source logic code and model LLaDA MoE v2 in the near future.
In terms of optimizing superparameters, the combined team matched system search volume sizes and learning rates at model sizes from 158M to 3.6B with computational ranges up to FLOPs to match the superparameter expansion laws of distributed language models: the optimum volume size increases faster than the regression model, and the optimum learning rate decreases faster than the optimization. In FLOPs, the optimum volume size predicted by the regression law is about 102 million words, while the new law gives 343 million words.
Figure 1: Superparameter Expansion Laws of LLaDA MoE v2: Optimal Batch Size ((left) with Optimal Learning Rate ((right) with Expansion Curve of Training Calculator, Red Viral Lines as the Adaptive Laws and Blue Viral Lines as the Reference Laws of the Self Returning Model
In terms of arithmetic allocation, the team sketched the optimal balance between model size and training data under fixed arithmetic using IsoFLOP analysis: the optimum model side arithmetic increased by 0.475 times the total arithmetic, the optimum number of training words increased by 0.525 times, and the overall approach was balanced and slightly biased towards the data side. Compared to the expansion laws of the already rigidly distributed model, the distribution front of the mixed-specialist spread model is the most biased data side to date, with the rare-activation and diffusion model's relative data streams showing a tendency to overlap in them, which means that the marginal arithmetic injection training words are more computational than the projected model side.
Figure 2: Expanded laws of arithmetic allocation in LLaDA MoE v2: Model under fixed arithmetic - IsoFLOP curve of data allocation (left) and the best model side arithmetic (middle) with the best trained number of words (right) along the arithmetic forefront of total arithmetic
In architectural design, the team analyzed the laws of expansion of mixed expert architecture in three key dimensions along activation ratios, expert grain ratios and shared expert ratios: as the scale expanded, the lower activation ratios, i.e. the larger routing pool of experts became more advantageous; the average expert grain ratios of 8 to 16 remained steady at all scales; shared experts carried about one-third of the activation capacity (33.3%) at all scales and remained constantly optimal with the 25% or even no shared expert experience used by the returning mixed expert models, thus gaining the rare loose architectural design guidelines for distributed language model specialists.
Figure 3: Laws of expansion of mixed expert architecture: Activation ratio (a) Expert particle ratio (b) Shared expert ratio (c) Controlled scanning at different computational scales, Red Star marking the optimal configuration of each scale
Under the guidance of the above extended laws, the combined team trained the LLaDA MoE v2 30B-A3B model from scratch: total parameter 30B, activation of 3B parameters per parameter, using 128 particle path experts, activation ratio 9.09%; expert particle 8; expert share ratio 33.3%, in the optimal range given by the extended laws. The model learned 23.5 million T words in five stages of pre-training, consuming about 46 NVIDIA B200 GPU hours. In 15 benchmarks tests, the LLaDA MoE v2 achieved the highest average distribution of 58.60 in a distributed language model, compared to the same-scale SDAR model at 3.78%, whereas the HumanE and Big Bench code received 16.46 calendar units; the QE and the QE models received only about 65% of the back-to-basics, compared to the QE models, which were based on the same-scale Sci-Bench models.
Figure 4: Comparison of the LLaDA MoE v2 baseline model with other distributed language models and autoreference models on a 15-point benchmark
Compared to the previous generation LLaDA-MoE 7B-A1B with no extended rule guidelines, the LLaDA MoE v2 achieved overtaking on MMLU, GSM8K, and KorBench with only about half the training capacity, demonstrating that the LLaDA-MoE can convert pre-training calculations efficiently into downstream performance paths.
Figure 5: LLaDA MoE v2 with extended rule guidelines achieves and exceeds the previous generation of LLaDA-MoE with less training on multiple benchmarks
In supervised modeling, the team used 7 million instructional data to train the resulting model, which surpassed the same-scale scattering model SDAR Chat on seven of the eight mathematical logic and code generation benchmarks, and approached Qwen3, which underwent additional reinforcement training at AIME 2024/2025, MBPP, LiveCodeBench, and reversed on MultiPL-E, a multilingual code generation benchmark that demonstrates the ability to release complex instructions following complex reasoning abilities.
Figure 6: LLaDA MoE v2 instructs the modeling of thinning with Qwen3 and SDAR Chat to contrast core indicators in reasoning with code benchmarks.
LLaDA MoE v2 for the first time established systematic expansion laws covering hyperparameters, arithmetic allocations and architectural design for mixed-specialist distributed language models, breaking the bottleneck of non-self-return model scalability relying on experience migration. Through law-guided lean architecture and masking noise-de-noise training, models with fewer pre-training vocabularies and lower activation costs led to the return model of nearly the same scale, providing theoretical guidance for scalability of distributed language models, paving the way for further expansion of model scales and exploration of intelligent boundaries.