Large-scale model training has already been run on a massive scale, and survival has always been an invisible threshold.
The MoE architecture pushes the big model to the trillion-parameter level and buries a time bomb: high-resolution peak. Once a small batch of tokens is concentrated to a small number of experts, the activation of high-resolution instantaneously rises, an OOM card, and the entire distributed task hangs up directly.
Heavy-computing, activated unloading are common means, but the problem lies in static configuration. Radical configuration typically has an MFU of 30%, peak crashes, and a cold reboot takes 284 seconds; full-level recalculation is stable, with MFU falling directly to 24.6%, and long-term computing latency.
Is it possible for the system to run on its own at a constant pressure, running at full speed when the peak is temporarily encoded, passed and recovered?
The answer given by the team of Prof. Zhang Zhijing of the University of Qingdao is eight boilers 2 ((BaGuaLu-2)), the relevant paper has been accepted by SC26. Adaptation in operation, zero-configuration intervention, valve control computation, expert computation in this window, the system selects to save, recalculate or unload, respectively, the different calculators based on real-time decisions.
According to the Wankas Center for Computing: 128 to 12,288 DCUs, 188B to 1.04 trillion parameters, with an average throughput of 1.14×, a total throughput of 2,012 EFLOPS under 12,288 cards, with an average MFU of 32.9%, up to 35%.
Link to the paper:https://arxiv.org/abs/2610.23064
A: Background of the problem: high-resolution pain points in MoE training
If you've trained a MoE model, you've probably encountered something like this: the training runs for hundreds of generations, and it's pretty steady, and suddenly a small batch of tokens flows into a few experts, activating instant memory, an OOM card, and the entire distributed task hangs up straight.
The problem arises in choosing your own mother.
The core of MoE is that routers are executed by a select few experts for each token. Modern load balancing methods can suppress long-term expert congestion but cannot predict the specific content of the current batch. Test data: 97.1% of iterations have an expert load imbalance of less than 1.2x, but 2.9% of abnormal iterations have a maximum of 2.6x. This is the very few iterations that repeatedly interrupt the training process.
Water pipes make the problem worse
The preheating phase of the flow stream is only forward, with no reverse release activation, with multiple small batches of activation simultaneously residing on the device. If one of the batches routes a large number of tokens to a small number of experts on the current device, additional MoE activation is superimposed on the residing activation, and the local slope is directly converted into the device-level OOM.
The difficulty of static configuration
Recomputing and uncomputing are common tools, but the traditional practice is to write down configurations before starting. In the extreme configuration normal phase MFU is 30.0%, but the peak must be OOM, with a cold restart of 284 seconds; the conservative configuration MFU for full-level recomputing is reduced to 24.6%, and the algorithm is long overdue.
Figure 1: MoE load peak pushes up activated cache, and static strategies have to switch back and forth between OOM restart and maintenance operation.
Design thinking: Adaptation when running, zero-configuration intervention
The core idea of Boiler 2 is straightforward: the storage pressure is changing, and the strategy should be changed as well. Intervention should be reduced when steady, maintenance of high throughput, temporary strengthening of mitigation at peak times, and automatic recovery at peak times. The boiler does not terminate tasks, modify configuration, or restart.
Figure 2: The steady-state phase uses lighter reflection attenuation, temporarily increasing intensity at peak arrival, and the reflection curve is always controlled within the device's upper limit.
1 Activated cache modeling: scalable cost model
The system puts the saving activation, recalculation, and unloading into the same cost model, while estimating the marginal savings and time expenses, and considering whether additional operations can overlap with the original computational communications. Unloads of the same size, occurring at different flowline locations, and exposure times on key paths can be completely different.
2 Self-adapted supply: Algebraic level precision control
The decision points are selected in a key window: after the MoE gate control calculation is completed, before the expert calculation begins. At this time the real load is known, the risk has not yet been settled, and there is still time to temporarily strengthen the recalculation or unload. The strategy is no longer a whole unit, with different calculators choosing to save, recalculate or unload respectively, and only the minimum combination that meets the budget. The system also tracks the activation sources, excluding those with different accuracies of recalculation, one unloading another remaining redundant combination.
Figure 3: The system completes activated cache modeling, adaptive supply and dual batch fusion execution, depending on the current flowline status and MoE load.
3 Dual batch mergers: dynamic planning scheduling, spending neglected
Re-computing and unloading are not free, and the key path is directly through throughput-switching security. Eight boilers 2 use the flow line gap. The steady stage equipment simultaneously exists in the current forward direction, current reverse direction, front-to-front unloading and next reverse recovery. These operations belong to different sub-groups, and are executed interchangeably depending on the sequence. The system uses lightweight dynamic planning to re-order computing flows and communication flows, clogging additional work into hardware vacuums; continuous calculators merge, budget state cache, most iterate direct re-modulation results.
Figure 4: First, by using dynamic strategies to squeeze down the display peak, then swap to schedule adjacent batches, and hide additional computing and data transfer in the hardware vacuum bubble.
3 Testing effects: Multi-scale verification, open source code
Full-scale performance leading
In one ICU, the model ranges from 188B to 1.04 trillion parameters and clusters from 128 to 12,288 DCU (BW1000). Compared to Megatron-Kuai and AdaPipe, the average elevation of 1.14× and 1.13× (with OOM reset loss) is 12,288 cards with a total throughput of 2,012 EFLOPS, an average MFU of 32.9%, up to 35%.
Figure 5: From 128 to 12,288 acceleration cards, eight-cylinder boiler 2 maintains a lead throughput across all test scales.
Dissolving experiments
Starting from a full weight computation baseline: fine grain activation strategy 1.16× → + dynamic management 1.22× → + overlapping scheduling 1.38×. Each module has clear benefits.
Figure 6: Dimensional activation strategies, dynamic memory management and overlapping scheduling are superimposed on each other, resulting in an overall gain of 1.38×.
Open source and fast
This is the first time that I've been able to do this.
Open-source content: Activate the cache analytics model, activate the core code such as unloading and reloading of the router integrated
Low Threshold Integration: Provides adaptive sampling and neutral assessment benchmarks, conveniently integrated into existing training flow lines
Zero-configuration use: open-box use, supporting scientific training and rapid verification
Welcome to submit Issue and PR and participate in technical co-creation. The warehouse public domain does not contain the full runtime mechanism described in the paper, and the core modules are now available for use independently.
Four: Teams and communities
Co-developed by a team led by Prof. Xi Jinping of Qingdao University, Qingdao University of Technology, Beijing University of Science and Technology and independent researchers, the team is committed to open-source co-creation of large model training and reasoning systems, and welcomes developers and researchers to explore cutting-edge technologies together.
Red Hat reasoning engine:https://github.com/thu-pacman/chitu
The training system for the eight boilers:
The team's assessment technology is also used to control energy consumption in smart computing centers, making the financial cloud effect even more apparent when more domestic computing is used.