The next big disaster zone for AI computing could be inside a multi-million-dollar system.
In the past, GPU clusters raised concerns about poor bandwidth, too slow communication. Today, rack-scale systems such as the Atlas 900 A3 SuperPoD, NVIDIA NVL72 rack-scale systems put dozens of GPUs in the same high-speed interconnected area, and the chips, interconnections, switching networks, and entire cabinet systems are jointly designed into a larger computing unit.
High-speed connectivity of GPUs solves the problem of moving data quickly, but whether the computing system can actually run full and not be able to handle ever-changing loads when running. This problem is particularly prominent in large-scale MoE (hybrid expert model) training.
A large number of experts in the MoE model are scattered on different GPUs, with each token routed through the Gate to a few experts. This rare activation mechanism results in the routing of the actual load-dependent token of each GPU. Once a few experts become hotspots, a large number of tokens are concentrated to carry their rank. Other ranks will also wait for the slowest side, even if the calculations are completed in advance.
The load slope caused by the hotspot can also slow expert computation and token all-to-all communication, and may also raise the active display peak of the overload rank. In real life, this imbalance can lead to a gap of up to twice as much between actual throughput and ideal state. More tricky, hotspot is difficult to predict in advance. Experts load updates with input, network layer and routing bias, and current hotspot experts can rapidly cool down in the next batch of data.
Last month, a team from Little Red Book dots infra, together with Peking University and others, presented the UltraEP, which for the first time transforms the MoE expert load balancing into a microbatch-by-level, real-time system capability and places it in the actual production of MoE training.
Previously, representative system schemes such as EPLB were typically based on historically routed data estimates of expert loads, then periodically adjusting expert copies and placement. UltraEP made decisions directly based on actual loads already generated at the current level: Once Gate was completed, it was clear how many tokens each expert received, and immediately completed the copy query before the token dispatch, and the hotspot expert's authority was copied to the reserved redundant slot, and the token was distributed to the proprietary expert (main expert) and copies.
Address of the technical report:
Address of the project:https://github.com/Dots-Infra/UltraEP
UltraEP is even more noteworthy in that it balances real-time experts into a truly critical execution path for MoE, controlling the extra exposure costs of related operations in about 300 microseconds.
On Qwen3-235B, the total cost of the critical path is basically less than 300 microseconds.
UltraEP's value was ultimately reflected in the model throughput, where, on the mainstream MoE of parameters 106B to 671B, the throttle throughput under real dynamic loads averaged 94.3% of the force-balanced ideal performance. Compared to the industry's SOTA training throughput framework (Megatron-LM/SGLang), it rose 1.49 times on average.
The value of high-bandwidth connectivity is further unleashed, allowing expert computing resources to be rescheduled microbatch and layered to particle in operation. For MoE models with a constantly expanding number of experts and parallel scale, such real-time scheduling capabilities could be a key component of large-scale AI systems that make full use of GPU computing power.
How UltraEP accomplished a real-time equilibrium in 300 microseconds
UltraEP's most direct engineering challenge is to complete real-time balancing in a very short time. Any excessive computing or communications expense can offset the performance gains from real-time balancing.
For this purpose, it first deployed a complete EP Group within the same high-bandwidth Scale-up domain, allowing hotspot experts to duplicate power through high-speed in-frame connectivity, avoiding access to lower-bandwidth cross-bandwidth Scale-out networks.
In addition, UltraEP is targeted on both the control side and the data side: the control side uses a GPU-native online solver to duplicate and re-rout token solutions based on the actual load of the current layer; the data side completes the dynamic transfer of expert weights and gradients between different ranks by designing a dedicated communication kernel.
Specifically, a forward-to-real-time equilibrium consists of three closely coupled links: the forward-to-real-load phase in turn generates the equilibrium program and prepares the required expert copy before the token dispatch starts; the reverse phase uses the forward-to-cache program to return the gradient of the copy generated to the native expert.
All of these operations were strictly time-restricted. The results showed that the extra expense of introducing UltraEP in the forward phase was about 300 microseconds, while the extra expense of exposure in the reverse phase was close to zero.
Contrast predictive balancing schemes such as UltraEP and EPLB, including three indicators of load accuracy, decision timing, and operating frequency.
Get the current layer of real load after Gate
As mentioned above, the time of intervention of UltraEP is after the end of the Gate, before the start of the token dispatch. At this time, the system can see the actual number of tokens that each expert in the current layer actually receives. Unlike predicting the next load based on historical statistics, this information corresponds to the actual computing needs that are about to be executed.
UltraEP first collects the routing metadata of the entire EP group to form the current microbatch and overall load matrix of the current layer.
After each rank receives the same load data, they independently execute the same deterministic resolution process on the GPU, generating consistent expert copies and token re-routing schemes without the need for an additional global synchronization.
From time to time, experts have been copying the solutions and the actual transmission costs on key routes.
Quota at the same time decides to distribute expert copies and tokens
Once a real load has been received, UltraEP needs to answer two questions simultaneously: which hotspot experts need to add copies, and how many tokens should be borne for each instance and temporary copy, respectively.
UltraEP introduces quota, which is the token load received by each expert instance after re-routing. By directly solving the quota, UltraEP puts the expert copy and token re-routing into the same optimization process. Each step explores how to both instantiate new expert copies and update the target quota distribution. Subsequent re-routing, according to these quotas, allocates the token needs of each source rank to specific expert instances.
The solving process also takes into account several engineering constraints, including the number of redundant slots for each rank; the same logic expert forbids repeat deployments on the same rank; new copies must obtain a minimum valid quota.
Once the quota is established, UltraEP gives priority to the token using local expert instances of the same rank. After the local quota is used, the remaining token is sent to remote rank according to the remaining amount of other instances, while meeting the load limit. Finally, the quota is implemented in the specific token to determine which expert instance each token should be sent to.
Compared to EPLB+ which uses the same exact load, but is duplicated along EPLB and Round-robin routing, UltraEP decreases the average load imbalance from 1.19 to 1.03, decreases the delay in solving by 27.4%, and decreases the number of redundant expert slots used by 57.9% on average.
Share redundant slot Manage the dynamic expert copy
Once the query is complete, UltraEP already knows which experts the current layer needs to replicate, which ranks to replicate, and how many tokens each copy should bear. The next step is to allocate the weight of the hotspot experts to the target rank before the token dispatch begins.
In order to avoid frequent requests and releases of cache in each microbatch, UltraEP retains a fixed number of redundant slots in each EP Rank. The position of the proprietary specialists remains the same, and the system only needs to rewrite the expert privileges required by the current layer into those cache spaces, forming temporary copies on the other ranks.
Such a layout forms a pair of multiple mapping relationships: each logical expert always has an inherent instance and can temporarily add zero or more copies to other ranks depending on the current load. While redundant slots do not maintain an independent optimizer state, they are mainly used to store the expert copy weight and gradient buffer needed to execute the current load.
UltraEP Expert Memory Management Tip: The system consists of 8 logic experts, with EP4 configuration, each Rank reserving 1 redundant expert Slot.
UltraEP also uses a multi-layer duplication mechanism for redundant slots, where different MoE layers share the same set of redundant slots and buffers. When the current layer is executed, the system rewrites the required expert permissions for that layer into the shared buffer. After the current layer is executed, the group space can be duplicated by a subsequent layer; when the system enters the reverse phase, it redistributes the copy weight according to the forward cached copy scheme.
For example, for Qwen3-235B-A22B, each rank requires 9.9 GB of additional storage if each layer has redundant slots reserved separately. After multi-layer replication, each rank only needs 36MB of weight space and 72MB of gradient space, totaling 108MB.
Streamline communication algorithms accelerate the distribution of dynamic weights
Rack-Scale systems provide high-bandwidth Scale-up connectivity, but hardware peak bandwidth does not automatically translate into effective bandwidth. UltraEP's communication patterns change with microbatch and model layers: the combination of expert, source rank and target rank that needs to be replicated also changes, forming a sparse, dynamic and highly asymmetrical communication chart.
UltraEP designs Persistent Tile Streaming, which cuts the expert's weight or gradient into fixed-sized tiles and then converts the current copy program into a device-end transfer task.
When a tile is writing a remote copy, the kernel begins to obtain the task index for the next tile and perform address analysis and data loading. Task search, dynamic address synchronization and other control expenses are integrated into the data transfer process, reducing the extra time exposure to each transmission separately.
Persistent Tile Streaming improves the efficiency of single dynamic transmission, but when a hotspot expert needs to copy to multiple ranks, the rank of the intrinsic expert can still appear as a pair of multi-send bottles. UltraEP further introduces Chunk Streaming Relay, or split stream relay.
Source rank Based on the sending load in the current replication program, select the lower-pressure part of the device from the target copy's rank as a relay, and distribute the multicast traffic that was originally concentrated at the source end to multiple rank. The relay is relayed to the receiving side of the unit as a chunk, without waiting for the full expert authority to arrive, and without having to set a global barrier between end-to-end sending and relay relay, reducing the gap between phases, etc.
Hotspot expert multicast relay scheme: An expert on rank 0 needs to be copied to rank 1×9, where rank 2, 5, 8 are selected as relay.
In an end-to-end test by EP64-256 experts, UltraEP reduced the imbalance at different initial loads to 1.01.[6] With Adaptive Relay, the weight synchronous delay further stabilized at around 0.19 milliseconds and accelerated by up to about 1.7 times in high-load imbalance scenarios.
Once all of the expert weights are written into the target redundant slot, the token dispatch can begin according to the redirecting program requested by the previous step.
Re-use forward program to complete weight restoration and gradient regression
Upon entering reverse propagation, UltraEP does not retrieve the duplicate program, but directly replicates the pre-stage cached copy layout and token re-routing information.
Because redundant slots are reused across layers and run backwards to the current MoE layer, the UltraEP needs to redistribute the weight of the corresponding expert to the redundant slot according to the forward program to restore the copy to a state consistent with the forward computation. After each copy completes the reverse computation, the UltraEP returns the copy gradient to the default expert's gradient buffer via the Tile Streaming Kernel. Once the gradient buffer is completed, the redundant gradient buffer is cleared and the subsequent MoE layer continues to be used.
In reverse propagation, the redistribution of expert weight and the gradient regression of expert copies can overlap with other reverse computations.
In the reverse phase, UltraEP also controls the number of SMM processors and shared memory space occupied by the communication kernel, keeping enough resources for the reverse computing kernel to run simultaneously. The exposure costs of computing overlapping communications, reverse weight recovery and gradient regression are close to zero.
To this end, UltraEP has completed a complete process of pre-expert copying, Token distribution and reverse copy gradient regression. MoE load balance is not dependent on past statistical results, nor does it need to be adjusted once in a while, so that experts can adjust computational tasks in real time according to the result routed.
Targeting the immature real-time scheduling layer of MoE Infra
Putting the entire process of UltraEP into the entire MoE infrastructure, the role it plays is also clear.
Over the past few years, the field of MoE Infra has gradually developed a set of infrastructure components that focus on different system nodes. DeepEP, as represented by DeepSeek, focuses on the cross-card communication between the MoE token dispatch and the combine to improve the efficiency of token distribution and the retransmission of computational results.
The UltraEP team of the Little Red Book dots infra focuses on another type of fundamental problem exposed after Gate, real-time load bias: how the system immediately adds copies to hotspot experts after a real token load is generated and allocates some of the tokens to a lower load rank.
In both studies, DeepEP optimized the communication efficiency of the MoE token, and UltraEP timed the expert resources in real time.
The Megatron-LM, SGLang and DeepEP-like communications libraries are responsible for the efficient transmission of tokens, while the UltraEP runs independently to handle real-time load balances after the Gate and can be used in combination with existing frameworks and communication components.
UltraEP shows us that layer-by-layer, microbatch, real-time expert scheduling has production feasibility. It has great potential to become a replicable infrastructure component in large-scale MoE systems if more extensive hardware adaptation and framework integration is completed.
The first test of the Dots Infra system's engineering capabilities
From UltraEP, it can be seen that the actual deployment of the system often depends on the ability of multiple infrastructure links to work together. The lack of online solutions, communication algorithms, cache management, framework integration and production verification makes the team's system design and engineering deployment capabilities more demanding.
Thus, UltraEP is just a shorthand of the overall technical strength of the small red book dots infra, which demonstrates the team's ability to break down complex system problems into general infrastructure modules and perform end-to-end production verification.
As the core infrastructure team supporting the Red Book Dots series of models, Dots infra is advancing distributed training stability, low-precision training, RL System, Agentic Rollout, KV Cache management and Tool scheduling.
Including the previously open-source BigMac and the recently released UltraEP, the dots infra team has achieved competitive results in a number of key infrastructure areas. These underlying capabilities ultimately serve a more complete little red book dots technology system.
Dots Studio, which is part of the Dots infra team, has been upgraded by Little Red Book's internal Big Model Technical team and currently covers a full chain of big models such as pre-training, post-training, reasoning, multi-modeling and Agent.
The official address:
UltraEP, as a representative module currently revealed by the dots Infra team, releases a clear signal that the Little Red Book's input into the larger model goes deep into the basics of the system upon which training, reasoning and intelligent body operation depend.