In Agentic RL post-training, the model not only generates text, but also invokes external tools such as search, code execution, and returns logic based on the environment. This interaction gives the model a stronger operational capability and allows the training system to cope with a more unstable workload than an ordinary RLHF: the same set of requests can produce tens of times different lengths of trajectories, with a small number of over-long trajectories delaying the entire rollout; at the same time, training and rollout demands for GPUs also change as the strategy evolves.
In response to the above, a team of researchers from the Chinese University of Hong Kong and Hong Kong's Hong Kong University of Technology has proposed Libra, a resource management system aimed at Agentic RL Post-Training. Libra no longer sees rollout as a fixed bottleneck, but instead optimizes training and rollout as a combined system, and allows limited GPU resources to flow dynamically with real-time workloads through heterogeneous logic clusters, causal sensing scheduling and flexible resources switching.
On 48 NVIDIA A800 GPUs, Libra achieved the highest throughput, reaching 3.0 times the baseline in all three categories of Search-R1, Dapo-Math-17K and R2E-Gym; with the near-final reward, the maximum time required to reach the target reward was 1/2.5 to the baseline.
Libra: Efficient resource management for Agentic RL post-training
Link to the paper:https://arxiv.org/abs/2606.03077
Open source code:https://github.com/NetX-lab/Libra
Figure 1: Overview of the Libra system. The system consists of global resource planners, heterogeneous rollouts, clusters, C-MLFQ schedulers and flexible execution mechanisms.
Agentic RL brings more than just a longer output cylinder.
A post-standard RL training iteration typically involves trajectory generation, trajectory evaluation and strategy updates. Due to the relatively light weight of the evaluation phase, system efficiency depends mainly on two factors: how quickly the rollout produces trajectories, and how quickly the training absorbs these trajectories and updates the strategy.
In traditional reasoning, the length of the request is usually strongly correlated with the input prompt; however, in Agentic RL, the length of the trajectory is changed when the event occurs. For example, search tools may return large chunks of content, code execution may fail and trigger multiple iterations of repairs, and models may be further reasoned based on environmental feedback. Thus, the final length of the trajectory is difficult to predict reliably before generation.
The research team observed on R2E-Gym that the longest 10% trajectory occupies more than 50% of the rollout time. More importantly, this distribution is unstable: as strategies gradually change in training, the way tools are used and the length of reasoning of the model also shifts.
This drift increases the structural differences between the rollout and the training. Experiments show that when the sequence length increases from 1K to 32K tokens, the rollout delay increases 95 times, while the training time increases only 3.9 times. The reason is that the rollout requires self-return decoding and is more sensitive to the sequence length and KV cache; training can be affected by changes in the thinness of the length by batching.
Figure 2: (a) As the sequence grows longer, the rollout delay grows much faster than training; (b) During training, the average sequence length and rollout time continue to drift.
This means that a static GPU that is reasonable at the beginning of training can become seriously imbalanced after hundreds of steps. If the rollout slows down, the training GPU will wait for new data; if the training slows down, the trajectory generated by the rollout will bounce back into the queue.
T_iter = max (T_rollout, T_train)
As a result, the problem cannot be solved by continuing to optimize the rollout, but by seeking a balance between training and rollout from a global dynamic perspective.
Global resource planning: how to train and rollout GPUs at the same time
Libra's first core design was Global Resource Planner (global resource planner).
How much GPU is allocated to training, how much is allocated to rollout;
What combination of TP, PP, DP, and EP of the MoE model is used on the training side;
How many TP-1, TP-2, TP-4 or TP-8 reasoning instances should be deployed on the rollout side;
Current configuration training time, rollout time and final iteration makespan.
The training side uses topology-sensitive decision trees to list feasible parallel strategies, and cuts in advance according to constraints such as display storage, communication expense and pipeline bubble. The rollout side searches the requests by historical length to find the optimal distribution between heterogeneous TP instances and request areas using dynamic planning. The bottom Cost Evaluator simultaneously models the execution times on both sides, allowing the planner to compare different global configurations.
Planning is not only a one-time operation at startup. Libra reads the latest trajectory statistics periodically to retrieve resource allocations; it is only when the expected return exceeds the cost of retrofitting that resource movements are actually triggered. This can both track workload drift and avoid the turbulence of frequent switching.
Elastic Hybrid Pool: Not to rebuild a core communications group, but to move computing
In traditional distributed training, adding or removing a trained worker often means rebuilding a communication group and redistributing a state, often at too high operating costs.
Libra divides resources into three pools:
Core Training Pool: Maintain a fixed training ground;
Core Rollout Pool: assumes the creation of stable isomorphic trajectories;
Elastic Hybrid Pool: Switch between rollout and training modes depending on the bottle.
Its key principle is to keep the core training topology unchanged. The hybrid worker is incorporated into the complete data in parallel copy form without altering the core TP/PP structure. The system further unravels the gradient switching between NCCL communication within the copy and the copy, with member changes occurring only in independent cross-copy communication domains.
When the rollout worker re-joins the training, it receives asynchronous access to the latest models and optimizer snapshots. During recovery, the core training continues to progress; the worker in the rollout sends a zero-level positioning through a side channel, so that the core All-Reduce remains mathematically equal to the worker who has not yet joined the rollout.
Do not predict the final length, and cause routing when the tool returns
The second core design of Libra is the C-MLFQ (Causality-Driven Multi-Level Feedback Queue).
The traditional length prediction method attempts to guess the final length at the beginning of the request, but the key changes in Agentic RL often occur in the middle. Libra observes that tool resizing, success or failure is not an ordinary related feature, but a direct causal signal of subsequent trajectory expansion. For example, a large payload will immediately add to the context, and tool failure may trigger retesting, diagnosis, and code modification.
C-MLFQ uses historical trajectories to create a causal-perceived forearms tree. The tree nodes are determined by the prompt ID and the sequence of returns of all previous tools, and preserve the remaining length distribution from the current node to the end of the trajectory. The runtime process is divided into three steps:
Ask to enter a small TP bucket suitable for short sequences at the beginning;
Each time the tool is returned, the forward trunk is query based on the tool type, payload size and execution status;
Only when the mean of the remaining length with P90 points to the same bucket is migrated does the remainder remain in the current bucket; the tree is updated offline after the trajectory ends.
Figure 3: C-MLFQ reads the causal state at the return point of the tool and decides whether to migrate the request, then updates the foreshore tree after completion.
This design avoids additional model reasoning and does not require repeated training length predictors as the strategy changes. Compared to the traditional MLFQ, the C-MLFQ can make a one-time decision earlier when the tool returns.
On Search-R1, C-MLFQ's single-route accuracy was 91.1%, significantly higher than embedding-based length prediction methods (65.2%) and traditional MLFQ (44.8%); at the same time, its migration token ratio was only 8.2%, and system throughput reached 2700 tokens/s.
48 end-to-end results on the A800
The team experimented on a total of 48 NVIDIA A800-SXM 4-80GB GPUs across six nodes using NVLink/NVSwitch and 200 Gb/s RoCE network supporting GPUDirect RDMA between nodes. The experiment used GRPO, with a maximum model length of 40,960 tokens, and each prompt sampled 16 trajectories.
The workload covers three distinct Agentic RL scenarios:
Search-R1: The model needs to generate multiple rounds of search queries and use external knowledge;
R2E-Gym: Software Engineering Agent operates real code warehouses and calls tools such as Bash, Python, and others;
DAPO-Math-17K: Contains 17K path competitive level mathematical problems.
Comparisons include verl-Colocated, verl-Static-Uniform, verl-Greedy-Heuristic, and AReaL-Static-Optimal, which selects the best static configuration based on the initial workload.
Figure 4: Libra throughput and reward matching results on Search-R1, DPO-Math-17K and R2E-Gym. The red line is Libra.
On Search-R1, Libra's average throughput was about 2700 tokens/s, up about 63% from AReaL-Static-Optimal, up 80% from verl-Greedy-Heuristic, up 300% from verl-Colocated. On DAPO-Math-17K and R2E-Gym, Libra also maintained the highest throughput.
Since each method trains the same model and performs the same number of steps, they ultimately achieve similar rewards, with the difference being mainly wall-clock time. Libra completed the training on Search-R1, Dapo-Math-17K and R2E-Gym in 17.9, 26.7 and 63.2 hours, respectively, to achieve the maximum speed of 2.5 times the target rewards.
In R2E-Gym, the Static-Uniform baseline is 423 tokens/s; after adding 510 tokens/s to the simulation resource planning, the simulation TP brings 41 tokens/s, C-MLFQ adds 115 tokens/s, and ultimately the resilience execution continues to add 97 tokens/s, bringing the full Libra to 763 tokens/s, up about 80.4% overall.
Figure 5: Steady-state separation is gradually incorporated into global planning, asymmetrical TP, C-MLFQ and throughput changes after elastic execution.
System implementation and open source
Libra contains about 130,000 lines of Python and C++/CUDA code, the core RL training loop is verl-based, the generating side uses vLLM, and the training side uses Megatron-LM. The open-source warehouse provides Slurm and non-Slurm Quick Start, data preparation, configuration references, observational descriptions, and experimental scripts.
The core view of Libra is that in Agentic RL, rollout is not a perpetually unchanging system bottleneck. As strategy and trajectory distribution evolve, the real bottleneck moves between training and rollout. Only by addressing cross-stage resource allocation, interstage structural execution, and low-cost resource switching can the system remain close to optimal in the long-term training process.
Author's note
The team of authors, composed of Chen Kewin, Liu, Li Qichong, and Xu Hong, comes from the Chinese University of Hong Kong and Hong Kong's University of Hong Kong, and has long been engaged in AI infrastructure, machine learning systems, resource scheduling, big model reasoning, distributed training and computer networking research. The first author, Chen Kewin, is a PhD student at the Chinese University of Hong Kong and the team's results have been published in international conferences such as SIGCOMM, ASPLOS, NSDI, ICML.