North Big Open Source, Strengthened Learning Data Strategy for easier, more equitable access to data

  It's the first time in the history of the world.

What data should each round of reinforcement be given to the model?

This problem is becoming increasingly important with the development of reasoning models and RLVR. As reinforcement learning's role in post-model training increases, it is also necessary to arrange more carefully what and how to practice the model. A question may be difficult at first, but it can be solved after some time of training.

While there is often more than one data strategy method, there are also some practical difficulties in how to compare different methods. There is a set of implementations for different methods, and a different selection rule may change the trainer; when comparing results, the model, reward function and training budget are not necessarily consistent. Even with higher scores, it is difficult to determine whether the benefits come from data strategies or other training settings.

In order to solve the problems of data access and comparison of effects of data strategies,North Korea's DCAI team joins UCAS, ShanghaiAlgorithmsDataFlex-RL is the latest release from the Institute of Innovation, Guangdong University of Technology。

GitHub is an open source repository:

Address of the paper:https://huggingface.co/papers/2609.06107

Less time spent revising the trainer repeatedly, and more effort to verify the effectiveness of data strategies is the most direct value of DataFlex-RL. 

DataFlex-RL uses verl's training process to configure data selection, sample weighting, and domain allocation adjustments into configurable components, allowing developers to configure three types of actions in the same set of RLVR/GRPO processes:

• Select which rollouts participate in the current update;

• How much learning weight is assigned to a sample or token;

• Decide which areas to sample the next batch of data from.

These strategies directly replicate the rewards, advantages, tokens, probabilities, and prompt clusters that have been generated by the RL cycle, and the signal, rollout, verification, and policy update code remain largely unchanged.

The paper gained community attention after its release and was featured on the HuggingFace #2 Paper of the day. The paper organized 591 experiments to test data strategies on a variety of models and mathematical, logical, and scientific tasks.

In GRPO training, the model generates multiple responses to the same question, is rewarded by the verifier, and updates the model based on the group's reward difference. DataFlex-RL divides the data strategy into three categories: selecting which responses participate in current updates, assigning the authority to adjust their contributions to updates, and mixing up areas that change subsequent batches based on historical feedback.

The answers generated by the same question may provide different learning signals. In a GRPO based on group reward differences, a group of answers is either entirely correct or entirely incorrect and usually cannot form a differentiated advantage.

DataFlex-RL supports the screening of outcomes, advantages, rankings, rewards and response efficiencies within groups.

difffilterA specific example is that each question generates five answers, with only the resolution rate strictly between 0.2 and 0.8, i.e. in the case of two or three answers, the update will focus on the middle difficulty of the problem.maxvarThe largest subset of rewards within the reserve group,gfpoThe results are based on the ratio of the length of the reward to the answer.topkKeep the advantage value ranked according to the previous answer.

Here, the responses that have already been generated are filtered out, which are currently achieved through the 0/1 weight control strategy loss, so that there is no reduction in the production costs that have already occurred.

Sometimes researchers wish to retain all of the answers in the current batch, while allowing some samples to assume greater update weight and overweight for this situation.

DataFlex-RL supports response level and token level weights:softmax、perAccording to the Advantages Adjustment, the contributions are adjusted according to the advantages.diffbandIn addition to the weighting of samples within the designated allocation range,arUsing pre-existing token probabilities, a relatively low weight is applied to low-probability tokens.

These methods reduce the weight average to 1 to control changes in the overall loss scale as much as possible.

The learning progress of mathematical, logical, and scientific tasks may vary, with fixed ratios not necessarily appropriate for each stage. DataFlex-RL uses a sliding window to accumulate training feedback in different areas and then update the target ratios for subsequent batches by the mixer.

The mixed strategy will first accumulate feedback during the warmup phase and then update the interim ratios according to the configuration. The current plugins are selected by field from available samples via a custom replay buffer, and the actual access to the training data is also limited by the number of candidates in each field, so it is necessary to observe both the target ratio and the actual data distribution.

All three types of strategic roles require access to training processes, reading feedback, and translating judgments into specific operations.verlPlugins are trained to access and reuse already existing signals, and will be separated from execution so that the strategy components can be replaced independently.

To compare different data strategies, it is first necessary to have them run in the same training process. DataFlex-RL accesses verl as a plug-in and provides its own trainer through the registration mechanism.

For selection and overload, the plugins read signals after the original advantage calculation is completed, run data strategies, and then write the results.verlIt's already there.rollout_is_weightsFields. Selection results are converted to 0/1 weight and the weight-added strategy returns continuous weight, both of which are processed by existing strategy loss interfaces. Fields are mixed using separate trainers and custom replay buffers, with cumulative feedback and available samples selected according to the proportion of updated fields.

Different strategies share model generation, reward verification and model update processes, and change the focus on the location of access to data strategies.When new methods are added, developers can use these training capabilities to focus their main work on the strategic components. The current plugins are directed to the verl v1 trainer interface, which requires verification of the compatibility of the host version before use.

After entering the training, the strategy also needs to be judged on the basis. The reward reflects the response performance, the advantage provides the strategy update signal, the token log-prob record token probability. These data are generated with the training, and the scorer can directly read and calculate the required scores.

The strategy does not require the same signal processing code to be used in order to execute an additional model forward. Sharing signal source and computational logic does not require all strategies to use the same score; field mixing also requires cumulative field-by-field feedback into historical statistics.

The training process also records retention ratios, weight statistics, field goal ratios, and feedback, making it easy to check that strategies are running as expected.

The next step is to decide how to use it. DataFlex-RL gives the two steps separately.ScorerandActuator。ScorerDepending on the input signal calculation score, the executor converts the score into a conservation index, continuous weight, or domain ratio.

If you have a score, then it's up to you to decide how to use it.topkIt can also be screened and givensoftmaxWeight loss is continuous.

Components are registered by name and then by configuration combination: usually only execution rules are implemented based on existing signal-added screening methods; scorer is extended when a new judgement basis is required.

The paper carried out a total of 591 experiments covering different basic models, as well as mathematical, logical and scientific tasks. The core experiments were conducted on the Qwen2.5-7B-base, comprising 13 configurations, each using 12 paired random seeds, completing a total of 156 operations. The experiment first confirmed that the GRPO was able to train the model effectively, and then observed how the three types of data strategies were involved in subsequent training.

On the Qwen 2.5-7B-base, the average score for untrained models was 42.01, up to 49.77 after standard GRPO baseline training with uniform sampling.7.76 pointsOn the Lama-3.1-8B-base, the model was also upgraded from 10.87 to 21.12 on average.10.25 points。

This set of results established a reference system for subsequent experiments: GRPO brought added benefits to basic training, data strategy continued to arrange responses, learning weight and field comparisons.

Selection and overloading experiments coveragedifffilter、maxvar、gfpo、topk、ar、per、softmaxanddiffbandThey adjust training signals in response to screening, sample weights and token weights, and compare them under the same model, data and training budget.

In mixed field experiments,reward_gap、dump_ucbandtsclThe average improvement of the three approaches is 0.45, 0.61 and 0.16 percentage points, respectively, compared to the fixed and mixed percentages, showing how the field is allocated according to the training feedback adjustment.

In Llama-3.1-8B-base, the paper continued to test selection, overweight and field mix configurations, verifying that the same set of components can migrate to different models and follow the same training and evaluation processes.

In addition, Table 8 complements this comparison from the evaluation side: when covering the three fields of mathematics, logic and science, the results obtained by different complete summarizations are more approximate overall; the order of methods also changes after adjusting the scope of evaluation coverage.

These experiments demonstrated the close loop of DataFlex-RL's work: read signals such as rewards, advantages and token probabilities during training, dynamically adjust the sample selection of the current batch to the learning weight, or update the field allocation of subsequent batches; verify by configuring switching strategies, using the same set of training and evaluation processes, record execution processes and compare results.

DataFlex-RL provides not only a dynamic training framework, but also an effective verification interface that allows researchers to observe the actual role of data modeling in different models, tasks and training stages.

From the perspective of the larger AI+Data ecosystem, the DCAI team has previously been open source DataFlex and supports seamless integration with LLaMA-Factory.Select、Mix、ReweightandReorderDataFlex-RL extends this set of flexible data processing ideas further to the RLVR/GRPO training process, based on training feedback dynamic scheduling rollout and field data. The two sets of tools complement each other, covering more links from data preparation to reinforcement learning updates, making data strategies more convenient to the actual training process.

Dr. Liang Liu, Ph.D. from the Center for Research in Big Data Science at Peking University, has received a National Scholarship, two consecutive years as President of Peking University, and is the first author to publish 10+ CCF-A papers/journals.

He led the open-source project design development of the Data-Centric AI series, which garnered tens of thousands of GitHub Stars, with DataFlow winning the ICML SeePhy Competition and the Intelligent Source LIC Challenge. He led the team in developing the data module design for the Camel and LAMAFactory projects, earning 17k+ and 74k+ stars, respectively.

The DCAI team at Peking University focuses on big model data systems research and the construction of data-centric AI infrastructure, open source data flow, dataflex, one-eval, 3AGameFactory, and openworldLib.