Extremely Sparse Supervision Incentivizes Reasoning Ability
Link to the paper:https://arxiv.org/abs/2609.04565
X Post：
image
Post-training is a key part of improving the ability to reason about large models, with post-training containing at least hundreds of millions of token gradient signals.A large-scale rollout typically contains several thousand tokens, and if the gradient signal in only one of the tokens is retained to update the parameters, will the training collapse?
A recent study by Amazon and Duke University suggests that under this extremely rare gradient signal, training not only fails but achieves better reasoning performance. The study challenges a commonly accepted assumption:Are token-level signals really necessary in large-scale post-modeling training?
The rich information contained in a token
On-Policy Distillation (OPD, Online Strategy Distillation) is a post-training paradigm that has gained a lot of attention in the industry and academia in recent years, and has become an important technological pathway for improving model capability and achieving efficient capacity migration in front-end model training. OPD naturally has dense token-level monitoring signals: student models generate reasoning trajectories, teacher models provide feedback on each token in the trajectory. This dense token-level monitoring signal has long been considered an important feature of OPD success, but it also increases the difficulty of understanding the mechanisms within OPD.
Using OPD and mathematical reasoning as the starting point, the team conducted an extreme experiment: for each rollout generated by the student model, only one token was randomly retained in the gradient, allowing it to participate in parametric updates while masking all other tokens' signals. In other words, if a rollout contained 4,000 tokens, only 0.025% of the tokens received a supervised signal from the teacher model.Under such an extreme scarcity of signals, the reasoning ability of the student model is still significantly improved.
The team built the Qwen3 series.9 different groups of teachers and studentsIt covers:
1) Small-parameter student model distilled large-parameter teacher model;
2) Teacher and student models of equal size;
3) The large-parameter student model distilled the small-parameter teacher model.
Different settings represent different degrees of representational differences between teacher models and student models. In all combinations, random monitoring of a token can consistently observe improvements in student model reasoning performance.
Extremely scarce surveillance signals are better than intensive surveillance
This involves the selection of the monitoring signal. In OPD, the monitoring signal of each token represents the discrepancy between the teacher model and the student model in the token, with the research team choosing to retain the largest difference in each rollout token.Experimental results show that only one to two token monitoring signals can match or even exceed the entire signal training.
Math Reasoning OPD: Qwen3-8B student ← Qwen3-4B-Instruct-2507 teacher
Another interesting observation is that although the standard OPD aims to reduce the differences between the output distributions of the small student model and the teacher model, under extreme scarcity supervision, the differences between the output distributions of the student model and the teacher model not only do not shrink, but may increase further. This suggests that under the training method of extreme scarcity supervision, students are not merely able to improve their reasoning ability by mimicking the teacher model.
In order to further investigate the universality of this phenomenon, the research team conductedVerification of cross-tasking, cross-model series and cross-post-training paradigmsThe experiments were expanded to: 1) code reasoning tasks; 2) Llama series models; and 3) Proximal Policy Optimization (PPO) based Reinforcement Learning with Verifiable Reward (RLVR).
Coding Reasoning OPD: Qwen3-4B student ← Qwen3-30B-A3B-Instruct-2507 teacher
Analysis of the mechanisms behind extreme scarcity supervision
This work gives a noteworthy data point: a successful post-training can only take the gradient of a few thousand tokens. At this scale, researchers can directly observe the semantic information carried by activated tokens and their impact on learning dynamics. The following table shows some of the activated and positively rewarded tokens (teacher models want to increase their probability).
Additionally, the research team made some interesting conclusions using melting experiments:
1) The intensification of model reasoning performance is not unilaterally related to the intensity of the monitoring signal: as the number of tokens monitored decreases, the model reasoning performance initially decreases, but then recovers; when the monitoring becomes extremely scarce, the performance can even exceed the entire signal training. However, when the monitoring is further scarce to some extent, the learning signal will eventually become insufficient and will no longer be able to bring meaningful reasoning performance improvements.
2) In the OPD experiment, when the student model has sufficient representation capacity, the rewarded token can better stimulate the student's reasoning ability; and when the student model's representation capacity is weaker than the teacher model, the rewarded token achieves better training results.
3) Extreme scarcity monitoring has also resulted in more scarcity of neural network parameters updates, with one-tenth of a million monitoring signals updating only 10% of the network parameters, resulting in significantly improved reasoning performance.
Finally, why is a token's surveillance signal so effective? This is still an open question. The research team gives a analogy: extreme scarcity surveillance is more like a natural human learning process. When people learn a complex task with a certain level of prior knowledge and basic ability, often no finite feedback is required, and a small but critical correction signal can significantly change subsequent behavior and strategy. Similarly, a large model of pre-trained SFTs and RLs itself has a certain reasoning ability; when further trained on it, it may not need to cover every token with intensive supervision, and a small but critical learning signal is enough to guide the model further and bring about significant improvements in reasoning performance.
The author's description:Dr. Xingzi Xu and Dr. Mehmet Saygin Seyfioglu are Applied Scientists at Amazon and Dr. Karim Bouyarmane is Senior Manager at Amazon.