SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering
This is the second edition of the EMNLP 2026 Main Conference.
Link to the paper:https://arxiv.org/abs/2606.00593
Open source code:https://github.com/KhanCold/spader
The model is open source:https://huggingface.co/KhanCold/qwen3-8b-spader
One sentence summarizes: Multi-response questionnaires require the intelligent to look for the entire set of answers, rather than just looking for one to stop. The existing RL method has two flaws in this scenario: advantage computation stays at the trajectory level, credit allocation is rough; F1 rewards the head to the long tail entity, and the intelligent body lacks the drive to explore the long tail. SPADER refines the trajectory level advantage of the GRPO to step-by-step, and uses a two-way incentive mechanism to stimulate the long tail to find that the recall has been fully enhanced with the F1 on four benchmarks and two sets of models.
Question: Multiple answers to questions, finding the whole is harder than finding one
Large-scale language models are moving from "ask a question, answer a question" to "side-answer": the intelligent body generates search queries, reads search evidence, decides when to pick up and gives the final answer. But the current work focuses largely on the task of "only one correct answer". In the real world, the huge demand for information is coverage-oriented, requiring systems to find a complete set of effective answers, which is multi-answer QA.
The correct answer is a set of entities, and the distribution is uneven: high-frequency "head" entities such as Kennedy, Vandenberg, Jiuquan, and PSCA, Spaceport, have long-tail answers hidden in the remote corners of knowledge space and need to be continuously excavated.
Figure 1: Contrast the search trajectory of the average intelligent body under the same question above and below the SPADER intelligent body under the same question. The average intelligent body responds in advance after recovering the weight of the head entity; SPADER captures clues from observation, continuously rewrite inquiries extends to the long tail, and the answer covers significantly more.
The first step is to search for the "Global Orbital Launch Summary of 2021", and get the three main entities: Kennedy, Vandenberg, Jiuquan; the second step is to change the name to the "Famous Satellite Launch Center" and search again, returning either of these entities, with no new findings; and the third step is to answer directly and the answer stays in the head.
The following are the smart bodies trained by SPADER: the first search was the same, but captured from observation the uncovered clue to the "commercial launch", the second actively revised the "commercial launch of 2021", newly excavated PSCA, Spaceport; and then along the "low-intensity launch site" "continued to rewrite queries, excavated long tail (step 3-5), until step 6 was answered, and the set of answers became clearer".
To achieve this continuous exploration, the intelligent body must learn to make multi-step decisions: when to rewrite queries, what evidence to adopt, and when to stop answering. Such decisions are difficult to "teach" out of engineering, so recent work has increasingly used tools to enhance question-and-answer modeling as a strategic learning problem, and to optimize it with reinforced learning.
Two problems arise when training with RL
Question 1: Credit assignment difficulties are heavy on long-term search tracks
Long-range search trajectories make credit assignment difficult: Intelligent entities intertwine to perform effective, redundant and unrelated searches in dozens of consecutive steps, and it's not easy to identify which steps really extend response coverage.
PPO ((actor-critic): relying on a value network to estimate long-term returns at each step, while value estimation errors accumulate over time, credit allocation is unreliable, training is expensive and unstable.
GRPO (critic-free): Remove the critic and improve stability, but Advantage remains on track: the entire track wins, and it's impossible to tell which track is the best in the middle.
Intermediate-step reward methods such as R3-RAG, StepSearch: Ideas similar to PRM score intermediate steps, but rely on external evaluators or process labels, and still optimize on the basis of the GRPO trajectory-level Advantage, which essentially does not provide step-level credit signals.
State matching routes such as GiGPO, SALT and others: The step-by-step credit assignment is achieved through rigorous state matching, but the observation of natural noise in multiple-choice queries varies greatly, and the "precise state matching" assumption itself is difficult to establish.
Problem 2: Frequently misinterpreted reward form with "answer coverage"
Rewards based on F1 look at each entity as a peer, there is no additional incentive for finding the "long tail answer" that is difficult; and there is no signal during the search to tell the intelligent body "to continue searching or the receiver". The result is that the intelligent body is satisfied only with easy-to-recover high-frequency head entities and finds a few obvious answers before terminating.
The method: SPADER
To address these two misconceptions, the authors propose SPADER ("Step-wise Peer Advantage with Diversity-Aware Exploration Rewards"), a reinforcement learning framework for the use of multi-response and long-range questionnaire tools. It contains two complementary components: a gradual peer advantage ("SPA") for the allocation of granular credit, and a diversity sensory exploration reward for preventing premature adherence to head entities.
Figure 2: Overview of the SPADER framework. After sampling parallel trajectories, the framework uses diversity perception exploration to evaluate each step of Search and Answer action; SPA then coordinates trajectories by decision-making step, calculating future returns and step-level advantages through step-wise GRPO co-optimization strategies.
Component One: SPA (Step-wise Peer Advantage), aligned with decision-making, with peer feedback as a baseline.
SPADER builds on the GRPO's classification mechanism: the G-string trajectory is parallel to the same cue word, and the group compares relative advantages to each other. The GRPO provides only the trajectory-level Advantage, while SPA refines the advantage estimate from the trajectory-level to the step-level, specifically four steps.
In the t-decision step, the trajectory set sampled within the group constitutes a baseline with the same prompts, the same interacting sequences, and without assuming semantic or state equivalence between the trajectories. For the trajectory that has been terminated in advance, the maximum length of Lmax in the group is complemented by a future return of the 0 participant baseline, and the effective masquerade is used to block the termination gradient of Mt.
2 Future returns. Using Monte Carlo to estimate the future cumulative discount returns of trajectory i at decision step t:
In the same decision step, the average value μt of the whole group Gt and the standard difference σt are given as the baseline, which gives the step-level relative advantage:
。
Where μt is the same cue word, the same decision step t is the future expectation obtained by Monte Carlo's sampling, the steps of winning that expectation have a positive advantage and the steps of losing have a negative advantage.
4. Training objectives. Replace the trajectory level Advantage in the GRPO goal with At (i), and take important samples and clips step-by-step according to the decision-making process, usually including the KL rule, i.e. step-wise GRPO.
Component 2: Double-axis exploration rewards to make rare discoveries more valuable
It's not enough to just distribute credit: if the reward itself doesn't reward the long tail, the intelligent body still doesn't have the power to go far. SPADER divides the reward into two phases: the double-axis exploration reward in the search phase, and the overall assessment reward in the final response phase.
The reward for each newly discovered valid entity e for each step of the search operation consists of two positive cross-dimensional weightings:
Horizontal information enhancement Ugain (e): Retrieving the basic utility of a valid new entity, defined as a constant 1.
Unovelty (e): a reward dilution mechanism: set N (e,T) for the number of trajectories retrieved within the group to e, and Unovelty (e) = 1/N (e,T), as opposed to the frequency of group-level retrieval.
This design directly aligns the coverage objectives of multiple-choice questionnaires: "head" entities that are easily detected by most parallel trajectories (N≈G) are diluted by severe new concentrations, marginal gains are left with only basic gain α; and unique long-tail findings (N=1) receive the highest reward α+β.
At the same time, each step of the search must deduct the cost of the fixed tool, and the size of the standard answer set. The EGT is divided into a steady gradient. The cost of the tool makes "continue search" no longer a free action: when the expected newbies reward exceeds the cumulative search cost, the smart body should naturally take over.
The results of the experiment
The main experiment: four benchmarks, two baselines leading the way
The paper compares the two baselines of Llama3.1-8B and Qwen3-8B to three baselines: tip type method (RAG, ReAct), result monitoring RL, PPO, GRPO, step-level monitoring (R3-RAG, StepSearch).
Table 1: SPADER compares the Precision/Recall/F1 baseline on four multiple response QA benchmarks with two baselines, Llama3.1-8B and Qwen3-8B.
SPADER took first place in F1 on two baselines and four datasets, with a relative increase in PPO, GRPO to 86.1% on QAMPARI, and a consistent lead on step-level baselines R3-RAG, StepSearch.
Meltdown: Two components are missing
The authors remove the individual components one by one and re-train them on Qwen3-8B:
Table 2: The melting experiment on Qwen3-8B (F1): "w/o" indicates the removal of a component from SPADER; NP, IG, respectively, refers to the new density premium and information gain.
The results show that all three are missing: removing SPA, a steady decline in performance that validates the role of step-by-step credit allocation; removing the new concentration premium, a large area of performance degradation, which is key to preventing early collapse to high-frequency entities; removing the information gain, a strategy that will be driven by new concentration to "rare but useless" entities.
Search efficiency: No over-exploration for coverage
The paper statistics the average number of searches required for each retrieval to a valid entity:
Figure 3: The average number of searches required by each method to retrieve a valid entity (Qwen3-8B)
SPADER's efficiency is still competitive: it recalls lower than its more economical approach, and the "high efficiency" comes in part from catching the head entities and terminating them early; while the GRPO with the highest number of searches recalls the worst, trapped in redundant searches.
Time Dynamics: Continuing Exploration to the End of the Goal
The paper tracks the number of valid entities found on QAMPARI each time they interact with each other:
Figure 4: The cumulative number of different valid entities for each interaction in QAMPARI.
GRPO and ReAct collided prematurely after the first round; StepSearch extended the search to the next round; SPADER maintained the highest cumulative output over the entire trajectory until the effective target was exhausted and entered the platform period: as long as there were new entities to dig, the dig was stopped, rather than looking for a few answers.
Summary
For the multi-response "find all" goal, the existing RL tool for enhanced intelligence exists in two wrong places: credit allocation stays at the trajectory level, rewards are not aligned with coverage. SPADER responds with two components, respectively: SPA is decidedly aligned with the trajectory, refining the GRPO from trajectory level to step level, using the same step-by-step instructions, and interacting with peer feedback as a baseline, achieving non-critical, non-external evaluator credit allocation; diversity sensing exploration rewards for horizontal information enhancement underpinnings; alongside new density premium dilution head entities; combining tool costs with terminal calculations, allowing "when to continue, when to stop" to emerge from the reward structure.
© THE