When large models do In-Context Learning (ICL), they typically retrieve several demonstrations from the sample library and put them in a prompt along with the current problem.
However, the recovered Top-K is not necessarily suitable for the target model. Too many examples will increase the cost of context and reasoning, and unrelated or repetitive examples may also affect the model's judgment on the current problem.
The number of combinations will increase rapidly if the optimal k examples are retrieved directly from the pool of candidates; some model-aware methods also require repeated calls to the target LLM to evaluate different demonstration sets.
Researchers in the Department of Automation at Tsinghua University have proposed Local Demonstration Editing (LDE) and trained a small editing model Jev-LDE. It is not responsible for the final answer, but allows only three operations: Keep, Delete or Replace.
In the experiment, Jev-LDE outperformed the Semantic TopK initial outcomes in 46 of the 48 classification configurations, achieving the best or parallel best outcomes in 44 configurations; the average one-shot accuracy for LLMs across three classification datasets and four target groups increased from 81.2% to 88.1%.
You Only Edit Once: Incentivizing In-Context Capability of LLMs through Local Demonstration Refinement
The paper was written by Jiarong Wen, Cheems Wang, Yun Qu, Yixiu Mao, Heming Zou, Haoang Chi, Lizhou Cai, Yiqin Lv, Kaiyu Zhang, Yuhang Jiang, Xiangyang Ji.
The following is a link to the article: arxiv.org/pdf/2609.33609
Background: The key to ICL is not just the number of samples
ICL does not need to update large model parameters, only provides a small number of examples in the input, and the model can be adapted to current tasks based on these demonstrations.
It's not just the results, it's the examples themselves.
The above context windows, cache and reasoning calculations have costs, so it is not possible to infinitely increase demonstration. Even if you add context, it does not necessarily mean that the accuracy rate will increase. Examples that are unrelated to the current query, duplicate content or not matched to a high degree, will both occupy tokens and may affect the attention distribution of the model.
So one of the real problems of a few-shot scenario is:
With a limited few-shot budget, what examples should be included in the prompt?
There are methods that can be roughly divided into two categories.
The first class is retrieval methods, such as Semantic TopK and BM25. The former selects examples based on semantic similarity, while the latter focuses on word-side matching. These methods are faster, but similar to queries, and do not necessarily mean that examples are best for the target model to complete the task.
Another category is model-aware methods, such as using generating probabilities, cross-differentiation, conditional arrays, or target model feedback to re-evaluate candidate examples. These methods can more directly leverage the behavior information of the target LLM, but the number of calls to the target model will increase if different demonstration sets are constantly tested.
In addition, the standard k-shot subset selection itself is a combined search query.
If a large number of combinations are given to the target LLM test, the cost will soon be high.
The paper takes a more local approach: if the retriever has found an even better initial set, can it not retrieve the entire space and only make a modification near that set?
A key observation: just changing one demonstration could be a lot worse.
In TREC's 4-shot setting, the researchers first considered the demonstration set obtained with Semantic TopK as the starting point, considering only one editing of its neighbourhood.
There are three possible scenarios for an editing: to remain unchanged, to remove an example, or to replace an existing one with another in the candidate pool. In this very small local space, there is still significant performance improvement space.
For Llama-3-8B, Oracle could improve by 26.0 percentage points compared to Semantic TopK by using test labels to select the best local moves; Qwen2.5-7B improved by 16.8 percentage points. Even without using test labels, Jev-LDE's self-selected editing moves received 8.2 and 7.4 percentage points, respectively.
A random local editing has no obvious benefit.
This is also the starting point for LDE: you don't necessarily have to look for a complete optimal demonstration set in a large composite space, and in some cases, you just have to change the current retrieval results to one of those locations.
Method: Turn the demonstration selection into a local editor
The LDE does not replace the original detector, but rather the entire process is more direct after the detector is placed.
First, take the initial demonstration set from an existing pre-selector.
The paper uses Semantic TopK by default, while retaining part of the candidate demonstration as a replacement pool.
Then, give the query, the current demonstration set and the replacement candidates to Jev-LDE.
In particular:
Keep: Keep the current demonstration set;
Delete: Delete one of the examples;
Replace: Replace one of the examples with another example in the candidate pool.
Finally, the edited demonstration set is given to the target LLM to complete the prediction.
The limitations here are clear: Jev-LDE can only edit membership once.
It will not continuously attempt multiple contexts, nor will it repeatedly call the target LLM for different candidate sets during the test phase. The previous detector is responsible for finding an initial set, and the Jev-LDE makes another local adjustment on this basis.
Jev-LDE is not responsible for answering questions, but for judging whether or not to change them.
Jev-LDE uses Qwen3-1.7B for demonstration editing. The input includes task instructions, current queries, already retrieved demonstration sets, and candidate demonstrations that can be replaced.
The model ultimately produces structured JSON actions, such as Keep, Delete, or Replace. If the output format is invalid, the training reward is 0, and the test returns to Keep.
In training, researchers need to figure out how to let the small model know if an editing is actually useful. The paper directly uses the predictions of the target model as feedback. For the same editing state, the Jev-LDE takes multiple actions, each of which corresponds to a new demonstration set. The frozen target LLM uses this set to make predictions, and if the predictions are correct, the reward is 1; otherwise, the reward is written as:
。
Therefore, training does not require manual labeling of which Replace action is the correct answer, but directly observing whether the target model is correct after editing.
The researchers then used GRPO for RL post-training of Jev-LDE. Since there may be multiple effective movements in an editing state, reinforcement learning enables the model to explore between these movements without having to specify a single supervisory label.
The test phase is simpler: Jev-LDE selects a move based on the current query and candidate demonstrations, with the goal of LLM completing the formal prediction in the context after editing.
Why is local editing so effective?
The paper analyzes LDE from two perspectives:
First, the researchers examined how far the current retrieval set is in the nearest correct context. In the majority of text voting, the tail of the optimal editing distance decreases geometrically. In other words, if the current retrieval results themselves already have some quality, then some errors may require only small local modifications to be corrected.
Second, the paper analyzes the amount of feedback required to learn editing policies. Under fixed strategy classes, accuracy and confidence conditions, the upper bound of full feedback is related to the linearity of the number of actions available.
These analyses do not demonstrate that one-time editing is sufficient for all tasks, but that, after explaining why search scope is limited, there may still be many effective opportunities for modification while reducing learning and search difficulty.
Experimental setup: a 1.7B editor that serves multiple target models
Jev-LDE uses Qwen3-1.7B as an editing model and conducts RL post-training on AGNews, TREC, DBPedia and GSM8K. At the training stage, Meta-Llama-3-8B-Instruct is the only frozen feedback model.
The main experiment covers three categories of tasks:
TREC: Classification of issues;
DBPedia: Category of topics;
Banking 77: Intent to classify.
Target models include Qwen2.5-7B-Instruct, Meta-Llama-3-8B-Instruct, Qwen3-8B and Qwen3.5-9B, and shot budget from 1-shot to 4-shot.
Comparison methods include Semantic TopK, BM25, TopK+ConE, LMS3 and CASE. In addition, the paper uses held-out benchmarks such as Banking77, SST-2, AQuA-RAT, TACRED that did not participate in post-training, and further uses DeepSeek-V4-Flash to test cross-target model migration.
Results of the experiment: 46 out of 48 classification configurations were better than the initial retrieval results
In the main experiment, Jev-LDE outperformed its Semantic TopK initial results in 46 of the 48 classification configurations and achieved the best or parallel best accuracy in 44 of those configurations.
Through TREC, DBPedia, Banking77 and four-target LLM statistics, the average one-shot accuracy increased from 81.2% of Semantic TopK to 88.1%.
If you look at TREC's 1-shot, the change from single editing is more obvious:
The paper also tested 8-shot and 16-shot: under a larger shot budget, Jev-LDE still outperformed Semantic TopK in 18 of the 24 configurations, but the overall advantage was not as concentrated as it was with 1-shot.
Experiment 2: Training with only Llama, and strategies can be migrated
Jev-LDE's training reward only comes from Meta-Llama-3-8B-Instruct, but the target model must be this one when tested. The paper reports that in other Qwen target models, Jev-LDE improved initial detection results in 35 out of 36 configurations; in held-out Banking 77, 14 out of 16 configurations gained improvements; in held-out SST-2, all of the settings shown in the chart were positive gains.
From these experiments, it can be seen that Jev-LDE is not the only target model used for training purposes. The paper argues that it may have learned some more general demonstration editing judgments, such as the relevance of examples to queries, redundancy of content, and the help of label boundaries. However, this still belongs to the explanation given in the paper and has not been verified by individual experiments.
Experiment 3: Replace a pre-selector and continue editing
The LDE has no hardness requirements for the front detectors. Researchers used the results of BM25, CASE and TopK+ConE respectively as the initial demonstration set, and then gave it to Jev-LDE for editing.
Therefore, it is closer to a post-processing module placed behind the retriever: first look for a set of examples, Jev-LDE and then make an adjustment based on the current query. The size of the candidate pool will also affect the results. On the one-shot TREC, the default N=16 candidate pool reaches 80.6% accuracy; after expanding to N=32, the accuracy rate decreases to 79.4% instead.
Dissolving experiments: Random modification does not produce the same results
The paper further compares different editing policies. The one-shot TREC, Meta-Llama-3-8B-Instruct is set as the target model:
Jev-LDE：65.6% ；
Keep the original set directly: 52.8%;
Directly instruct Meta-Llama-3-8B-Instruct to edit itself:57.8%;
Qwen3-1.7B:31.2% without RL post-training;
Random local editing: 23.2%
The Jev-LDE is 12.8 percentage points higher than the original Keep set and 7.8 percentage points higher than the direct editing of Meta-Llama-3-8B-Instruct as instructed.
According to the results of this group, the effect is not to modify the context sequence. Following RL post-training, Jev-LDE is more effective at selecting the three categories of Keep/Delete/Replace actions.
Reasoning costs: Accuracy from 65.4% to 80.6% and end-to-end time increased by 11%
If demonstration refinement itself requires a large amount of additional computation, the value of local editing will also be affected. The paper compares end-to-end wall time on the one-shot TREC + Qwen2.5-7B-Instruct. Semantic TopK's running time is about 82.1 seconds, while the Jev-LDE is 91.1 seconds, with an additional time of about 11.0%; at the same time, the accuracy rate increased from 65.4% to 80.6%.[3]
In contrast, TopK+ConE, which requires more target model evaluation, runs about 197.5 seconds. The time expense of Jev-LDE is closer to LMS3 and CASE.
Thus, the Jev-LDE is not entirely free of additional costs, but instead replaces the search process that might have required multiple target model sorting, with a small model editing and a formal target model reasoning.
There are also boundaries: the effects of classification tasks are more stable, and mathematical reasoning does not have the same benefits
On GSM8K, the average accuracy difference between the Jev-LDE and Semantic TopK is basically controlled by less than 1 percentage point; on the held-out AQuA-RAT, Qwen2.5-7B-Instruct and Meta-Llama-3-8B-Instruct decreased by 0.8 and 1.9 percentage points respectively. Thus, the most obvious and stable results are currently concentrated on the classification task.
The method itself has clear limitations. LDE can only be done once in the current candidate pool, Keep, Delete or Replace. If a really useful demonstration does not enter the candidate pool, or requires multiple examples to be modified at the same time, a single edit cannot be done. In other words, LDE is a smaller search space in exchange for lower reasoning and learning costs, and it does not replace a complete demonstration selection.
In the end, I'm going to try to recover it and change it.
The core idea of this work is not complicated: a search engine that already has a set of available demonstrations is usually able to find a set of available demonstrations, so it is not necessarily necessary to re-search the entire composite space every time.
The LDE retains the original demonstration selection and then makes a local modification around the current result. The Jev-LDE is trained by validating feedback from the target model to learn when to Keep, when to Delete, when to Replace.
From the results of the experiments, this method was more apparent in few-shot classification tasks: 46 of the 48 classification configurations improved initial retrieval results, with an average one-shot accuracy increase from 81.2% to 88.1%; on one-shot TREC, the end-to-end reasoning time in Semantic TopK increased by about 11%.
At the same time, Jev-LDE does not need to be retrained for each new target model, and migration effects have also been observed on multiple held-out classification benchmarks and new target LLMs.
For ICL, this work provides a more specific way to optimize: not by adding a demonstration, not by re-searching the entire portfolio space, but by re-adjusting the results that have already been retrieved.
© THE