Guidance: The accuracy of the visual questionnaire model is improving, but the high scores don't necessarily come from visual understanding. When the reference image and the answer have entered the training material, the model may simply reproduce memory. Teams such as Zhejiang University have proposed ReKey: retain the original real scene, only update the little visual clue that determines the answer, so that each round of evaluation is directed to unseen content.
After an artificial marking, the system can randomly generate new visual clues in each round of evaluation and automatically construct corresponding questions and answers.
REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation
Link to the paper:https://arxiv.org/abs/2606.20736
The main page of the project:https://xxxxxsun.github.io/rekey-vstar/
It was written by Tengjie Lin, Yutao Sun, Jingwei Ni, Shuhan Ge, Hao-Xuan Ma, Yanting Miao, Wangyue Lu, Mingshuai Chen, Tiancheng Zhao, Jianwei Yin.
It's not about changing the picture, it's about changing the question: how many high scores do you get from memory?
V*Bench is a common benchmark for assessing fine-grain visual search capabilities, containing 191 high-resolution real-scene images. The difficulty is that the problem is often not pointed at the subject of the image, but at local details far from the center of the image, which are positively small and easily overlooked. The paper calls the content that really determines this type of answer visual key. Because the answer depends only on this small piece of content, once it is fixed in the long run, the model has the opportunity to rely on memory rather than observation.
To test this, the researchers designed the Diagnostic Experiment Relabel V*: to retain the original image, only rewrite the problem, so that the new problem points to another visual clue in the graph that is equally small and equally detached from the center. On all 191 images, the accuracy of eight VLM subjects decreased by 9.5 to 18.8 percentage points.
After the image was unchanged and only replaced, the accuracy of all eight models fell below the original score, decreasing by 9.5 to 18.8 percentage points.
Since the image remains unchanged, this decline can only be attributed to updates to a combination of questions and answers. It is not enough to prove data pollution alone, but it is consistent with the assumption that the model receives memory gains from static parameters. However, this practice has a clear limitation: every round has to be rewritten manually, and the cost cannot be borne in the long run.
How do we do things differently every round of evaluation?
ReKey's idea was to compress the artificial input into one-time preparatory work, and then give it to the automated process to repeatedly produce new tasks. The entire process was divided into four steps: first, the labeler spins out small areas that can be modified on the chart (the paper is called the edit slot), then the system randomly samples a new set of visual content and writes an image editing model into the area, and finally, based on this sampling record directly generates the corresponding questions and answers.
Artificial marking occurs only once on the leftmost side, and the sampling, editing and Q&A structure on the right automatically repeats in each round of evaluation.
There are two types of areas that markers need to circle: Replace slots that point to an existing target in the graph to change its color; Add slots that point to a reasonable blank location for inserting small objects that do not exist in the original graph. Both types of areas require compactness; location is reasonable, and the graph contains the only indicator. At the start of each round of evaluation, random seeds decide which slot is enabled and sample a new combination of colors, objects, or locations.
In this way, the answer is determined by the sampling process before the image is generated, without the need for another model to check the generated results and re-mark, with each question tracing back to a specific slot, random seed and sampling content. Image editing is done by GPT-Image-2, and the edited local area is synthesized back to the prototype.
In the same scene, only the color of the hat that determines the answer is replaced, with the background confusion and small target sizes unchanged.
Have you been able to update the visual key and do you have memory advantages?
The results should return to the level of the artificial rewriting problem after updating the visual key if the previous gap was indeed memory-based. The researchers used three random seeds to generate a set of ReKey instances and re-evaluated them on eight VLMs: the original V*Bench averaged 82.1%, the ReKey rose to 72.2%, the average dropped 9.9%, and all models declined. The greatest decrease was concentrated on the original high-score model: Qwen3.6-Plus dropped 14.9%, Gemini 13.0%, while the original performance MiMo and Llama dropped only 4.4%, more in line with the predictions of memory gain and loss after updates.
Eight models on the ReKey were lower than the original V*Bench, averaging 9.9 percentage points lower overall than Relabel V*.
Qwen and GPT-5.5 are almost right on this baseline (1.4 and 1.2 percentage points, respectively) and show that ReKey recovers the same difficulty as the problem of artificial rewriting; the rest of the models averages 5.3 percentage points higher, and the researchers attribute only one set of fixed themes to Relabel, whereas ReKey has a variety of themes.
Why is the most critical step still being taken by humans?
The researchers tried to make the GPT-5.5 automatically select the edit slot, but the results were not ideal: it gave only 33.6% of the Replace area to accurately cover the target; although the majority of the Add area was available, it was generally oversized, about three times the area of the manual marking.
VLM's selected areas are significantly wider and noticeably larger, with an area about three times that of manual marking, giving the auto-marking version an accuracy of 8.4 times 18.5 percentage points.
If the model has been able to stabilize the location of the most hidden visual cues, it may already have the sensory capacity to evaluate the required measurements; while making insufficient model selections, it is easy to bring its own blind spots to the benchmark.
All 191 images in the slot are marked by about 16 people, and the same set of tags can be used repeatedly across random seeds. ReKey is not about releasing a larger static dataset, but about converting high-quality real-life scenarios into sustainable updates: maintaining the original visual search environment, updating only those bits that really determine the answer.