Figure 1:RefCaptioner's mission motives. Normal video descriptions generate only text, making it difficult to express the correspondence between the candidate reference image and the local video syntax; RefCaptioner selects the actual content of the video from a maximum of 22 candidate charts, labeling the image after the corresponding phrase, while excluding uninterrupted items.
In recent years, with the development of multi-modal models, the field of video understanding has been able to provide detailed descriptions of video content. However, with the development of multiple reference image video generations and editing, in addition to understanding the video itself, it is often necessary for the model to process multiple reference images at the same time, such as characters, clothing, scenes, to establish a relationship between the reference image and the video.
This is the blind spot for the existing video description paradigm. Common Caption only requires text to be faithful to the video; multi-reference image scenarios also require the model to choose between candidate images, to place each effective reference image in the correct local phrase, and to understand how many different points of view of the same subject, and how gestural images may belong to the same subject.
This raises a central question: How can we better establish a relationship between video semantics and reference graphics and give a reasonable and correct understanding of video content when the input is video and a large amount of reference graphics?
For this problem, we will address the issue of video captioning under multiple reference image scenes. Firstly. Multi-reference arrays may actually have objects and scenes that appear, or may mix with seemingly similar distractions, existing VLMs either over-confident all references, or over-conservative ones that miss out on valid information. Secondly, many VLMs can identify the same person or object from multiple viewpoints, and models can also identify different people or objects for the same person or object when identifying a reference image. However, in this case, the model needs to assign these images to the same entity, rather than dismantle them into multiple objects.
For this problem, we have proposed RefCaptioner, a post-training enhanced model for the recognition and perception of reference graphs, while we structured the output caption to build structured caption formats from scenes, characters, objects, actions, appearances, cameras, etc., and strengthened the output format constraints through SFT to enhance the ability of different reference graphs to respond to different types of video language. We have established the dual-layer adaptive reward function HCD-GRPO, while maintaining the fine-particle video context, while optimizing the model for the recognition of interference graphs and the recognition of different styles of reference graphs by the same subject.
Core members of the team include Ph.D. students from Peking University, Zhang Xiaoping, and Zhang Fareng, the leader of the Koring team for multi-modal understanding.
Link to the paper:https://arxiv.org/abs/2607.28509
The code link:https://github.com/pkucs-Ltf/RefCaptioner
The model is open-source:https://huggingface.co/TengfeiLiuCoder/RefCaptioner
The following is a link to the open source database:https://huggingface.co/datasets/TengfeiLiuCoder/MRVBench
RefCaptioner：
Link the video caption to the reference image end-to-end
Figure 2: Overview of the RefCaptioner method. On the left is a mixed data supervision subtlety that maintains general descriptive capabilities; on the right is HCD-GRPO, which uses two reward branches, respectively, for constraining fact descriptions and multi-reference graph grounding.
RefCaptioner uses Qwen3-VL-8B-Instruct as a base to train reference image recognition capabilities while not losing the perception and description capabilities of visual models for video. At the SFT stage, we will use more reference image data and general video caption training data in a 1×1 balanced sampling. The former contains an artificially verified image label description to teach reference image selection and local binding; the latter is used to preserve factual, coverage and detail expression capabilities.
At the same time, using SFT alone, models can learn to output formats without necessarily being able to accommodate factual descriptions, reference summons and error suppression in complex samples. HCD-GRPO therefore breaks down the rewards into two complementary branches: on the one hand, checking that the description is faithful and complete, and on the other hand, checking that the reference graph is selected correctly, tied correctly, and discounted for observable errors.
Fact-describing Rewards: First remove the image tag in the answer, then check whether the critical facts are covered from the subject, appearance, action, background, lens, and style in six dimensions, and use a video questionnaire to identify facts errors.
Properly tied coverage rewards: only reference charts that have been selected correctly and placed in the correct local phrases can contribute directly to the coverage signal. Simply increasing the number of tags will not result in additional benefits.
Distractor-Aware Evidence Suppression (DAES): A training sample will randomly insert an unrelated image into a normal reference graph, but the model will not know which of the interference items are; an additional penalty will be imposed if the image is not referenced in the video.
Cross-Reference Semantic Coherence (CRSC): For multiple different viewpoints of the same entity, the evaluator extracts from the responses a cluster event a cluster entity a cluster reference to a cluster relationship to check whether the images are aggregated into the same local description rather than broken down incorrectly.
By designing the reward function above, the model can not rely on the use of multiple image tag tags to improve coverage, nor can it rely on the use of multiple image tag tags to avoid errors, but instead allows the model to improve the recognition of interference graphs on the basis of effective binding.
MRVBench：
Describe the equation correctly and separate the equation from the corresponding equation.
Existing video description benchmarks typically only judge whether the text covers the video content, but cannot answer whether the model has selected a reference graph, whether the label is after the correct phrase, or whether the interference clause has been misused. For this purpose, the research team built the MRVBench, which uses a unified protocol to assess video factuality and multi-reference graph grounding simultaneously.
Figure 3: MRVBench is the process of constructing data. Starting from real-time video and AIGC video, it completes key string extraction, six-dimensional caption labeling, reference pool sorting and interference insertion, local tag binding and artificial expert verification.
MRVBench covers a wide range of situations, from simple correspondence to complex multi-graph mapping, with training content:
The training content includes 20,000 videos and 171,354 reference charts.
The public MRVBench test set contains 462 videos that are completely isolated from the training content, of which 185 are AIGC videos and 277 are real videos, together with 3,831 candidate reference charts.
Individual samples contain a maximum of 22 reference charts; about 60% of samples have complex maps of multiple charts corresponding to the same subject or visual unit, and about 40% contain human-injected interference charts.
The freeze review library contains 5,846 video key facts and 2,172 short answer questions covering subjects, appearances, action, background, footage and style.
Not just to see if Caption is floating, but how does MRVBench rate it?
MRVBench will break the evaluation down into five aspects: video fact coverage with accuracy, reference chart selection, local phrase binding, interference stubbornness, and consistency of multiple reference charts of the same subject. All models receive the same video, task instructions and orderly reference charts, scores dependent on frozen critical facts, query libraries and structured labels, and try to distinguish as closely as possible between a mouse and a mouse.
The results of the experiment:
The more references to the chart, the more obvious the advantages
The paper compares the RefCaptioner with multiple open source models of the 7B X38B and includes Gemini-3.1-Pro and GPT-5.4 as closed source references. The RefCaptioner with the 8B parameter achieves the best open source model performance on all multi-reference map grounding indicators, with overall results approaching Gemini-3.1-Pro and higher than GPT-5.4.
Figure 4: Video factual understanding of different models on the MRVBench and results of multi-reference chart grounding.
On the most core reference grounding, RefCaptioner maintains a higher selection chart accuracy, recall and local binding quality, and is significantly superior to other open-source models in the exclusion of interference charts and the aggregation of multiple viewpoints. This is important: high recall is not substituted by multiple tags, and the model does learn to suppress error citations while covering valid reference charts.
Figure 5: VDC Benchmark and VCapsBench video description of different VLM ratings.
It is worth noting that multi-reference graphics training does not compromise the ability of ordinary video descriptors. On the VDC Bench of fine-grained video captions, multiple dimensions such as short descriptions, backgrounds, subjects and details, the RefCaptioner achieves the highest accuracy; on the VCapsBench of the rating benchmark, its description coverage and consistency exceeds the Qwen3-VL-8B baseline. In other words, the model does not degenerate into a VLM that only inserts image tags.
Figure 6: HCD-GRPO melting results. Based on Qwen3-VL-8B-Instruct, the complete model uses the same factual descriptions as the rewards, DAES and CRSC.
Dissolving experiments have shown that SFTs can significantly improve subject reference recall, but temporarily affect the performance of video queries; with the addition of the full HCD-GRPO, the ability to describe facts is restored, interference chart elimination and subject consistency are further improved. The three reward components correspond to fact, error reference suppression and cross-reference chart consistency, respectively, and the role is complementary.
More references to charts
Is the model still stable?
In order to observe the stability of the model in the face of complex reference graphs, the paper divides the sample by the number of reference graphs into 2×4, 5×8, 9×12 and 13+ groups, and multiplies the reference recall, local binding accuracy and interference graph exclusion rates to obtain a rubricity score sensitive to any shortboard.
Figure 7: Grounding robustness of different models when increasing the number of reference graphs.
Based on Figure 7, we found that by comparing the results of different models, the RefCaptioner remained ahead in all four ranges; it was still higher than GPT-5.4 in the most difficult 13+ reference charts.
Artificial evaluation and video reconstruction:
Does Grounding really work?
Back to video from Caption: end-to-end testing
In order to test the downstream value of the grounded caption, the paper uses the description generated by the caption model + the reference tab that describes the actual reference to reconstruct the original video. Based on the open source video generation model, Wan2.1-VACE-14B, the parameters remain unchanged when video is generated, with the only change being the input prompt, which is that the original video caption provided by different VLM also has a set of reference charts. In order to evaluate the recovery of the video generated to the original video.
Three experts used the Good/Same/Bad (GSB) protocol to anonymously compare refcaptioner's reconstructed videos to those generated by each subline. Good said the results were more in line with the source video, Bad said the subline was better, and Same said the two were equivalent.
Figure 8: Describe the results of the conditional video reconstruction on GSB's artificial evaluation. In comparison to each baseline, the Good ratio of RefCaptioner is higher than that of Bad, and the advantage is particularly evident when compared to the open source model.
This measurement measurement is not an isolated text quality, but describes the end-to-end functionality of the filter generating + reference image selection filter. RefCaptioner gets more Good than Bad in all comparisons, indicating that its output descriptions and reference graph combinations retain more material, appearance, action, and scene information in the source video.
Figure 9: Video reconstruction contrast of AIGC sample 000070. Different methods use the same Wan2.1-VACE rear end and the same generation settings, replacing only the captioner generated description and reference subset.
In summary:
More Reference Images Caption Not only the quality of the text is required
RefCaptioner deals not with the addition of image citations after ordinary Caption, but rather with the model's response to three questions while generating descriptions: which images are really relevant to video, which images correspond to which local visual facts, and whether similar but unrelated images should be rejected.
With the development of multi-reference video generation and editing, multi-reference grounding will become a key interface to linking reference graphics to video contexts. RefCaptioner provides an effective algorithm for understanding multi-reference video, while MRVBench provides a set of benchmarks for assessing the accuracy of multi-reference video captioning, both of which together form the basis for more accurate, controllable multi-reference video understanding.