The visual encoder is one of the core components of the multi-modal large-language model (MLLM). Even using the same large-language model (LLM), different visual encoders can lead to significantly different downstream performance. To find the most suitable visual encoder, the most direct method is to access the candidate encoder one by one to the LLM and evaluate it after completing cross-modal alignment training, but this process requires a large amount of computational resources. Thus, how to accurately predict the downstream performance of the visual encoder before complete training becomes an important research issue.
The existing assessment methods are mainly divided into two categories: the single-mode method focuses on the capabilities of the visual encoder itself, such as the performance of image classification; the cross-mode method further considers the goal LLM by measuring the relationship between visual and linguistic expression and assessing the degree of adaptation of different visual encoders to the LLM. Intuitively, the cross-mode method takes into account the matching relationship between vision and language and should be able to provide more accurate predictions.
As the diversity of candidate visual encoders increases, the performance of traditional cross-modal evaluation methods decreases significantly, even less than that of single-modal methods. In contrast, RAVEL maintains a significant advantage in all candidate pools, demonstrating greater stability and reliability.
However, the research team, by training and evaluating 210 large multi-modal models, found that the advantages of existing cross-modal methods were not stable. In a pool of candidates of smaller size and relatively limited types, cross-modal methods are usually superior to single-modal methods; however, as the number and diversity of candidate visual encoders increases, the evaluation performance of cross-modal methods decreases significantly and even weakens over single-modal methods.
To address this issue, the team proposed the non-trained assessment method RAVEL. RAVEL maintains the basic idea of cross-module assessment and introduces two simple technical improvements: Stablized PCA Whitening (PCA whitening with stable elements), reducing interference in the dominant direction in feature space; and Patch-level Similarity Scoring (computing similarity based on image block matching), using image localization to more fully capture details in visual tables. These two improvements effectively improve the accuracy and stability of cross-module assessment, allowing RAVEL to maintain better performance in candidate information across different sizes and diversity pools.
The results of the experiments showed that in three different LLMs, RAVEL's assessment scores were significantly higher in relation to actual downstream performance than previous work: without additional training, the average Spearman correlation coefficient for RAVEL was as high as 0.85, not only exceeding the Alignment Probing method needed to be trained ((+36.1%), but significantly ahead of previous non-training methods GW ((+93.5%) and MutualNN ((+101.1%)).
RAVEL demonstrates that no additional training is needed to significantly improve the accuracy of evaluation of visual encoders, providing an efficient and reliable new approach to the selection of components for multi-modal large models, only by making more rational use of existing visual and language representations.
A strong baseline for evaluating vision encoders in multimodal large language models
Link to the paper:https://arxiv.org/abs/2610.05413
Open source code:https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval
Model weight:https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo
The team simultaneously open-sourced the 210 MLLM weights obtained from this experimental training, covering a combination of multiple visual encoders and LLM. Researchers can directly use these models to conduct studies such as visual encoder selection, cross-mode alignment, and character analysis without repeatedly completing large-scale multi-mode training.
One, choose a visual encoder, why can't it just look at its own performance?
The most intuitive approach is to directly look at how powerful the visual encoder itself is, such as how it performs on tasks such as image classification. But in MLLM, the visual encoder also needs to collaborate with the language model master: the visual information it extracts is ultimately understood and utilized by the language model master.
Experimental results also confirm this: when an LLM changes, the best performing visual coders change as well. This means that relying solely on purely visual indicators, such as linear detection, makes it difficult to determine which visual coders are truly suitable for the current LLM.
Figure 1: Performance of visual encoders under different LLMs, with the stars representing the best performance of visual encoders under each LLM.
Can existing cross-modal alignment indicators be used directly to select a visual encoder?
Studies have also attempted to use visual and linguistic information simultaneously to predict the downstream performance of visual encoders by comparing two patterns of representation. But is this type of cross-mode indicator still reliable in a larger, more complex range of candidates?
To answer this question, the team expanded the candidate pool, incorporating visual encoders with different architectures, pre-training modes and input resolution, while also covering continuous visual encoders and discrete visual terminology. The experiment found that existing cross-modal indicators were indeed better than pure vision indicators in smaller candidate pools; however, as the scope of candidates expanded, their predictive relevance significantly decreased, with some methods not even reaching pure vision indicators.
Figure 2: As the diversity of candidate visual encoders increases, the performance of traditional cross-mode assessment methods decreases significantly, even less than that of single-mode methods. In contrast, RAVEL maintains a significant advantage in all candidate pools, demonstrating stronger stability and reliability.
RAVEL: Is the close relationship between images and text consistent?
RAVEL uses a framework for assessing cross-modal close-neighborhood consistency. Its core intuition is that if the candidate visual encoder and the target LLM form a similar semantic structure for the same batch of samples, then the close-neighborhood of images and text in their respective representational spaces should also be more consistent.
Specifically, given a set of images and their corresponding text, RAVEL uses a candidate visual encoder and a target LLM, respectively, to extract characteristics and search for neighbors in their respective representation spaces: the image side looks for similar images and the text side looks for similar text. For a pair of image samples, if the nearest neighbors to the image side are found, the corresponding text also appears in the nearest neighbors to the text side, indicating that the two representation spaces find the same semantic neighbors.
RAVEL summarizes the close-neighborhood consistency of all samples and classifies them as candidate visual encoders. Because it only compares close-neighborhoods within each representational space, it does not require visual and text characteristics to have the same dimensions, nor does it require any training or downstream performance labels.
Figure 3: Framework for assessing cross-modal near-neighborhood consistency
Within this framework, in order to more accurately reflect the semantic structure of neighbourly relations, RAVEL has further designed two things:
PCA whitening with stability: mitigating the uneven effects of the characteristic on the similarity calculation.
Patch matching: retaining localized semantic information that is easily lost in the overall pools.
4-210 MLLM experiments: is the predicted sequence reliable?
The team configured 70 visual encoders with Qwen3-1.7B, Qwen2.5-1.5B and SmolLM2-1.7B, and trained a total of 210 MLLMs. All models used a unified LLaVA two-stage training process and were evaluated on 11 downstream benchmarks, covering visual questionnaires, multimodal reasoning, and text recognition tasks.
On this basis, the team compared RAVEL with nine existing assessment indicators, covering purely visual indicators, as well as a cross-modal approach that requires or does not require additional training. To ensure fairness, the methods were all compared under a budget of approximate calculations (FLOPs).
Table 1: Comparison of predictability for different evaluation indicators.ρ and r represent the Spearman and Pearson correlation coefficients respectively.
Five, why does RAVEL work?
Dissolving experiments showed that the improvement of RAVEL was primarily due to two designs: PCA bleaching significantly improved the quality of near-neighborliness, and local matching further improved predictability on this basis.
Table 2: Results of dissolution experiments.
1) PCA whitening with stability
The representational spaces of different models often have different characteristic distributions: a few higher-dimensional directions may dominate the similarity calculation, and there may be strong correlations between different dimensions, so that the recovered neighbors cannot accurately reflect the semantic relationships between the samples. PCA whitening can mitigate this problem, but conventional whitening is easy to over-amplify the lower-dimensional directions. RAVEL therefore adds stability to suppress this magnification, and L2 is unified after whitening. Further analysis shows that performance improvements come mainly from stability, while L2 unification brings further gains.
Figure 4: The effects of stability and unification on predictive effects, Baseline representing conventional PCA whitening.
2) Local image matching blocks
Two images may contain similar objects, but the point of view, location, and background of the object are different. Global polarization combines this information, and the similarity of the object itself may be blurred by unrelated background.
For this purpose, RAVEL matches directly between local image blocks: let each image block look for the most similar region in another image and then summarize the results of the two-way matching without having to establish a one-to-one correspondence beforehand.
Figure 5: The red line on the left connects the local area with a high degree of similarity; the brighter the color on the right, indicating a stronger match.
Six, the ending.
By training and evaluating 210 MLLMs, the team found that existing cross-modal indicators did not always maintain an edge in a more complex pool of candidates, and accordingly proposed RAVEL: by measuring the near-neighborhood consistency of visual and language expressions, combined with stabilized PCA whitening and localization matching, more reliably predicted the downstream performance of the video encoder.
© THE