Loop Transformer，
Is it really possible to extend to visual language models?
In recent years, Loop Transformer has gradually shifted from simple module replication to more complex layered loop computation, exploring how to continuously update the model status by sharing parameters. A natural question is: Is the mechanism still in place when the model starts looking at the graph?
Images often contain text, objects, and spatial relationships at the same time, and different cycles may need to focus on different visual evidence. Again, if the same parameters are invoked, will the model continue to focus on the original location, or will it turn to the one that was ignored the first time?
Recently, researchers from institutions such as the People's University of China, Baidu, Qingdao, Shanghai Transportation University, Qinghua, and others have proposed LoopVL, which completes system exploration around this problem from language pre-training to multi-mode post-training. In addition to verifying the feasibility of circular visual language modeling, the team observed a noticeable cross-loop visual attention rearrangement, and called it the Visual Aha Moment.
The title of the paper is LoopVL: Recurrent Visual Intelligence.
The document is addressed at: arxiv.org/abs/2609.38426
This is the first time I've ever heard of this project.
Weight of the model: huggingface.co/TierFlow/LoopVL
Let's look at the results:
How far can a 1B language master go?
Let's take a very direct comparison: LoopVL uses only 0.14T training tokens, while Qwen3.5-2B and InternVL3.5-2B use about 36T, and the amount of training data differs by about 260 times. Despite the small size of the data, LoopVL is still capable of achieving even greater data efficiency and reasoning potential in evaluations such as LogicVista, MMMU-Pro, MMStar, RealWorldQA, MMK12-Math, and MathVision.
LoopVL and public VLM performance comparisons such as Qwen 3.5 and InternVL cover general understanding, hallucinations, mathematics, and science. The scale of each model's training tokens is listed in the first row.
The team constructed three non-circular baselines using the same Transformer block design: a 32-layer Transformer-VL 1B, and two approximately 4B models that were expanded by adding depth or width, and unified using a 0.14T training token budget.
Compared to the two Transformer baselines of about 4B, LoopVL maintains a language core of about 1B, increasing computational depth by cyclically reusing shared modules. In eight controlled assessments, LoopVL has five simultaneous outcomes of Deep and Wide:RealWorldQA, reaching 70.98, higher than the latter 66.54 and 66.01; MathVision reaches 38.49, higher than 35.92 and 33.55.
A controlled comparison under the same training token budget, intercepting five indicators: Deep and Wide, respectively, through the deepening and widening expansion model.
It should be emphasized that parameter replication does not mean lower computational cost of reasoning. LoopVL and 1B baselines have 32 sets of independent Transformer layers, but execute 128 and 32 layers of calls, respectively, so the above results are not the same as the comparison of reasoning costs.
32 layers of parameters, 128 calculations:
The two-layer loop of the LoopVL
LoopVL follows HRM-Text's layered looping architecture, dividing the language mainly into L、H modules with 16 layers of Transformer. Under the default H2L3 configuration, L executes three consecutive times, then executes one H, repeating the entire process in two rounds, i.e. [L×3→H]×2.
Module-Loop repeats the L module, Model-Loop repeats the entire L/H update cycle.
The team first trained their language master with 75B tokens and then went into multi-modal training: the visual alignment stage froze the language master and the visual encoder, training only the random initiation projector; then trained the language master with the visual interface, then completed the supervision of fine tuning and visual reinforcement learning.
I'm not sure what I'm going to do with my eyesight.
What's going to happen?
LoopVL does not use visual tokens that enter the language mainstream as fixed features. They correspond to the spatial position in the image, but change with the cycle. After the end of the first round, the next round is read as a representation of visual language interaction.
To determine whether these updates were necessary, the team kept the first round of calculations, blocked the transmission of L module visual updates in the second round, respectively, or froze the visual state completely. Both interventions reduced performance. But this was not enough: a sudden change in the rules of computation in reasoning could in itself result in training and inconsistency with reasoning.
Therefore, the team trained a Read-only version separately: the first round allowed to update the visual state, and the second round allowed only reading, training, and reasoning following the same rules. The visual token remained readable and the non-visual state continued to update. As a result, the normal LoopVL remained the highest of the four assessments, with the re-trained read-only version still lagging behind it.
A normal model with three visual state updates controls. Read-only trained is a variant trained separately, not just frozen in reasoning.
This group of experiments supports a key judgment: continuous updating of the visual state is a useful part of the LoopVL loop calculation. It is not the first round of image processing, after which only the language continues to run.
A view of the intersection of the circles Aha Moment!
The Visual Token maintains a spatial correspondence with the image area, which provides an intuitive window for study cycles: where exactly does the model focus when the same shared module is called again?
In multiple diagnostic samples, the team recorded changes in visual attention along 128 layers. The Spatial Entropy describes differing attention distributions, and the Gini coefficients measure how uneven the attention weights between visual tokens are: the lower the former, the higher the latter, usually means the more focused the visual attention.
The most interesting phenomenon occurred at the intersection of the two rounds of the Model-Loop. The average Gini coefficient suddenly jumped from 0.405 to 0.858 from the 63rd to the 64th layers of the call, and the second round's spatial slope was also lower overall. This meant that the model shifted from the relatively decentralized visual focus of the first round to the concentrated focus of the second round on a small number of visual areas.
Figure 7: Average Gini trajectory in multiple diagnostic samples. Significant attention-concentration phenomena occur at the intersection of two Model-Loop wheels.
This exciting phenomenon demonstrates that the model is not simply a repeat of the calculation, but a redistribution of visual attention in a new round of cycles. The first round may focus on broadly integrating visual information, while the second round may further focus on a few key areas, presenting a trend of attention from broad perception to selective attention. This phenomenon persists even with the adoption of complete reverse propagation, or the removal of visual focus points and visual gate control mechanisms.
Thus, the value of circular computation may lie not only in increasing the computational depth, but also in allowing models to adjust the processing of visual information step by step, but this change provides important clues to understanding how circular models can achieve deeper visual reasoning.
The focus shift, of course, is still not indicative of the model's finding of different evidence. For this reason, the team further compared the visual attention distribution of the two endpoints of the loop to the final layer of the H module sharing the same circuit, to see exactly which regions the model shifted its attention to.
Interestingly, some of the more important visual tokens that received less attention in the first round jumped to the top 25% of attention ratings in the second round. This change is not only seen in simple color judgement and counting tasks, but also in more complex visual tasks such as word recognition, direction judgement.
Figure 8: Redistribution of visual attention in the same shared H layer in two rounds. The first two lines show the two rounds of attention distribution, and the third line marks a significantly improved visual Token, in which orange represents the target area and blue represents the target area outside.
This phenomenon is called the Visual Aha Moment. It is most notable in that the model does not change the parameters, but rearranges the visual attention area in the second cycle. This means that the cycle calculation does not simply repeat the existing visual processing process, but may use the information accumulated in the previous cycle to rediscover the visual cues that are worthy of attention.
These findings together reveal a feature of the Loop architecture: even if the models share exactly the same parameters, they can present different visual information processing behaviours in different cycles.
However, the cycling isn't much better either. With fixed H2L3 training weights, MMStar dropped from the default 63.47 to 58.20 and 24.00, respectively, after the reasoning scheduling was changed to H3L3 and H4L3.
Finally, share the parameters.
No need to repeat the same calculation
LoopVL puts two things together: on the one hand, complete training and controlled experiments verify the feasibility of modeling the loop visual language; on the other hand, Visual Aha Moment reveals a remarkable phenomenon: even if it shares exactly the same parameters, the model can redistribute visual attention across different loops. This means that the expansion of the model's capabilities may depend not only on the size of the parameters, but also on how the computation is organized, and how the information accumulated from the previous round is used.
This provides a new way of thinking for large multi-modular models. Faced with complex visual tasks, the model requires not only stronger perceptual abilities or longer reasoning processes, but also attention to visual information with the deepening, dynamic adjustment of reasoning. In the future, it will be worth exploring whether the model can independently decide which areas should be re-analyzed, which information needs to be further integrated, and when to continue or stop the cycle.
Furthermore, this also provides a new direction for the multi-mode Agent. In complex tasks, the Agent needs to continuously adjust its understanding of the environment according to the mission objective, gradually focusing on critical visual information from overall perception. The cross-loop visual attention redistribution observed by LoopVL shows that even sharing the same parameters, models can change visual attention as internal states evolve. The future is worth exploring whether the Agent can independently decide which information needs to be analyzed in depth, when to continue computing, and when to take action to achieve more efficient perception, reasoning and decision-making.
Perhaps the future direction of multimodal models will not only allow the model to have more parameters, perform more calculations, but also allow it to learn to allocate the calculations more efficiently: when should it be reviewed, what should it be focused on, and when it has enough information to take the next step.
Information from the author
The study was carried out in collaboration with teachers from the Chinese People's University of Guangzhou Institute of Artificial Intelligence and the Baidu Lee and Liu team, the Shanghai Transportation University's Serious Professors' team, the Chinese University of Guangzhou's Korean Gong Professors' team and the TierFlow team.
Prof. Mariang, the team of Prof. Han Sung and the TierFlow team focused on research on optimization of intelligence, model compression and optimization of reasoning.
The Baidu Lee and Liu team focused on studying the construction of base models, training after large models, generative retrieval/recommendation.
Zhe Qian, PhD student at Shanghai University of Transportation, Zhongxing Xu, PhD student at Monash University, along with Hehan Li Li and Liu, PhD, Project Leader.
© THE