In the development of large models, the main means of enhancing capability has always been to "enlarge" the array of more parameters, more data, more computation. Now, another idea is beginning to gain attention: instead of stacking new layers, rather than having the existing layers run again. This is the Looped Transformer.
The idea has shown considerable potential in tasks such as reasoning and mathematics, but one key question has not always been answered positively: are the benefits of the Looped Transformer an advantage of the loop structure itself, or is it because it counts multiple times, actually using more computing power? A team from institutions such as Tsinghua University, Byte Jumping Seed, and others answered this question positively in their paper SMELT. They simultaneously coordinated the number of algorithmic parameters and the KV cache, making a fair comparison on multiple model scales for the first time.
Link to the paper:https://arxiv.org/abs/2609.01343
The first author of this article was Dr. Wang Chaowen, PhD student at Tsinghua University, professor from Lijiang, who has published several papers at international conferences such as ICML, NeurIPS, ACL, EMNLP, and won the LIT@ICLR 2026 Best Paper Award.
Budget matching framework
Fair comparison requires the simultaneous control of three variables: the FLOPs for each token that determines the cost of training and reasoning, the total parameter of the knowledge capacity that constrains it, and the KV cache that limits the context length that can be served on it.
The paper uses MoE to analyze parameters and computational quantities, with specific matching methods as follows: 1) shrink or shrink models to offset the calculation of loop execution of the Loop architecture; 2) increase the number of experts to supplement the total parameters; and 3) adjust the attention configuration (GQA Ratio / Head Dim) to match the KV cache.
Figure 1 (a) The core practice of SMELT is to repeat the middle 50% layer twice while matching the computation, parameters and KV cache.
The best cycling formula
In accordance with the rules of control of the budget, the paper searches the cycles on a small scale sequentially, performing the depth ratios and cycle frequencies, and concludes:
1) The maximum gain of the layer in the middle of the loop is 50%. The paper moves the ratio of the layer in the loop from 0% (non-loop) all the way to 100% (full loop). The results show that when the ratio is around 50%, the verification loss is minimal. The authors estimate that the layers at the end of the model each carry special functions such as input coding and output prediction. Therefore, these layers retain independent parameters, which may be more advantageous than participatory sharing.
Table 3 shows the best layer effect at about 50% in the middle of the cycle.
2) The optimal execution depth of a Looped model is higher than that of a non-Looped model. After a fixed 50% loop range, the paper further compares different execution depths (the ratio of execution depth to model width). The results show that the optimal execution depth of a Looped model is larger. The paper suggests that this may be related to parameter sharing.
Table 4 Looped Transformer is suitable for deeper configuration.
The paper scans the number of cycles between one and four times and finds two optimums. Due to the constraints of FLOPs matching, each cycle of the model needs to be narrowed further, so three and four times instead are retrograde due to the model being too narrow.
Table 5 shows the best effect of two cycles; continuing to increase the number of cycles is a step backwards.
Combining the above three items, the paper names the final formula as SMELT: in the rare MoE model, the middle half of the layer cycle is performed twice with a larger depth of execution ratio.
Scaling Law Adjusted for Additional Benefits
The paper expanded the SMELT formula to four model sizes (100M / 200M / 600M / 1.6B activation parameters) and set four scarcity levels under each scale (by increasing the number of experts while keeping the parameters activated for each token unchanged) to 54B. Each combination trained a pair of matching baselines with SMELT. The result was that SMELT achieved lower verification losses on all 16 matching configurations.
Figure 3 Under all model sizes and scarcities, SMELT can obtain lower verification losses at approximate computations.
In order to quantify the advantages of changing trends with scale, the paper adapted Chinchilla's Scaling Law to two architectures, respectively: where E is uncompromising; the smaller the intermediate capacity item is controlled by each token FLOPs F and scarcity S, the larger the F or S, the smaller the item; and the smaller the last data item is controlled by the number of trained tokens D, the smaller the data.
Table 6 shows the Scaling Law coefficients obtained from the baseline and SMELT respectively.
The matching results showed that SMELT's capacity index (a:0.3892 vs. 0.3703) and data index (c:0.7011 vs. 0.6594) were higher than the baseline, meaning that SMELT's loss decreased more rapidly with computing power and data growth. On the Compute-Optimal front, SMELT always had less training power required to achieve the same loss, saving between 6.8% and 18.0% and increasing and continuously expanding its computing budget.
Figure 1 (b) SMELT's computational optimum edge is lower; training calculations can be saved by 14.7% at 1021 FLOPs.
Benefits of downloadable tasks
The paper evaluated downstream tasks such as DCLM and MMLU, with SMELT scoring higher on almost all matching configurations.
Figure 6 SMELT scored higher DCLM Core scores on 83 of the 96 group configurations.
Of the 30 group configurations that were significantly higher than machine level, SMELT achieved a higher MMLU accuracy in 29 groups.
It is also worth noting that these downstream gains are not just from a reduction in loss. The paper first matches the correlation curve of verification losses with downstream scores, and then places the data points of SMELT on the same chart. The results show that under the same verification losses, the downstream scores of SMELT are always higher than the baseline, and the gap increases with scale.
Figure 9 Even if the verification losses are the same, SMELT still has additional gains on most downstream missions, and usually increases with model size.
The paper further analyses the distribution of benefits of the cycle across different fields and tasks.
In verification losses, the higher the structured degree, the greater the gain: Code leads, Finance and Math/STEM follows, Web low. At the same time, the gain of long text is significantly higher than short text. By contrast, increasing the number of experts or parameters does not produce this difference in length, indicating that this is a characteristic nature of the cycle.
Figure 10 (a) SMELT has gained in all data fields, with the biggest gains coming from code data.
Figure 11 (a) The longer the text, the more noticeable the benefits of SMELT relative to the baseline.
In downstream tasks, the more in-context examples given, the greater the advantage of SMELT relative to the baseline, particularly in the Dyck Languages task, which relies heavily on examples.
Figure 11 (bc) indicates that SMELT's advantages are more pronounced in the context of the example shown in Figure 11 (bc).
Internal mechanisms of circulation
Next, the paper analyses the differences between the second execution and the first execution of the loop layer from the three perspectives of expert routing, residual flow and attention patterns.
Expert routing: The second time will be repeated with the first part of the expert, and the overlap between the two selected groups of experts is much higher than the random level.
The second execution will replicate some experts while selecting some new experts; the actual overlap is much higher than the random level.
Residual flow updates: The second execution of the residual flow updates is always larger than the first one. The two updates overlap, but are not the same. This indicates that the second execution is made on the basis of the first one with more substantial modifications.
Figure 13 shows a larger overall update of the second execution written to the residual flow.
Attention retrieval: In two executions, Q and K have a residual resemblance of up to 0.89 times 0.93, indicating that the model focuses on the same position for the most part. The change mainly occurs in V and its subsequent output. In other words, the second execution is still aimed at the same point, but the content is already different.
Figure 16 shows that the two executions usually look at similar locations, but the content of the read and write-back changes considerably.
Case Study on Dyck Language: Dyck language is a bracket matching task in which a model needs to find the correct answer in a given example in order to make a correct prediction. The paper finds that the key attention head focuses on the BOS in the first execution, while the same head focuses on the answer location in the second execution.
Figure 18 In the Dyck task, the second execution shifted attention from the BOS to the sample response.
In the baseline model, the attention sink increases with depth. Loop architecture shows the opposite trend: after the second execution, the attention sink decreases instead. The paper suggests that the first execution has established a preliminary characteristic, and the second time it is no longer necessary to rely on the sink, instead of focusing on actual content retrieval.
Figure 19 (b) The attention sink of the normal model increases with depth, while the second execution of SMELT weakens the attention sink.
Notes
The paper, under budget matching conditions, draws consistent conclusions from three levels of Scaling Law, downstream evaluation and mechanism analysis: the Loop architecture has unique advantages. Existing limitations include limited scale of dissolution experiments and system efficiency optimization that does not involve actual running time levels.
In the future, this set of budget matching schemes and the SMELT formula are expected to serve as a standard set of experiments for the Loop community. On this basis, follow-up work can continue to explore more architectural variants under fair conditions.
