The first author of this article is Liu Liu, a 2022 graduate student at Renmin University, Shanghai Institute of Creative Thought, and an instructor, Liu Xiaobo. He has published more than a dozen papers on top international AI conferences such as ICLR, ICML, AAAI, EMNLP.
Background
Context on the surface has become an important underlying capability of large language models, but context on the surface also poses a real-world problem to model architecture: full attention can directly model relationships between any locations, but leads to a significant increase in computation and storage costs with sequence length; sliding-window attention and linear attention are more efficient, but there are obvious shortcomings in effect.
Mechanics of Long-Context Hybrid Models gives an interesting answer: I'm all for it.
From a mixed position perspective, the system compares the performance of mixed architectures of RoPE/NoPE full attention, sliding window attention (SWA), and gate control linear attention, such as GLA/GDN, in long-context tasks, and further analyzes the synchronization mechanisms of different attention.
Paper：
GitHub：
I know.
Observation
The team conducted experiments on four scales: 376M, 776M, 1B, and 3B. The model was preceded by short-text pre-training at 4k, followed by long-term training at 32k; the evaluation covered PG19 PPL, LongPPL, RULER, Babilong, and LongBench, including long-term outreach under short-text training and expansion training after long-text training, while comparing layers mixed LH) and head mixedHH) in two ways.
The first observation given in the paper is that mixed position gives mixed architecture better long-text performance. RoPE, sliding window and door control linear attention both have explicit or implicit position bias, while NoPE attention does not introduce position bias.
SWA/GLA/GDN-NoPE-LH length outputs
Performance of SWA/GLA/GDN-NoPE-LH in the long term
However, there are still differences in performance between different mixed architectures. After short pre-training, the SWA-NoPE mixture typically exhibits a strong length-output capability. Even if the context length in training is only 4k, the model can perform better in a longer context.
However, long-range push-out does not mean extensive training. After long-range training, long-range performance reverses: SWA-NoPE is outperformed by GLA-NoPE or GDN-NoPE, especially in layered mixed models.
The team called this phenomenon the sliding-window hybrid-expanded brushboard effect. In response, the linear hybrid model, although better suited to long-range instruction, is less successful than the sliding-window hybrid model. This constitutes another key conclusion: linear blending has no free lunch effect.
Improvements in the expansion of the sliding window mix
The team further observed confusion curves and parameter changes before and after long-term training. For the SWA-NoPE layer mixed model, the loss of short-text position after long-term training was very noticeable, but the loss of long-term position improvement was limited.
This is known as the short-text literacy trap: the sliding window branch is more trained to optimize, but the fixed window limits the range of reliance it can model, ultimately causing long-text training to focus too much on the short context.
An in-depth observation shows that short windows are more conducive to length outputs in the short-text pre-training phase; however, in the long-term training phase, short windows limit the context dependence of model learning. The paper generalizes these two trends as short window fatigue and long window laziness.
In long-term pre-training, long-term performance of SWA-NoPE-LH tends to change with increasing window size. Larger windows have better expansion training performance, reflecting short-term window fatigue.
In short-text pre-training, the long-text performance of SWA-NoPE-LH tends to change with increasing window sizes. Smaller windows have better length outputs, reflecting long window laziness.
To this end, the paper significantly improved the long-range performance of the SWA-NoPE/RoPE layer blending model by enlarging the sliding window and combining the LongCE loss, approaching and even reaching the level of the linear blending model on multiple scales.
Improved results of SWA-NoPE-LH expansion training
Improvements to the SWA-RoPE-LH expansion training
Linear blending outputs improvements
Linear attention is better suited to long-term training, but when pushed beyond training lengths, the performance is not as good as sliding window mixing. The research team re-examined the problem from the perspective of relative position information. Although GLA and GDN do not use traditional RoPE, door controls themselves contain hidden position biases. As the distance increases, door controls in state updates accumulate, and distribution changes may occur outside of training lengths.
Based on this assumption, the paper proposes Sliding-Window Linear Attention (SWLA): to restrict the effective relative position range of GLA/GDN in reasoning, while using numerical correction of NoPE attention to enhance its overall concentration.
This strategy limited the relative positioning range of the attention perception with position bias, and enhanced the overall perception of non-position bias modules, achieving a good thrust effect, known as the matte effect of mixed position thrust. Experiments showed that this strategy could significantly improve the length thrust of the GLA-NoPE, GDN-NoPE mixture. In RULER's NIAH-SK1 mission, the model pushed from 4k training length to 64k, reaching 16 times the training-free thrust, and maintaining 100% accuracy.
GLA-NoPE-LH improvements in length outputs
GDN-NoPE-LH improvements in length outputs
How Mixed Locations Collaborate
In order to understand why hybrid architecture is effective, the paper also analyzes the attention characteristics of different attention modules. The results show that NoPE attention tends to concentrate globally in roughness, maintaining a higher retrieval hit rate in a long context; RoPE, SWA and linear attention use more location bias to limit noise, while affecting the downstream performance of long-text tasks.
In the example of a RoPE-NoPE mixture, the NoPE mixture has a higher attentional punctuation hit rate and greater attention span than the RoPE mixture. The NoPE mixture exhibits a rough but effective overall concentration. As the mixture ratio changes, the NoPE and the position of the focus with the position deviation also change the distribution boundary in the nitrogen-vitamin space. The team called this collaboration model the mixed position tidal effect.
Contrast between different mixed ratios of RoPE-NoPE-HH
Thus, the key to a hybrid architecture is not just to place multiple modules in the same model, but to allow them to form a complementary location information structure that ultimately achieves the effect of going beyond a single focus.
What is the meaning of this verse?
The work revolves around two seemingly simple questions: why hybrid architecture works and how to design it more effectively.
Conclusion 1: From Mixed Attention to Mixed Position: Mixed Position, which refers to the mixing of NoPE Attention with other positioned deviant attention, is more effective in long-range push and long-range training.
Conclusion 2: Sliding window mix extended board effect: Sliding window mixing (SWA-NoPE) length outputs are better, but extended training is less effective than linear blending (LA-NoPE), especially in layered blending models.
Conclusion 3: Linear mixing has no free lunch effect: Linear mixing (LA-NoPE) is better for extended training, but length pushing is better than sliding window mixing (SWA-NoPE).
Conclusion 4: The tidal effect of mixed-position synchronization: Effective synchronization of mixed-position, relying on a small number of NoPE attentions to achieve coarse polishing, and noise-restricting attention with most bands of position bias, can achieve effective synchronization; the distribution boundaries of the two in the graph of the spectral velocity drop proportionally.
Conclusion 5: Short-window fatigue and long-window laziness: Sliding-window mixed long-contextual training exists as a trap for short-contextual literacy lessons. While short-windows are more advantageous in short-contextual preliminary training than long-contextual training, long-windows are more advantageous in long-contextual training than extended training.
Conclusion 6: Matthas effect of mixed-position deflection: In a mixed-position model, the focus on positional deflection can be achieved by sliding windows to limit local perception, and non-position deflection can be achieved by amplifying the concentration of corrected numbers.
The team began with the analysis of the differences in the long-term effects and the internal behavior of attention at the stage of training reasoning, building analytical chains from phenomena to mechanisms to improvements. The results showed that the advantages of a hybrid architecture come from functional complementarity between different attention modules, rather than one module being stronger in all scenarios.
Linear attention is suitable for efficient learning and adaptation to context, sliding window mechanisms are suitable for controlling the relative position during reasoning to achieve length outputs, while NoPE attention can provide a more rough but effective overall aggregation. Organizing these capabilities may be the direction in which the mixed architecture continues to develop in context.
The paper focuses on the interaction between efficient attention and full attention, such as sliding window attention and linear attention, and studies their long-term performance in the text pre-training phase. In the future, the research team will also explore further the interaction between the post-training phase, the multi-modal training phase, and the interaction between the rare-attention hybrid and the compression-attention hybrid at the same stage.
© THE