He is a PhD student at the Harbin Industrial University (Shenzhen), where his main research focuses on efficient basic model architecture, linear/mixed attention mechanisms.
The quadratic complexity of Softmax's attention, which has always been the biggest obstacle for large models to move forward in the context. Although linear attention is fast, training a large linear model requires tens of thousands of GPUs from the start. Thus, "directly linearising the existing pre-training LLM" has become the most cost-effective route.
STILL was recently officially adopted by NeurIPS 2026. STILL's core contributions can be summarized in two ways:
The first truly co-layered mixed attention framework. The previous in-layer scheme (LoLCATs, Liger, etc.) merely handed the token in a fixed sliding window to Softmax, handed the linear attention outside the window.
Using the local window to predict useful historical information for the future. STILL proposes a "Self-Saliency Score", which can only be calculated in a local slider window to predict that a token will not be followed in the future after it becomes distant history, theoretically ensuring a local - overall ranking consistent, with a proven crossover rate of up to 84.8%.[3]
Feature mapping with guarantees is implemented in parallel with NP-Map and Unified Training - Logic, STILL completes linearization with just 0.04B training tokens, achieving an 80% cache reduction in the 16K context, 83% pre-fill acceleration, and a maximum of 86.2% improvement in the relatively linearised method on the long-range benchmark.
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Written by Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan, Jingyi Zhang, Yingjian Li, and Zheng Zhang
Units: Harbin University of Industry (Shenzhen), Penang City Laboratory, University of Queensland, Riyadh and Shanghai)
Link to the paper:https://arxiv.org/abs/2602.02180
Open source code:https://github.com/ZacharyMeng/STILL
Introduction: Mixed within layers, is it really "mixed"?
Intra-Layer Hybrid Attention: Within the same attention layer, a small portion of the token is processed using Softmax Attention (SA) and the remainder is summarized in the following, with only a small amount of dilution and dilution required to allow the pre-training LLM to "transform" linear models.
However, there are two key flaws to this paradigm:
The existing methods generally rely on sliding windows for routing: tokens within the window run SA, where the entire window is handed over to LA Uniform Compression (LoLCATs/Liger), or as in Compression Rare Attention (CSA) where unselected tokens are directly discarded NULL. In the author's view, this way of dividing by fixed window is simply "picking" Softmax with linear attention to the position. Regardless of the real mix, some tokens with high processing value should be determined by their importance, rather than by how far away from the location they are.
Figure 1: Interlayer Mixed Attention Contrast. LoLCATs/Liger apply linear attention to the context unified on the outside of the window, long-text capacity limited; CSA uses compression sparse attention, but unselected tokens are completely discarded ((NULL); STILL maintains high fidelity Softmax attention for important tokens, while linear attention retains all context intact.
Previous studies (such as NaLaFormer) have noted that pre-training Softmax focuses on "parameter perception": vector amplification modulates the intensity of the expression, directly affecting the sharpness of the attention distribution. While mainstream linearization schemes (such as Hedgehog) are introduced to learn MLP before character mapping, the training allows for freely scaled characteristic parameters.
The two issues point to the need for effective linearization: (i) to reflect the importance of tokens rather than the routing of locations, and (ii) to maintain pre-trained parameter statistics rather than relearning kernel mapping. STILL is developing around these two points.
Highlight One: The Mixed Attention Framework
What constitutes a true "mixture"? STILL's answer is: let each token decide which branch it enters based on the importance of its content, rather than its location. The paper's title is a pun on the topic Selecting Tokens for Intra-Layer Hybrid Attention: Selecting Tokens for Intra-Layer Hybrid Attention: First select the token, then discuss the mixture.
Figure 2: Mixed linear attention architecture within the STILL layer. First in the sliding window, the significance scores are calculated for each token, Top-k high-point tokens are routed to Softmax attention, the rest of the tokens are processed by linear attention; each token is selected by block selection to be replaced, supporting parallel training and reasoning.
Fixed-budget routing. STILL sets a fixed Softmax KV caching budget m for each attention head that is not related to the sequence length: wherein one slot W is reserved for the nearest sliding window token, while the rest of the m−W slot is allotted to the historically significant token selected according to the score, each completed chunk retains λ = m−W) / T_max. This is the key token that, once selected, maintains a permanent, unselected cumulative linear attention loop state, with no need to rearrange or slide away the entire process.
Delay selection in parallel with the block. Each token route introduces a negligible expense, disruptive parallelism. STILL is changed to chunk as a graph to "delay selection": when the new chunk begins, the token in the previous completed chunk is selected Top-λ by scores. The training and filling phase takes the form of a pre-division block in parallel, sharing the same set of route rules with memory update behavior during the decoding phase, and the training-proposed architecture is fully unified.
Highlight 2: Self-Absolute Score
Use a local window to predict "future useful" history
True mixing requires a routing signal to answer a critical question: will a token be followed by a future query after it becomes a distant upper context?
STILL proposes a Self-Saliency Score: for each token, in its local slider, calculate separately the two slider attention distributions a^diag and a^nodiag for "containing its own item" and "removing its own item", and then use the KL-type sensitivity of the two as the token's score:
Intuitively, the more sensitive a token is to its attention distribution (the stronger the self-interaction), the more it tends to maintain a high ranking under the overall context of attention.
The ability to "predict locally and globally" has a rigorous theoretical guarantee:
Proposition 1 (local - global consistency): Set p_local and p_global as local and global self-attention probabilities, respectively, and when ε→0+ the self-significance score is satisfied: 1) monotony ∂Score_t/∂p_local > 0; 2) sequence Score_i > Score_j Score_i contains E [p^(i)_global] > E [p^(j)_global]
Experimental visualization also confirms this: high-score tokens marked in red in the sliding window are still high scores under full context attention. The majority of tokens are close to zero, suitable for LA, with only a few notable exceptions, worthy of SA refinement.
Figure 3: Distribution of Significant Points. Left: The highest point token under sliding window attention (red dot) is routed to SA; right: Placing these tokens in full context attention is still bright, showing strong local - global consistency.
An interesting proof is the word cloud analysis: the most frequently chosen by SA branches are negative words (not, no, never), mood words (will, would, may) and quantifiable words (all, any, nothing) such as "high jurisdictional" functional words. They have a global impact on the whole syntax, and precisely the most reliable Softmax attention is needed to accurately model.
Figure 4: The most frequently selected token word cloud for SA branches with significant scores. High-frequency enters are more likely to use high-domain functional words such as negative, mood, and quantization, showing that SAs prefer to retain accurate, context-dependent interactions.
The authors also did a set of direct verification: first select the token for Top-224 with local-calculated auto-significance scores, then run full attention, and then run the statistical follow-up query that really focuses on remote keys and their convergence rates. The results show that the convergence rate for low, medium, and deep representative attention heads is 62.9% and 84.8% respectively.
NP-Map: Put the pre-training parameter "back to the way it was".
Norm-Preserved Feature Map, NP-Map: No longer allowing MLP free scaling characteristics, but explicitly deconstructing "direction" and "modulation".
In which f (⋅) is learned for MLP. By rewriting the original parameter ‖x‖ into the transformed characteristic, NP-Map ensures that the attention mechanism is strictly aligned with the presentation intensity of the pre-training model, whilst fully inheriting the pre-training Transformer's fine-grained semantic information while meeting non-negative constraints. In addition, STILL also introduced the gate control mechanism to improve linear attention sequence, with complete linear branches:
Experiment: Short text versus the original, long text "resurrected".
In training, STILL uses a two-stage streamline (Attention Transfer + Low Linearity) with only 20M tokens on Cleaned Alpaca data, 1024 sequence length training, at a total cost of only 0.04B tokens.
Common sense and common sense:
Average reverse transformer
On PIQA, ARC-E/C, HellaSwag, WinoGrande, MMLU, with Llama 3 8B / 3.1 8B as the base, STILL scored the best average of all linearization methods. Under Llama 3.1 8B, the MMLU increased significantly from 46.9 on the Liger to 61.3; the average score of 74.7 on the MMLU excluded, even against the super-original Full Attention Model ((73.2) 1.5 points.
Table 1: Contrasts between common sense and general reasoning tasks. STILL achieved the best average performance in linearization methods in both Llama 3 8B and Llama 3.1 8B.
S-NIAH: The Great Shark Shark:
Under 4K 86.2, the baseline was almost completely covered.
With the same or even smaller caching budget (STILL's total caching volume does not exceed the baseline window size), STILL takes the overall lead in each length at 0.5K for 4K, and the gap increases to 86.2 at 4K for STILL, while LoLCATs are only 8.8, Liger 0.0 and SWA 5.8T, rising to +77.4 against the relatively strongest baseline.
RULER's overall benchmark:
47.9 vs 7.2 / 2.1
In 8 RULER sub-tasks that require long-range memory and state tracking, STILL scored the highest average of 47.9 points with 128 sliders + 896 total caching limits (baseline windows are 896), significantly higher than LoLCATs (7.2) and Liger (2.1), and higher than Mamba2 (44.7).
Table 2 (left): S-NIAH missions at 0.5K and 4K below, accuracy comparisons under the same caching budget. Table 3 (right):4K below, RULER overall benchmark comparisons.
BABILong is a long line of reasoning
Under a strict 256 token total caching budget, STILL is fully optimized for all lengths: QA1 is upgraded from 3 to 10 on LoLCATs at 4K, and QA5 is upgraded from 6 to 24, demonstrating stronger long-range evidence aggregation capabilities.
Table 4: The BABILong benchmark compares below the length of the following on 0K × 4K.
Compatible with the inter-level strategy:
S-NIAH-1 at 16K
Embedding the internal layer mechanism of STILL into the KL-LS interlayer framework ((Llama 3.2 1B, 25% layer retains full attention): STILL on 4K/8K/16K has an S-NIAH-1 total score of 100, 16K under S-NIAH-2 to 74.0 ((LoLCATs are only 8.6, +65.4), RULER total score of 27.20, outperforming Zebra-Llama ((13.37) 13.83 points.
Table 5: Results of the in-layer-interlayer mixed architecture on the RULER benchmark 4K and 16K, all models derived from Llama 3.2 linearization.
Efficiency: 16K pre-filled time is only 17% of Softmax
In a single H200, 1K and 64K were tested:
Pre-filled: Softmax Attention encounters a storage bottleneck after 16K, STILL steadily expands to 64K, consuming only 17% of Softmax's 16K time; the delay selection strategy only increases very little delay and storage expense compared to STILL's LoLCATs.
After 32K, OOM, STILL remained almost constant, at 16K it was only 20% of Softmax, and throughput was stable.
Figure 5: Pre-Fill Efficiency − 1K to 64K, STILL has the overall advantage over delay and display storage.
Figure 6: decoding efficiency. STILL display is almost constant, and long-term context decoding delays are lower.
Melting: Every component is essential
Melting Module BABILong 1K: Remove gate controls -5.8 points, remove NP-Map -4.9 points, both of which go to −8.3 points; then remove selectivity of significance, the accuracy rate dropped from 27.5 to 12.5 ((−15.0)).
Contrast route indicators: In S-NIAH, self-consciousness averages 42.6 points, which is much higher than opposite-angle attention (23.1); q·k intrinsic dimension (22.8); key parameter (12.9); homogeneous (12.3); and random selection (12.8); and in short contexts, self-consciousness can be approximated and cannot replace self-consciousness in long-range rare retention scenarios.
Chunk Size: C=64 for the best balance point; C increased training time from 16 to 128, dropped by 35.4%; displayed storage decreased by 21.7%; token-by-token routing C=1) will increase decoding selection delays by more than three times; training speed is halved.
Spinal generalization: MMLU 52.4 under Mistral 7B under Llama 3.2 1B/3B and Mistral 7B is equally optimal, with MMLU 70.9 under Mistral 7B under LoLCATs 69.1, Liger 65.1).
What is the meaning of this verse?
STILL re-answered "What is real peer-to-peer mixed attention" and gave a complete answer:
The first content-perceived peer-to-peer mixed framework: tokens entered Softmax branches based on content importance rather than location, sliding windows responsible only for nearby and remote names generated by content competition, coupled with a fixed caching budget and unified training - parallel logical segments, achieving O (N) linear complexity;
Use the local window to predict useful history in the future: local significance scores - theoretical assurance of overall consistency (monotony + sequence) - with full-attention in the future to inquire about the actual importance of the remote key with a weighting ratio of 62.9% and 84.8% respectively;
NP-Map guarantees characteristic mapping: orientation and dimensional decomposition, emphasis on pre-training paradigms, so that linear nuclei are strictly aligned to pre-training intensity.
With a training cost of just 0.04B tokens, STILL not only uses a common-sense model of total attention against the original version, but also rescues the long-text capability of "all-in-one" in the context of 16K (S-NIAH-1 complete 100), while bringing an 80% cache reduction and an 83% pre-fill acceleration.
© THE