The Full Attention layer has not yet begun to calculate, and the activation value of its first layer has already "rushed to a peak". This is like the protagonist of the concert has not yet reached the stage and the audience has already boiled in advance. Researchers from institutions such as StartLux, Qingdao University and others have used Massive Activations as a probe, and for the first time the system has carved out the "interlayer traces" left by a few Full Attention layers in a hybrid linear attention model, identifying the two core shapes of the "rushed peak" (PAS) and the "rushed platform" (ISP).
HLA LLM has become an important architectural route for models such as Qwen 3.5, Kimi Linear, Nemotron-H, and Zamba. However, how do they interact with each other when the two types of sequence mixing mechanisms are mixed into the same deep network?
The study was carried out by a joint team with StartLux at its core. StartLux, a technology company focused on all-local smart solutions, is committed to building high-performance AI systems that can be deployed on personal terminals. StartLux worked with researchers from the University of Qingdao, the Chinese Academy of Sciences, the University of Hong Kong, the University of Sydney and the University of Columbia to launch system research on this issue with Massive Activations (MAs, Large Point Activation). The research covers five Linear Attention architectures, six hybrid configurations, five data sets, and 12 open HLA LLM checkpoints, with total model parameters ranging from 1.2B to 397B.
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Link to the paper:https://arxiv.org/abs/2608.12149
Hugging Face：
The code link:https://github.com/StartLuxLabs/Massive-Activations-HLA
Full Attention "Peak" on the first floor
The researchers first summarized all the attention received by the Full Attention layer and the attention head, selected the token that received the most attention as the "Universal Attention Sink", and then fixed the token to trace its MAs deeply.
In the 1.3B,12:1 mixed-ratio M-A-P model, RetNet, HGRN, GLA, DeltaNet, and Gated DeltaNet (GDN) presented a consistent rule of five skeletons: within the continuous Linear Attention range prior to each Full Attention layer, the MAs associated with Sink always reach local maximum values prior to Full Attention. The authors call this form "Pre-Attention Spikes, PAS").
Of the five Linear Attention skeletons, the MAs are all peaked before Full Attention. Red lines are the front layer of Full Attention and green lines are the front layer of Full Attention.
This law was subsequently validated in five data domains, where the average of the maximum activation values in the linear Attention range in the first layer of Full Attention accurately reached 99.4% and 100%, well above the expected value of 9.1% in the random selection layer.
The real counterintuitive thing is that the peak is not a product of the Full Attention output, but was formed before Full Attention began to calculate. In other words, the corresponding structural imprint on the position of the Full Attention is already present in the upstream residual flow.
From an isolated "peak" to a persistent "platform".
When researchers fixed the model skeleton and changed the number of Full Attentions in only 24 layers of the model, another form emerged as opposed to the six configurations of Pure Linear Attention, 24:1, 12:1, 6:1, 3:1 and Pure Full Attention.
When Full Attention is scarce, for example 24:1, PAS is like isolated peaks. As Full Attention becomes more dense, for example 6:1:3, the MAs between the two peaks do not decrease fully, but instead remain high through the middle Linear Attention layer, gradually establishing the "Inter-Spike Plateaus" (ISP).
As the transition from isolating PAS to ISP becomes more intensive with Full Attention, the peak is gradually connected to the platform.
The quantitative results clearly illustrate this trend. In the 1.3B model, when the mix ratio changed from 12:1 to 3:1, the peak retention scores on the RetNet rose from 18.8% to 85.4%, and the HGRN from 7.2% to 92.5%.
The mixed ratio is therefore not only an efficiency parameter, but also shapes the "time shape" along the depths of the MAs: when Full Attention is scarce, the cluster value is briefly localized; when the cluster value becomes dense, the cluster value begins to reside between the layers, and the peaks separated from each other are thus connected to the platform.
Open source model verification: amplitude changes, rhythm is unchanged
To test its breadth, the authors further analyzed the 12 open model weights of Kimi Linear, Qwen3.5, Nemotron-H, and Zamba2, ranging from 1.2B to 397B. In different models, MA patterns aligned with Full Attention can be observed.
In cross-model comparisons, one difference is particularly clear: the layered form is more stable than the absolute width value. In Qwen 3.5 of different sizes, the peak height does not change with the parametric monochrome, but PAS and the ISP always follow the Full Attention layout. The results of Nemotron-H and Zamba2 extend this rule from the Linear Attention to the state space model (SSM) to the hybrid architecture of the Transformer. Different inputs also have similar characteristics: the data domain changes significantly with peak height and low, but the peak occurs in the location and the overall stability of the area covered by the platform.
The trajectory of MAs in public mixed models.
Peak and platform, written early in training
To observe their evolution, the authors trained the 24-layer GDN model of 340M and 1.3B from scratch.
The answer came early: PAS was observed after the 340M model trained 1 billion tokens; the 1.3B model had a weaker prognosis at 5 billion tokens; 10 billion tokens had a clear peak and remained aligned with Full Attention's position. Changing the insertion position of Full Attention, the PAS also moved with it.
Full Attention layer and GDN layer output gate control intervention experiments
The researchers also conducted two sets of gate control intervention experiments: adding element-by-element output gate controls to Full Attention, and removing the original GDN output gate controls. The results showed a marked asymmetry: the former significantly reduced the absolute values of PAS and ISP, but did not eliminate their interlayer structure; the latter only brought about temperature and magnification.
The group's results suggested that Full Attention's release was the main "organizer" of MA dynamics, while GDN gate controls were more likely to regulate the width of propagation away from the cluster.
"Writing, gathering, and eliminating"
A lifecycle that runs through PAS and ISP
In a 1.3B, 12:1 GDN model, the researchers fixed the same Sink token with the characteristic dimension, tracking changes in residual flow before and after the 12th layer Full Attention boundary. Systematic isolating analysis shows that PAS corresponds to a compact three-phase process:
The first layer of Full Attention forms extreme values on the token's characteristic coordinates, providing a starting point for the entire event.
After the sync is entered into Full Attention, the token that carries the isolated group value receives disproportionate attention from the subsequent query and becomes Attention Sink.
Removing the cancellation of a significant update of the symbols opposite to the same coordinate, significantly offsetting the previously written disaggregated value, and the rapid decline in PAS.
The authors summarize it as "write-in, fuse-out, eliminate-out, write-in-sink-cancel". In PAS, the three phases of concentration occur near the boundary of Full Attention, leaving a short, sharp peak on the activation curve.
PAS responds to a local write-in-sink-cancel process in three steps in succession within two adjacent layers.
The continuous form of the ISP is consistent with the "delay elimination" of the same lifecycle: after an extreme value is formed, the opposite offset updates of the symbols do not arrive immediately, but are deferred to a deeper level. The isolated values thus persist through multiple layers of Linear Attention, forming a platform that connects adjacent PASs.
The ISP responds to write-sink-cancel-delay-elimination status in the life cycle.
As a result, PAS and ISP can be integrated into the same interpretation framework as eliminated time organizations: localized, rapid elimination forms PAS, and later elimination extends high-value activation to ISP. As Full Attention becomes dense, the low valley between peaks is gradually flattened by the platform; to the limit of pure Full Attention, the distinction between peaks and platforms disappears, eventually restoring to the stable MA form in most of the middle layers across traditional Transformers.
Rare local PAS in the Full Attention configuration → Stable MAs in the ISP-connected PAS in the dense configuration → Pure Full Attention model
The three thus form a continuous form spectrum that provides mechanisms for understanding PAS and ISP.
Drawing an internal computational map for mixed architecture
This study provides a new map for understanding the internal calculations of the mixed linear attention model: the mix ratio is far from an efficiency formula, the number of layers that use Linear Attention, the length of time that Full Attention is inserted, and the "beat" that is calculated within the organizational model.
Full Attention's location corresponds to where the peak MAs form, whose density affects how long the isolated values can remain in the layers; Linear Attention's specific status update mechanism further regulates when the ISP begins to appear. A few Full Attention layers do not only work when they are in turn, but they are associated with the entry, transmission, and elimination of isolated values in multiple layers before and after.
Dispersed Full Attention will leave a regular cross-layer trail along the model's depth; PAS, ISP, and continuous MAs in traditional Full Attention models will also be integrated into the same shape-evolution pathway. Understanding this path, architectural options such as Full Attention's insertion location, mixed density, and door control methods provide new observational dimensions, as well as possible clues to model compression and quantization: layers and architectural boundaries for extreme activation concentrations may be addressed using more location-sensitive processing strategies.