And attention is also FFN? Starting with fast weighting, re-understanding efficient attention is Neu

  It's the first time in the history of the world.

In the Transformer architecture, Attention and Feed-Forward Network (FFN) have long been seen as two distinct and complementary modules:

The former allows tokens to interact with each other, while the latter makes nonlinear changes to each word.

But if you open up the formula for the mechanism of attention, you'll see a link that's easily overlooked:The standard attention can also be written as a FFN cube only its weight is generated by the current input.Hidden layersThe width increases with sequence length.

So there's another explanation for the expense in the context: we're not just processing more words, but we're running a network that's constantly expanding. So, can linear attention, mixed expert attention, top-k attention, these seemingly different technical routes be reinterpreted from "how to organize this network?"

On this issue, a team from the Beijing Postal University raised the question:Mixture-of-Top-k Attention（MiTA）What has already happened.Weighing fast (Fast Weights)The perspective developed into a set of five-dimensional classification frameworks, and based on this, Top-k experts designed variable, reproducible shapes that both retain global information and read original details on demand.NeurIPS 2026Received, code is open source.

Address of the paper:https://arxiv.org/pdf/2602.01219

The codeWarehouse：https://github.com/QishuaiWen/MiTA

InquiryKits and values in columns。

Multi-Layer Perceptron (MLP), commonly used by FFNs, is combined with standard attention, with the latitude of the previous hidden layer marked asD：

To unfold Softmax, the integration of scaling and relying on queries is absorbed into the weight:

In the meantime, there is a correspondence:

The query corresponds to the input, the switched keys and values correspond to the two layers of weights, and the index function corresponds to theActivate the functionand set to zero,Hidden layersThe width is transformed by D into the sequence length N.

The difference is in the source of the weight: the normal FFN is trained usingIt's the first time I've ever been in a relationship with a woman.Note that the current input generated by K, V is the fast weight of the changes in context.

The most important difference is the width:Each key value corresponds to a hidden unit, and the longer the context, the wider the fast weight MLP.Each query accesses N hidden units and processes N queries to generateThe total cost of the project.

Design for efficient attention, which translates into a unified question:How to make the ever-expanding Fast Weight Network scalable?

In the face of this growing network, the main idea of efficient attention can be divided into two routes.

Compression: Scaling by Compression to convert a wide network into compact modules.Linear Attention compresses key value information into a shared linear layer, which Linformer maps to a narrower MLP, while Test-Time Training (TTT) learns compact fast weight modules. This route provides a comprehensive overview, but it is difficult to directly read the details of the original key value pair.

Routing: Keep the original content and access only the required sections (Scaling by Routing).The key values are grouped into Fast-Weight Experts, and the query is selected on demand. These experts are made up of key values in the current context: the details of the selected content are retained, but the overall context is lacking.

Figure 1: The increase in context makes the MLP width shift faster. The route selects a part of it, and the compression constructs a more compact representation.

The two routes are not mutually exclusive.Scaling Strategy, Expert Count, Expert Type, Expert Construction, and Routing TopologyIt's a five-dimensional classification framework.

Table 1: The five-dimensional classification framework for efficient attention under the weighting angle of quickness.

The framework not only addresses existing methods, but also raises specific questions about new designs:What information does an expert build from? How much is needed? What queries can be repeated by the same expert?

In the example of a mixture-of-experts attention, such as MoBA, a common process is to cut the sequence into continuous non-overlapping blocks and then generate a path vector for each block, allowing the query to be selected as needed.

Routes can change with inquiries, but expert boundaries are predetermined.In an image, relevant information for the same target does not necessarily fall within the same matrix block; inquiries may require access to the entire block in order to read a small amount of relevant content.

Top-k's attention moves in a different direction: each query selects the most relevant k key value pair separately, no longer bound by a fixed block shape. However, accordingly, N's query also corresponds to the N set of "private sets".

Is it possible to retain the flexibility of the Top-k selection and allow the selection results to be replicated by multiple queries?

MiTA has changed its mind: instead of specifying the content of the experts, it takes a small number of representative queries, which are then selected from a complete context to form a number of customizable experts. The original queries only have to choose which experts to visit, without each one having to construct a comprehensive Top-k collection.

Figure 2: Complete attention to share all key value pairs, split block MoE to share fixed blocks, query-by-query Top-k to construct private sets, and MiTA to construct replicable variable experts.

MiTA extracts m by means of local average poolingLandmark Queries is an online community of landmark inquiries.Every landmark is explored.Complete sequenceAt the same time, a comprehensive summary of the information was completed and original details were retrieved.

Figure 3: For each query, both the universal key value pair accessed after compression and the original key value pair accessed through route selection.

First, build top-k reusable experts.The first landmark queryTo match all keys, select the most relevant k position:

These key values can cross different spatial locations for Routed Expert, allowing different experts to overlap as the input changes shape.

Following the questionnaire, experts selected to match their best-matched benchmarks:

Each query shares a set of candidate key values, but still independently calculates their respective attention weights.

And then build a global shared expert with the same set of landmarks.Cross-Attention is a cross-sectional query that reads the entire context to obtain the corresponding landmark value:

Using a landmark query as a key, a landmark value as a value, you get a globally shared key value of。

MiTA therefore contains m.One routing expert and one sharing expert (shared expert)The former provides basic details, while the latter provides a general overview.

Finally, let the summary interact with the detail at the same time.For each query, select a routing expert, which is configured as s=1:

Two sets of key values are used after spellingUnified Softmax is unifiedTop-k selection is combined with a global overview of shared landmark scores, and the scores for querying landmarks can also be used for routing. Compression and routing are not only complementary, they can also be used for computing.

For example, with ImageNet-1K, m=k=25, each query has final access.25 pairs of compressed keys with 25 pairs of primitive keys for a total of 50The task of scanning the entire sequence and constructing Top-k sets is done by concentrating a small number of landmarks.

Algorithm 1: First build an expert, then calculate the query subset based on the routing results, and then combine the results via online Softmax.

The ultimate inquiry of the aluminum expert attention spending from theIt has been scaled down and accelerated by FlashAttention with expert clustering. See below for more on task performance and actual running time.

On ImageNet-1K, the same DeiT training configuration is used first, as opposed to attention mechanisms that do not add additional components.

Table 2: Classification results on DeiT-T and DeiT-S. The DVC represents the gate control mechanism and the Gate represents the gate control mechanism.

On DeiT-S, the Top-1 accuracy of the basic MiTA is as follows:79.8%It's 79.9% less than full attention.0.1 percentage pointsThe parameters remain unchanged, and the floating-point operations (FLOPs) decrease from 4.6G to 4.4G. Under the same setting, Agent Attention is 78.5%.

After joining Deep-Wise Convolution (DWC), MiTA reached 80.6%; after further joining Gate Control (Gating), MiTA reached 81.2%.

Table 3: Results from combined ViT-5 architecture design and comparisons with other efficient Visual Transformers.

The MiTA-ViT-5-S reached its peak after combining the architectural design of the ViT-5 and joining the DWC.81.7%FLOPs decreased from 4.7G to 4.5G, compared with 82.2% for the ViT-5-S.

In ADE20K Semantic Segmentation, the attention of the pre-trained ViT is replaced by MiTA and the split model is re-trained. When set to m=k=49, each query accesses only 98 key value pairs, while the full attention accesses 1,024 or 1,600 key value pairs, respectively, at two input resolutions.

Table 4: ADE 20K splitting results after a different size ViT is replaced by MiTA.

In the example of ViT-S+CBT, FLOPs are derived from33G decreased to 25G, down about 24%.Mean Intersection over Union, mIoU) dropped from 45.8% to 44.5% on average. FLOPs in ViT-T configurations decreased by about 42% and mIoU decreased by 2.6 percentage points.

Of the five missions in the Long Range Arena (LRA) long-series benchmark, MiTA has an average accuracy of58.91%It's all about the people.58.19%The total training time is from10.6 hours decreased to 2.4 hours, reducing by about 77%.。

Table 5: Each task reports accuracy and training throughput (steps/s) and the last row averages accuracy and total training time (hours).

Testing was done on a single 24GB RTX 4090 with the full attention baseline using FlashAttention.

The compression branch also reduces the collection of the original key value pairs (Gather) expense: in each query accessed the same number of entries in the Ablation Study, compared to the routing version, adding the compression branch will increase the training time from5.5 hours reduced to 2.4 hoursThe average accuracy rate increased from 58.81% to 58.91%.

Figure 4: Three-layer, embedded dimension 128 Transformer on RTX 4090. Throughput. MiTA uses m*k≈N/4 to adjust batch size to maximize throughput.

As the sequence grows, the logical throughput advantage expands further: at 64K, 256K, and 1024K lengths, the acceleration ratio of full attention is about the same.19 times, 56 times and 159 times。

ArXiv long document classification: arXivClassification）It is necessary to identify the categories of long papers, which both require a comprehensive understanding and need to capture the fineness clues scattered in the text, and to test the combination of the overall generalization and detail retrieval.

Table 6: results of the arXiv long document classification (%)

At 16K, 32K and 64K lengths, the accuracy of the MiTA is achieved, respectively.88,0%, 92,9% and 98,7%The results of the study were published in the Official Journal of the European Union and published in the Official Journal of the European Union.18.2, 3.2 and 1.7 percentage points。

The first phenomenon is that without additional cutting modules, the target is gradually focused.As the network deepened, the key value positions of selected routing experts were gradually concentrated in category-related areas, and background positions were increasingly being selected, presenting emerging word-slicing effects (Emergent Token-Pruning Effect).

Figure 5: The leftmost is the input image, with the remaining columns showing the key values position of each layer selected by the expert, resulting in a concentration between the attention heads.

Unselected background words can still be included in the global summary through compressed branches.

The second phenomenon: weights can be reused and budgetary reasoning can be adjusted.The post-training freeze model parameters, which only replace the attention mechanism when reasoning, demonstrate better compatibility between MiTA and full attention.

Figure 6: Fixed model parameters, classification accuracy in training and reasoning using different attention mechanisms (%)

Further replacing the full-attention pre-trained ViT with MiTA and tweaking it to ImageNet-1K, ViT-S and ViT-L respectively reached the milestone.80.9% and 85.5%It's just a little bit of a distraction.0.3 and 0.4 percentage points。

In addition, by keeping the model parameters unchanged and only adjusting the number of experts m and expert width k when reasoning, the appropriate configuration can further improve accuracy.AlgorithmsAlgorithmic Generalization (algorithmic generalization)For example,Cost control in training, budget adjustment in reasoningIt is possible.

Figure 7: MiTA accuracy below different m, k. The red box marks the training configuration, and the orange border circle gives a higher accuracy than the original configuration.

So back to the beginning of the question:「AttentionThe value of FFN" is that not only are the two formulas compatible, but the efficiency of attention issues are given a new set of design dimensions.

Compression determines how global information is retained, what original content should be organized together, and which queries can be replicated by routers. These questions, attempts at design, and accumulated experience in the FFN can be used to counter the further development of attention mechanisms, even one day.Breaking upAttentionThe absolute boundary between the two and the FFN。

This work offers not only a variation of attention, but also a perspective on the belief in the design underlying the vertical architecture.