The scarcity of attention reduces the computation of the main attention, but the index responsible for screening tokens may still be a bottleneck for long context reasoning. DoSA, developed by the Little Red Book Dots team, uses scores on the bottom, accelerates the indexing on the premise of maintaining the original Top-K selection results, and filters about 97% of candidates in advance in the test. Under the above context length of about 1M tokens, the first Dots3-Note-preview tokens, M-5.2 and DeepSeek-V3.2 delayed their respective SGLang baselines by about 37%, 46% and 48% respectively.
DoSA: Dots Sparse Attention
Authored by: Dots Studio
Address of the project:https://github.com/llm-sys/dosa-sglang
01 Attention is scarce, why is it slow?
The time taken by the large model from sending the request to receiving the first token is known as the first token delay (TTFT), which is the "keyword waiting time" the user feels. The longer the context, the more time it usually takes to pre-fill.
The DSA using DeepSeek-V3.2 eased the computational burden of mainstream attention: the indexer (Indexer) scored the candidate token, selected the K with the highest score, and then the mainstream focused on the selected content. However, the indexer itself became the main bottleneck. Under the upper context length of about 1M tokens, the Dots3-Note-preview, GLM-5.2 and DeepSeek-V3.2 Indexer accounted for 81.6% of the pre-filling time, 73.0% and 90.8%, respectively.
Figure 1 left:Indexer time is proportional to the time taken to pre-fill the GPU; right:Indexer theoretical matrix computation is proportional.
02 What's the difference between Accelerated Indexer and DoSA?
1 Explain the acceleration routes of different base model team indexes.
Methods such as layer-by-layer replication, reduction of index heads and block-by-block sharing, save computation by reducing the number of independent scores or changing the selection graph, but do not guarantee equality with the original tier-by-tier, query-by-query Top-K selection results using all index heads.
The key difference between DoSAs is that they retain the original scoring function and mathematically demonstrate which calculations can be omitted from the bottom line. A small number of indexes are only used to construct the conservative top line and do not replace the final score. Only candidates who are proven impossible to enter Top-K are excluded, while the remaining candidates still use all the indexes to score accurately.
03 Decide first whether to be elected and then decide whether to be elected
In order to understand DoSA, you only need to capture two quantities: Top-K entriesDevalue below the boundaryAnd every candidateScores are upIf a candidate's top score remains below the threshold, it can be determined that he or she cannot be elected, thus excluding his or her safety.
Figure 2 of DoSA's screening and refinement process: first calculate the scores required for the screening, use the upper bound to exclude the candidate, then calculate all the index heads for the reserved candidate, and finally sum the scores according to the original rules and write back.
DoSA first selects a group of seed candidates and re-sequences all the index heads of the current queries. The score of the largest K in the seed candidate can serve as the bottom line of the entry threshold, since at least one K candidate has already reached that score. The seed candidate can be selected based on historical selection frequency and can be generated from a small number of hotspot queries at cold start; any source needs to be re-sequited for the current queries.
Next, the DoSA constructs an upper bound for each candidate that is not less than their actual score. The total score of the Indexers is weighted together by the contributions of multiple index heads. The DoSA maintains a small number of important heads to be accurately calculated and deals with the contributions of the remaining positive and negative heads, respectively: for the positive heads, it is projected to the reserve heads and estimates the upper bound of the residual contributions; for the negative heads, it is conservatively estimated by using the sub-bound of the aggregated deductions.
Candidates are excluded only when the scoring limit is strictly below the threshold limit. The remaining candidates continue to score accurately using all the index heads, ultimately executing the Top-K selection according to the original score. The limit will only increase the clearance even if the estimate is more loose, and will not erase the possible candidates selected incorrectly.
04 After less computation, the GPU should be more efficient.
After reducing point-to-point computation, it is also necessary to make the GPU perform the remaining work efficiently. Candidates retained in different queries vary, and if processed individually, it is easy to produce scattered computations, repeated key loads, and a large number of intermediate results. For this purpose, DoSA organizes the process into three stages: process selection, candidate selection, and compact rewriting.
Figure 3 Selects, refines and rewrites data flows. Blue and orange represent continuous processing of data blocks, while gray MMA blocks represent the total computation of saved data. G2S represents the transfer of memory from universal memory to shared memory, and A2R represents the transfer of memory from aggregated hardware modules to repositories.
The screening phase still scans the current query's visible key, but only calculates the small amount of information needed to construct the upper boundary. Keeping the number of heads close to the query is organized as a set of data blocks, handed over by Tensor Core to perform matrix operations; data loading and computation are performed overlapping through flow lines. The results of the screening are recorded in a compact bit mask for use in the next phase.
The clearance phase uses candidate overlapping between different queries to reduce data transfer. In the implementation, each of the four queries is made up of a set. If a key is retained by at least two queries and all queries within the group are visible, the shared processing path is entered: only one load and then the group queries are scored. This increases the amount of point-to-point computation, but reduces repeat loading; the rest of the candidates are still processed according to the queries.
When rewriting, directly filter the scores below the threshold boundary, retaining only the records that could enter the final Top-K. Each record contains tightly encoded scores and an accurate key ID. First, the Top-K boundary is encoded and the original scores of the boundary candidates are compared to preserve the accuracy of the selection results.
This also reflects the difference between DoSA and LiteTopK. LiteTopK combines scoring with Top-K options, reducing intermediate data transfer; DoSA further skips unnecessary scoring while reducing computation.
In this context, what is the acceleration effect?
The GLM-5.2's complete Indexer microclimate covers five subtext lengths, four query block sizes and three input domains, for a total of 60 configurations. DoSA is faster in all configurations than SGLang and LiteTopK: the relative acceleration ratio of the two is 1.12 tonnes 2.74 and 1.17 tonnes 1.69 respectively.
Figure 4 shows the full pre-filled phase of the Indexer relative to the acceleration ratio of SGLang. The three lines correspond to the English, Chinese and code inputs respectively; each line compares the size of 4K, 8K, 16K, and 32K four query blocks, covering the upper subtext length of 128K and 1M tokens.
Level-by-level analysis shows that the average candidate filtering rate for the three domains of English, Chinese, and code is about 97%. Of the 21 Indexer output layers tested in each domain, 20 have a filtering rate of more than 95%.
Figure 5 illustrates GLM-5.2 The average candidate filtering rate for each of the Indexer output layers. English, Chinese and code are tested at 21 layers in each of the three domains, each with a filtering rate of more than 95% on 20 layers.
The end-to-end TTFT more directly reflects the user's wait time. Under the upper context length of about 1M tokens, Dots3-Note-preview, GLM-5.2 and DeepSeek-V3.2's first tokens were delayed by about 37%, 46% and 48% respectively over their respective SGLang baselines.
The lower the token delay of the first token of the Dots3-Note-preview, the better. At the upper context length of about 1M tokens, the relative acceleration ratio of SGLang is 1.58.
The lower the latency of the first token of GLM-5.2, the better. Under the above context length of the 1M token, the relative acceleration ratio of SGLang is 1.86.
The lower the latency of the first token in DeepSeek-V3.2, the better. Under the above context length of the 1M token, the relative acceleration ratio of SGLang is 1.94.
In these tests, as the context evolves, the Indexer increases the amount of time it takes to pre-fill, optimizing the end-to-end gains it brings. However, computations such as the main focus and front-end layers still need to be performed, so the acceleration of the end-to-end TTFT is lower than the acceleration rate of the Indexer itself.
Author's note
Deep waterHe was the first author to publish a number of papers at academic conferences such as NeurIPS, ICML, PiSSA, TransMLA, etc. Before DoSA, he had proposed ways to accelerate HISA, MISA, LISA, LatentIndex, etc.
AstrologyHe has published numerous papers at academic conferences such as MLSys, EMNLP, and other academic conferences, primarily focusing on big language models and their algorithms and system collaboration design.
© THE