Sparse Attention in DeepSeek: NSA, DSA and V3.2

How DeepSeek stopped each new token from reading the whole conversation: NSA, the token scorer of V3.2, its measured costs and the price cut.

On 29 September 2025, DeepSeek released a model called V3.2-Exp and announced "API prices cut by 50%+". Output tokens went from $1.68 to $0.42 per million. The experts were the same as before, and so was the size: 671 billion parameters.

The only change to the architecture was in one place: how each new token looks back at the tokens before it.

That change is called DeepSeek Sparse Attention, or DSA. It came out of a research paper from February 2025, Native Sparse Attention (NSA). Both answer one question. When a model has read 128,000 tokens, which of them does the next token really need to look at? DeepSeek shipped its answer together with a 75% cut in the output price.

This part builds on two earlier pages. Attention, Part 3 covers sliding windows and a first look at DSA. Part 2 of this series covers the MLA cache. Short reminders appear where they are needed.

Why attention gets expensive at long context

Two phases, two different costs

A model that answers a question works in two phases. First it reads the whole prompt in one pass. Then it writes the answer, one token at a time. The two phases have very different costs. The first is prefill, the second is decode (LLM Inference, Part 1 explains both).

In attention, each token makes a query. It compares the query with the key of every earlier token. The comparison gives a score. A softmax turns the scores into weights that add up to 1. The weights then mix the values of the earlier tokens into one output. So the work grows with the number of (query, key) pairs.

Prefill: the pairs grow with the square

Count the pairs in a prompt of 4 tokens. Token 1 looks at 1 token (itself), token 2 at 2, token 3 at 3, token 4 at 4. That is 1 + 2 + 3 + 4 = 10 pairs. For a prompt of LL tokens the count is:

P(L)=1+2+⋯+L=L (L+1)2P(L) = 1 + 2 + \dots + L = \frac{L\,(L+1)}{2}

Here P(L)P(L) is the number of (query, key) pairs and LL is the prompt length. For L=4L = 4 it gives 4×5/2=104 \times 5 / 2 = 10, as counted. For L=131,072L = 131{,}072 (128K) it gives about 8.6 billion pairs, in every layer. Double the prompt and the pairs grow four times. This is the quadratic cost of prefill.

Decode: every new token reads the whole cache

Decode looks cheaper at first. One new token at position LL makes one query and compares it with LL keys. That is linear in LL, not quadratic. But it happens for every token of the answer, and each comparison needs the earlier tokens' records from memory.

The real numbers for DeepSeek-V3

Now put in V3's real sizes. V3 uses Multi-head Latent Attention (MLA, see Part 2). Instead of full keys and values per head, MLA stores one small latent vector per token per layer. It has 512 numbers, plus a 64-number key part that carries position (the RoPE key). All 128 query heads read that same 576-number entry.

MLA can run in two forms, and the DeepSeek-V3.2 paper draws them side by side.

Per (query, key) pair and per layer, the two forms cost these multiply-adds:

MQA mode: 128×(576+512)=139,264MHA mode: 128×(192+128)=40,960\text{MQA mode: } 128 \times (576 + 512) = 139{,}264 \qquad \text{MHA mode: } 128 \times (192 + 128) = 40{,}960

Read the first one left to right. There are 128 heads. Each head takes a 576-number dot product for the score, then adds a 512-number latent, scaled by its weight, into its output. The second one is the same with 192-number keys and 128-number values.

For one new token at position LL, the decode cost of attention over all 61 layers is then:

FLOPs per new token=2×61×139,264×L\text{FLOPs per new token} = 2 \times 61 \times 139{,}264 \times L

The 2 turns multiply-adds into FLOPs. At L=131,072L = 131{,}072 this is about 2.2 trillion FLOPs. Compare it with the rest of the model. V3 activates about 37 billion parameters per token, and each costs about 2 FLOPs, so the experts and projections cost about 74 billion FLOPs. Our calculator (part5_cost.py) prints both:

plain text
DECODE: one new token at position L, whole model (61 layers), attention core only
       L  dense GFLOP   DSA idx  DSA main   DSA tot dense/DSA  weights attn/weights
   8,192        139.2       8.2      34.8      43.0      3.2x       74         1.9x
  16,384        278.4      16.4      34.8      51.2      5.4x       74         3.8x
  32,768        556.7      32.7      34.8      67.5      8.2x       74         7.5x
  65,536      1,113.5      65.5      34.8     100.3     11.1x       74        15.0x
 131,072      2,226.9     131.0      34.8     165.8     13.4x       74        30.1x
Dense attention FLOPs equal the 2 x 37B weight FLOPs at L = 4,355 tokens

Look at the last column. At 8K tokens of context, dense attention already costs 1.9 times the whole rest of the model. At 128K it costs 30 times more. The mixture of experts made V3 cheap per token, but at long context attention eats that saving many times over. Above about 4,400 tokens, attention is the bigger bill.

04008001,2001,6002,0002,4008,19216,38432,76865,536131,072Dense MLA, 8,192: 139Dense MLA, 16,384: 278Dense MLA, 32,768: 557Dense MLA, 65,536: 1,113Dense MLA, 131,072: 2,227DSA, 8,192: 43DSA, 16,384: 51DSA, 32,768: 68DSA, 65,536: 100DSA, 131,072: 166Weights (37B active), 8,192: 74Weights (37B active), 16,384: 74Weights (37B active), 32,768: 74Weights (37B active), 65,536: 74Weights (37B active), 131,072: 74Dense MLA: 2,227DSA: 166Weights (37B active): 74context length (tokens, log scale)GFLOP per new token
Attention FLOPs for one new token of DeepSeek-V3, by context length (our calculator, attention core only). Dense MLA grows in a straight line with the context; DSA grows much more slowly; the expert and projection FLOPs stay at about 74 billion.

The bytes matter even more in decode

FLOPs are only half of the decode story. Each new token must also read the cache of every earlier token from memory. With the FP8 layout of DeepSeek's FlashMLA kernels, one token's entry is 656 bytes per layer. With plain BF16 it is 1,152 bytes.

plain text
DECODE: cache bytes read for one new token of one sequence (61 layers)
       L  dense FP8 MB  dense BF16 MB  DSA idx MB  DSA main MB  DSA tot MB dense/DSA
   8,192         327.8          575.7        66.0         82.0       147.9      2.2x
  16,384         655.6        1,151.3       131.9         82.0       213.9      3.1x
  32,768       1,311.2        2,302.7       263.8         82.0       345.8      3.8x
  65,536       2,622.5        4,605.3       527.7         82.0       609.6      4.3x
 131,072       5,245.0        9,210.7     1,055.4         82.0     1,137.3      4.6x
At 128K on one H200 (4.8 TB/s): reading the dense FP8 cache takes 1.09 ms per new token, the DSA reads 0.24 ms (one sequence, before any batching)

At 128K, one new token of one conversation reads 5.2 GB of cache. On an H200, whose memory delivers 4.8 TB per second, that is about 1.1 ms of pure reading per token. A server decodes many conversations in one step, and each one brings its own cache. The weights are read once per step and shared by the whole batch, but the caches are not shared. So at long context the cache reads, not the weights, set the speed.

0.002.004.006.008.0010.008,19216,38432,76865,536131,072Dense, BF16 cache, 8,192: 0.58Dense, BF16 cache, 16,384: 1.15Dense, BF16 cache, 32,768: 2.30Dense, BF16 cache, 65,536: 4.61Dense, BF16 cache, 131,072: 9.21Dense, FP8 cache, 8,192: 0.33Dense, FP8 cache, 16,384: 0.66Dense, FP8 cache, 32,768: 1.31Dense, FP8 cache, 65,536: 2.62Dense, FP8 cache, 131,072: 5.24DSA, 8,192: 0.15DSA, 16,384: 0.21DSA, 32,768: 0.35DSA, 65,536: 0.61DSA, 131,072: 1.14Dense, BF16 cache: 9.21Dense, FP8 cache: 5.24DSA: 1.14context length (tokens, log scale)GB read per new token
Cache bytes read for one new token of one sequence, by context length (our calculator). The dense lines grow in proportion to the context. DSA still grows, because the indexer reads a small key for every token, but about 4.6 times more slowly at 128K.

The way out: most of the weight lands on a few tokens

Attention weights are not spread evenly. In a long document, a token puts most of its weight on a few places. These are often the last few words, a name mentioned earlier, or the question at the top. The other tokens get tiny weights. If the model knew in advance which tokens matter, it could skip the rest.

The hard part is the "in advance". To know which tokens get large weights, you normally have to compute all the scores, and that is the cost you wanted to avoid. Each method below is an answer to this puzzle. The picture shows the patterns on 16 tokens.

Denseevery earlier tokenSliding windowlast 4 tokensNSA selectionwhole blocks of 4DSA top-kany 5 single tokensRows are queries (newest at the bottom), columns are earlier tokens. Filled = computed.
Which (query, key) pairs get computed in four kinds of attention on 16 tokens. Dense computes every pair. A sliding window keeps the last few tokens. NSA's selection branch picks whole blocks. DSA picks single tokens anywhere.

A sliding window (second panel, see Attention, Part 3) is cheap, but it forgets anything older than the window. DeepSeek wanted to reach a fact 100,000 tokens back, so it needed selection that depends on the content.

There is a second trap. A model trained with full attention learns to rely on all its tokens. Cut most of them away after training, and quality drops.

Native Sparse Attention (February 2025)

NSA is a research paper by DeepSeek-AI and Peking University, posted on 16 February 2025 and published at ACL 2025. No released DeepSeek model used NSA exactly as published. But its ideas are the direct parents of DSA, and of the compressed attention in V4.

Native Sparse Attention: one query, three views of the pastearlier tokensblocks of l = 32 tokens (drawn as 4)Compressioneach block -> 1 keyby a small MLPcoarse view of allSelectiontop-n blocks, usingthe compressed scoresfine view of a fewSliding windowlast w = 512tokenslocal viewgated sumgates g in [0, 1] come from an MLP + sigmoidof the query token
NSA in one picture. The past is cut into blocks. Compression turns each block into one summary key. Selection reads a few whole blocks in full. A sliding window keeps the most recent tokens. Learned gates mix the three outputs.

The gated sum

Branch 1: compression

Branch 2: selection, in whole blocks

The compression branch does more than give a coarse view. Its attention scores already say which parts of the past matter. NSA reuses them as importance scores for selection blocks of 64 tokens. It then reads the top 16 blocks in full. So the scorer is free: it is a by-product of the compression branch.

Why whole blocks and not single tokens? The answer is the hardware.

A second hardware rule decides who chooses the blocks. When heads share one cache, as in GQA or MQA, NSA adds up the importance scores of all heads in a group. So the whole group picks the same blocks. If 16 heads each picked their own 16 blocks, the group could load 256 blocks, and the saving would be gone. DSA takes this rule to the end: in V3.2, all 128 heads of a layer share one selection.

Branch 3: the sliding window

The numbers

From the settings, the number of keys one query reads at position tt is:

Nt=td+n l′+wN_t = \frac{t}{d} + n\,l' + w

Read it left to right. There are t/dt/d compressed keys, one per stride of 16 tokens. There are n l′n\,l' selected keys, 16 blocks of 64. And there are ww window keys, 512 of them. At t=65,536t = 65{,}536:

Nt=65,53616+16×64+512=4,096+1,024+512=5,632N_t = \frac{65{,}536}{16} + 16 \times 64 + 512 = 4{,}096 + 1{,}024 + 512 = 5{,}632

Full attention reads 65,536 keys at that position, so NSA reads 11.6 times fewer. The exact count of compressed blocks is ⌊(t−l)/d⌋+1=4,095\lfloor (t-l)/d \rfloor + 1 = 4{,}095, one fewer. The paper rounds to t/dt/d, and our calculator prints both. Table 4 of the paper gives the same numbers.

8K full8K full: 8,1928,1928K NSA8K NSA: 2,0482,04816K full16K full: 16,38416,38416K NSA16K NSA: 2,5602,56032K full32K full: 32,76832,76832K NSA32K NSA: 3,5843,58464K full64K full: 65,53665,53664K NSA64K NSA: 5,6325,632NSA: t/16 compressed + 16 x 64 selected + 512 window keys (paper settings)
Keys read per decoding step by full attention and by NSA, at 8K to 64K tokens (our calculator with the paper's settings; the same numbers as the paper's Table 4).

The training kernels were timed for real.

And quality? The test model was a 27-billion-parameter mixture of experts with 3 billion active parameters, trained from scratch. NSA matched or beat the same model with full attention. It scored higher on 7 of 9 general benchmarks and got perfect retrieval in a 64K needle-in-a-haystack test. On the LongBench long-context suite it averaged 0.469 against 0.437 for full attention.

These are the authors' own results on their own model. They show that native training works at that scale, not that NSA beats full attention in general.

From NSA to DSA: what DeepSeek kept and what it dropped

Seven months later, DSA appeared in DeepSeek-V3.2-Exp. It is not NSA with new numbers. It keeps one idea, drops two, and adds one.

NSA (Feb 2025)DSA (Sep 2025)
What is selectedwhole blocks of 64 tokenssingle tokens
Who scoresthe compression branch, for freea separate lightning indexer
Branchescompression + selection + window, gatedone: attention over the top-k
Shared selectionper group of headsall 128 heads of a layer
How trainedfrom scratchcontinued training of an existing model
Tested on27B research modelthe 671B production model

The biggest change is the last row. DeepSeek did not want to train a new 671B model from scratch to try sparse attention. It wanted to add sparsity to V3.1-Terminus, a model already trained with dense MLA. That decided the design. The new parts had to fit around MLA without changing it, and they had to learn in continued training.

DeepSeek Sparse Attention in V3.2-Exp

One DeepSeek-V3.2 layer (x 61). Only the box at the bottom is new.RMSNormMLA attentionsame as V3RMSNormMoE FFN1 shared + 8 of 256lightning indexer+ top-k selectortells MLA which 2,048 cached latentsto read; about 14M weights per layer,about 0.85B in all (on top of 671B)Experts, router, MTP, tokenizer and the 128K context are unchanged from V3.1-Terminus.
Where DSA sits in one of DeepSeek-V3.2's 61 layers. The lightning indexer and top-k selector attach to the MLA attention and tell it which cached entries to read. Everything else is V3.1-Terminus.

The new weights are small. By our count from the V3.2-Exp reference code, the indexer adds about 14 million weights per layer, about 0.85 billion over 61 layers. That is 0.13% on top of 671 billion. DSA has two parts: the lightning indexer, and the top-k selection that uses its scores.

Part 1: the lightning indexer

Read the formula left to right:

It,s=∑j=1HIwt,jI⋅ReLU(qt,jI⋅ksI)I_{t,s} = \sum_{j=1}^{H^I} w^I_{t,j} \cdot \mathrm{ReLU}\big(q^I_{t,j} \cdot k^I_s\big)
  • It,sI_{t,s} is the score that query token tt gives earlier token ss.
  • HIH^I is the number of indexer heads (64 in V3.2), and jj counts them.
  • qt,jIq^I_{t,j} is head jj's query, a vector of dI=128d^I = 128 numbers made from token tt.
  • ksIk^I_s is the indexer key of token ss, also 128 numbers, and the same for every head.
  • wt,jIw^I_{t,j} is one number per head, also made from token tt: how much this query trusts head jj.

A tiny example by hand. Take 2 heads and vectors of 2 numbers. The query token makes q1=(1,0)q_1 = (1, 0), q2=(0,1)q_2 = (0, 1) and weights w=(2,1)w = (2, 1). There are three earlier tokens with keys kA=(3,−1)k_A = (3, -1), kB=(−2,2)k_B = (-2, 2) and kC=(1,1)k_C = (1, 1).

Tokenq1⋅kq_1 \cdot kReLUq2⋅kq_2 \cdot kReLUI=2×ReLU1+1×ReLU2I = 2 \times \text{ReLU}_1 + 1 \times \text{ReLU}_2
A33-106
B-20222
C11113

With k=2k = 2, the selection keeps A and C. Head 1 liked A strongly, and head 2's dislike of A was cut to 0 by ReLU, not counted against it. Each head can vote for tokens, but it cannot veto them. The per-query weights then decide whose votes count more.

The real thing. The V3.2-Exp reference code on Hugging Face shows how each piece is built. This is the core of its Indexer.forward, unchanged:

python
q = self.wq_b(qr)                                   # qr: MLA's 1,536-number query latent
q = q.view(bsz, seqlen, self.n_heads, self.head_dim)  # 64 heads x 128
...
k = self.wk(x)                                      # x: the 7,168-number hidden state -> ONE 128-number key
k = self.k_norm(k)
...
q = rotate_activation(q)                            # a Hadamard rotation before 8-bit rounding
k = rotate_activation(k)
q_fp8, q_scale = act_quant(q, block_size, self.scale_fmt)
k_fp8, k_scale = act_quant(k, block_size, self.scale_fmt)
self.k_cache[:bsz, start_pos:end_pos] = k_fp8       # the indexer keeps its own FP8 key cache
...
weights = self.weights_proj(x.float()) * self.n_heads ** -0.5
...
topk_indices = index_score.topk(min(self.index_topk, end_pos), dim=-1)[1]

Four details are worth seeing. First, the indexer query is not made from scratch. It reuses MLA's compressed query, so it costs one extra matrix, 1,536 by 8,192 numbers. Second, the key is one 128-number vector per token, with a normalisation, and RoPE on its first 64 numbers (left out above).

Third, both are rotated by a Hadamard transform and then rounded to FP8. Our reading: the rotation spreads large outlier values over all 128 numbers, so 8-bit rounding loses less. Fourth, the indexer keeps its own key cache in FP8, next to MLA's cache. Remember that cache; it comes back when we count bytes.

Part 2: top-k selection, shared by all heads

With k=2,048k = 2{,}048 and the selection in place, the attention of query token tt becomes:

ut=Attn(ht, { cs∣It,s∈Top-k(It,:) })u_t = \mathrm{Attn}\big(h_t,\ \{\, c_s \mid I_{t,s} \in \text{Top-}k(I_{t,:}) \,\}\big)

Here utu_t is the attention output for token tt and hth_t is its hidden state. The set holds the cached entries whose index score is among the kk largest. For a query at position t≤2,048t \le 2{,}048 the set holds every earlier token, so short contexts get dense attention.

Why can DSA pick single scattered tokens when NSA insisted on blocks? Because of the MQA mode.

One query token through DSA (one layer of DeepSeek-V3.2)query token h_tevery earliertoken h_s64 queries q^I (128 dims)+ 64 weights wONE key k^I (128 dims)FP8, kept in a cacheI = sum_j w_j ReLU(q_j . k)one score per earlier tokenkeep the top 2,048MLA core attention, MQA mode128 heads read the same2,048 latents (656 bytes each)outputTop: cheap, but touches every earlier token. Bottom left: expensive, but touches only 2,048.
One query token through DSA in one layer of DeepSeek-V3.2. The indexer path is cheap but touches every earlier token. The core attention is expensive but touches only the 2,048 selected latents.

How much does this save?

Per (query, key) pair and layer, the indexer does 64×128=8,19264 \times 128 = 8{,}192 multiply-adds, in FP8. MQA-mode MLA does 139,264, in BF16. That is 17 times fewer operations, and FP8 arithmetic runs about twice as fast. So the quadratic part is roughly 30 to 35 times cheaper per pair. This is our estimate from the configuration, not a number from the paper.

Now redo the prefill count for a whole prompt, with the calculator:

plain text
PREFILL: a whole prompt of L tokens, whole model, attention core only (TFLOP)
       L  dense MHA  dense MQA   DSA idx  DSA main   DSA tot  MHA/DSA  weights
   8,192      167.7      570.2      33.5     249.4     283.0     0.6x    606.2
  16,384      670.7    2,280.5     134.1     534.5     668.6     1.0x  1,212.4
  32,768    2,682.9    9,121.8     536.6   1,104.6   1,641.2     1.6x  2,424.8
  65,536   10,731.4   36,486.8   2,146.3   2,244.8   4,391.1     2.4x  4,849.7
 131,072   42,925.3  145,945.9   8,585.1   4,525.2  13,110.2     3.3x  9,699.3

Per-query break-even, dense MHA-mode vs DSA: t = 8,704 tokens (7,737 if the FP8 indexer counts half)
Below that, a dense pass is cheaper: why V3.2 runs short prefills in a masked MHA mode.

Look at the 8K row. DSA costs more than dense prefill: 283 against 168 trillion FLOPs. The reason is the MHA/MQA split from earlier. Dense prefill uses the cheap MHA mode at 40,960 multiply-adds per pair. DSA's selected attention uses MQA mode at 139,264 per pair, 3.4 times more. At short lengths, the saving from skipping tokens is smaller than that penalty.

The break-even per query follows from one equation. Dense MHA mode costs 40,960 t40{,}960\,t for a query at position tt. DSA costs 8,192 t+139,264×2,0488{,}192\,t + 139{,}264 \times 2{,}048. They are equal at t=8,704t = 8{,}704, or 7,737 if FP8 counts half.

Below that, dense is cheaper. That is exactly the masked MHA mode in the paper. For short prompts, run dense and mask out the unselected tokens. It gives the same answer for less work.

At 128K, DSA's prefill is 3.3 times cheaper than dense MHA mode, by our count.

0.010.020.030.040.08,19216,38432,76865,536131,072Dense (MHA mode), 8,192: 0.2Dense (MHA mode), 16,384: 0.7Dense (MHA mode), 32,768: 2.7Dense (MHA mode), 65,536: 10.7Dense (MHA mode), 131,072: 42.9DSA, 8,192: 0.3DSA, 16,384: 0.7DSA, 32,768: 1.6DSA, 65,536: 4.4DSA, 131,072: 13.1Weights (37B active), 8,192: 0.6Weights (37B active), 16,384: 1.2Weights (37B active), 32,768: 2.4Weights (37B active), 65,536: 4.8Weights (37B active), 131,072: 9.7Dense (MHA mode): 42.9DSA: 13.1Weights (37B active): 9.7prompt length (tokens, log scale)PFLOP per prompt
Attention compute for a whole prompt of DeepSeek-V3 (our calculator). Dense MHA-mode prefill grows with the square of the length. DSA starts more expensive and wins after a few thousand tokens. At 128K it is 3.3 times cheaper.

Teaching the indexer: two stages of continued training

A new indexer starts with random weights. It must learn which tokens the model's own attention cares about. DeepSeek teaches it by imitation, in two stages, starting from the V3.1-Terminus checkpoint with its 128K context.

From V3.1-Terminus to V3.2: continued training with sparse attentionV3.1-Terminusdense MLA128K context1. Dense warm-uptrain ONLY the indexerKL to head-summed attention1,000 steps, 2.1B tokens2. Sparse trainingtrain everything, top-2,048indexer: KL on chosen set15,000 steps, 943.7B tokens3. Post-trainingdistillation + RLwith DSA onTokens per stage, to scale:warm-up 0.2%sparse stage 99.8%
The path from V3.1-Terminus to V3.2. A short dense warm-up trains only the indexer. A long sparse stage trains everything with top-2,048 selection switched on. Post-training then runs with DSA as well. The bar shows the tokens of the two stages to scale.

Stage 1: dense warm-up

A tiny example. Say the target over three tokens is p=(0.7,0.2,0.1)p = (0.7, 0.2, 0.1) and the indexer's softmax gives q=(0.5,0.3,0.2)q = (0.5, 0.3, 0.2). Then:

DKL(p ∥ q)=0.7ln⁡0.70.5+0.2ln⁡0.20.3+0.1ln⁡0.10.2=0.236−0.081−0.069=0.085D_{\mathrm{KL}}(p \,\|\, q) = 0.7 \ln\tfrac{0.7}{0.5} + 0.2 \ln\tfrac{0.2}{0.3} + 0.1 \ln\tfrac{0.1}{0.2} = 0.236 - 0.081 - 0.069 = 0.085

The first term dominates: the token the teacher cares about most is under-scored. The gradient pushes the indexer to raise that token's score. Note the direction of the KL. With pp first, missing a token the teacher needs costs a lot, while giving a little score to an unneeded token costs little. For a selector that keeps 2,048 tokens, that is the right bias.

Stage 2: sparse training

Check the arithmetic again: 15,000 steps times 480 sequences is 7.2 million sequences, and times 131,072 tokens that is 943.7 billion tokens. The warm-up is 0.2% of the total, as the bar in the figure shows. The data in both stages is "totally aligned" with V3.1-Terminus's 128K long-context extension data, so the only new thing the model sees is the sparsity.

Our experiment: a lightning indexer from scratch

Attention, Part 3 trained an indexer on a real 0.5B model. Here we do something smaller and more controlled. We build the whole DSA path from scratch on random tensors, with V3.2's shape in miniature, and measure everything. The script is part5_indexer.py and runs on a laptop CPU in under a minute.

The setting has 4,096 cached tokens and 16 query heads that share one 32-number latent per token (MQA mode). The indexer has 16 heads of 16 numbers. Each query keeps k=64k = 64 tokens. That is 1.56% of the cache, the same share as 2,048 of 131,072 in V3.2. The "main model" is two fixed random matrices. The hidden states are random, plus one shared mean direction, because real hidden states are not centred on zero.

The indexer is the formula, line for line:

python
class Indexer(torch.nn.Module):
    """I[t,s] = sum_j w[t,j] * ReLU(q_I[t,j] . k_I[s])."""
    def __init__(self):
        super().__init__()
        self.wq = torch.nn.Linear(D, HI * DI, bias=False)          # query token -> HI small queries
        self.wk = torch.nn.Linear(D, DI, bias=False)               # earlier token -> ONE small key
        self.ww = torch.nn.Linear(D, HI, bias=False)               # query token -> one weight per head

    def forward(self, h_q, h_kv, fp8=False):
        B = h_q.shape[0]
        q = self.wq(h_q).view(B, T, HI, DI)                        # (B,T,HI,DI)
        k = self.wk(h_kv)                                          # (B,L,DI)
        if fp8:                                                    # store and multiply in 8-bit floats
            q = q.to(torch.float8_e4m3fn).float()
            k = k.to(torch.float8_e4m3fn).float()
        w = self.ww(h_q)                                           # (B,T,HI)
        dots = torch.einsum('btjd,bsd->btjs', q, k)                # (B,T,HI,L)
        return torch.einsum('btj,btjs->bts', w, F.relu(dots))      # (B,T,L): the index scores

The sparse attention gathers the 64 selected latents and attends over them only:

python
def sparse_attention(q, c, idx):
    """Method 1: gather the k selected latents and attend over them only. idx: (B,T,K)."""
    B = q.shape[0]
    sel = torch.gather(c.unsqueeze(1).expand(B, T, L, DC), 2, idx.unsqueeze(-1).expand(B, T, K, DC))  # (B,T,K,DC)
    s = TEMP * torch.einsum('bhtc,btkc->bhtk', q, sel) / math.sqrt(DC)
    a = torch.softmax(s, -1)
    return torch.einsum('bhtk,btkc->bthc', a, sel)                 # (B,T,H,DC)

Then the script runs the dense warm-up of Stage 1. The main model is frozen, and only the indexer trains, with DKL(p ∥ Softmax(I))D_{\mathrm{KL}}(p \,\|\, \mathrm{Softmax}(I)), for 400 steps on fresh random sequences. It tests on a fixed batch that it never trained on. Here is the full output:

plain text
Setup: 4,096 cached tokens, 32 queries per sequence, 16 heads sharing one 32-dim latent, indexer 16 heads x 16 dims, keep k = 64 (1.56%)
Dense attention: the best 64 tokens hold 90.1% of the head-summed attention on average

warm-up step    1: KL loss 11.653 per query
warm-up step   25: KL loss 4.727 per query
warm-up step  100: KL loss 2.308 per query
warm-up step  200: KL loss 1.855 per query
warm-up step  400: KL loss 1.134 per query
warm-up took 37 s on CPU

selector                attention kept  recall of true top-k  output error
random tokens                     1.5%                  1.6%         71.9%
untrained indexer                 2.0%                  2.0%         74.1%
trained indexer                  77.1%                 56.6%         17.2%
oracle (true top-k)              90.1%                100.0%          5.0%

k =    16 ( 0.39% of tokens): attention kept, trained indexer  52.9%, oracle  73.2%, random   0.4%
k =    32 ( 0.78% of tokens): attention kept, trained indexer  65.6%, oracle  83.0%, random   0.8%
k =    64 ( 1.56% of tokens): attention kept, trained indexer  77.1%, oracle  90.1%, random   1.6%
k =   128 ( 3.12% of tokens): attention kept, trained indexer  86.2%, oracle  94.8%, random   3.1%
k =   256 ( 6.25% of tokens): attention kept, trained indexer  92.8%, oracle  97.7%, random   6.2%
k =   512 (12.50% of tokens): attention kept, trained indexer  97.0%, oracle  99.1%, random  12.5%
k =  1024 (25.00% of tokens): attention kept, trained indexer  99.0%, oracle  99.8%, random  25.0%

FP8 indexer (float8_e4m3fn q and k): 95.2% of the selected tokens are the same as in FP32
Method 1 (gather 64 latents) vs Method 2 (SDPA with a mask over all 4,096): max difference 4.8e-06

Multiply-adds per query token:
  dense attention:          16 x (32+32) x 4,096 = 4,194,304
  DSA indexer:              16 x 16 x 4,096      = 1,048,576
  DSA main (top-64):        16 x (32+32) x 64     = 65,536
  DSA total:                1,114,112  (3.8x fewer than dense)

"Attention kept" is the share of the true head-summed attention that falls on the selected tokens. "Output error" is the gap between the sparse and the dense output, relative to the dense output.

  • Random tokens keep 1.5% of the attention, as expected from picking 1.56% of tokens blindly. The output is badly wrong (72% error).
  • The untrained indexer is no better than random.
  • The trained indexer keeps 77.1% of the attention after 37 seconds of warm-up. The output error falls to 17.2%. It finds only 56.6% of the exact top 64. But the tokens it misses have small weights, so most of the attention mass is still covered.
  • The oracle, the true top 64, keeps 90.1% with a 5.0% error. That is the ceiling for any selector at this kk.
0%20%40%60%80%100%1632641282565121,024Oracle (true top-k), 16: 73%Oracle (true top-k), 32: 83%Oracle (true top-k), 64: 90%Oracle (true top-k), 128: 95%Oracle (true top-k), 256: 98%Oracle (true top-k), 512: 99%Oracle (true top-k), 1,024: 100%Trained indexer, 16: 53%Trained indexer, 32: 66%Trained indexer, 64: 77%Trained indexer, 128: 86%Trained indexer, 256: 93%Trained indexer, 512: 97%Trained indexer, 1,024: 99%Random tokens, 16: 0%Random tokens, 32: 1%Random tokens, 64: 2%Random tokens, 128: 3%Random tokens, 256: 6%Random tokens, 512: 12%Random tokens, 1,024: 25%Oracle (true top-k): 100%Trained indexer: 99%Random tokens: 25%k, tokens kept out of 4,096 (log scale)% of attention kept
Share of the true attention held by the selected tokens, against k, in our toy (4,096 cached tokens). The trained indexer closes most of the gap to the oracle as k grows. Random selection keeps only k divided by 4,096.

Three more lines answer three more questions. FP8: rounding the indexer's queries and keys to 8 bits changes only 4.8% of the selected tokens. A ranking survives coarse numbers well, which is why DeepSeek can run the indexer in FP8. Method 2: PyTorch's own scaled_dot_product_attention, given a mask that allows only the selected tokens, gives the same output as our gather to within 4.8×10−64.8 \times 10^{-6}. So the from-scratch sparse attention is correct. And the multiply-adds: 3.8 times fewer than dense here.

Why only 3.8 times, against 13.4 for V3.2 at 128K? Our toy indexer is large next to its main attention: a ratio of 4 per pair, against 17 in V3.2. And its context is 32 times shorter.

What it costs: the measured curves

The calculator counts operations. DeepSeek measured money.

The figure has no table of numbers, but it is a vector drawing. The PDF stores each line as exact coordinates. So instead of reading the plot by eye, we read the coordinates with part5_fig3_read.py. It finds the grid lines for the scale, finds the blue and orange paths, and converts their corner points.

plain text
prefill  V3.1-Terminus  corner points (token position, $ per 1M tokens): [(0, 0.0529), (131072, 0.6657)]
prefill  V3.2           corner points (token position, $ per 1M tokens): [(0, 0.0611), (6746, 0.0999), (131072, 0.1871)]
prefill  at 128K: Terminus $0.666, V3.2 $0.187, ratio 3.6x; curves cross near 10,688 tokens
prefill  at 32K:  Terminus $0.206, V3.2 $0.118; at 64K: Terminus $0.359, V3.2 $0.141
decode   V3.1-Terminus  corner points (token position, $ per 1M tokens): [(2, 0.0706), (131070, 2.1543)]
decode   V3.2           corner points (token position, $ per 1M tokens): [(2, 0.0953), (2107, 0.1458), (104046, 0.2275), (131070, 0.2492)]
decode   at 128K: Terminus $2.154, V3.2 $0.249, ratio 8.6x; curves cross near 4,928 tokens
decode   at 32K:  Terminus $0.592, V3.2 $0.170; at 64K: Terminus $1.112, V3.2 $0.197

The corner points tell a story. Each curve is made of only two to four straight segments. So these are fitted cost models drawn as lines, not raw measurements with noise. Inside them, three facts stand out.

  1. At 128K, V3.2 is 3.6 times cheaper in prefill and 8.6 times cheaper in decode. Decode gains more because decode is memory-bound, and DSA cuts reads hardest.
  2. The decode curve of V3.2 bends at about 2,100 tokens. That is k=2,048k = 2{,}048. Below it, every token is selected anyway, and the cost rises like dense. Above it, the main attention stops growing.
  3. The prefill curve of V3.2 bends at about 6,700 tokens, and the two prefill curves cross near 10,700. Below the bend, DeepSeek runs the masked MHA mode, which costs about the same as Terminus. Our break-even estimate from operation counts was 7,700 to 8,700 tokens, in the same range. The real crossing depends on kernels, which our count ignores.

The decode curve of V3.2 also has a small bend at about 104,000 tokens, where it gets slightly steeper. The paper does not explain it, and we do not guess.

0.0x2.0x4.0x6.0x8.0x10.0x8,19232,76865,53698,304131,072Decode, 8,192: 1.3xDecode, 16,384: 2.1xDecode, 32,768: 3.5xDecode, 65,536: 5.7xDecode, 98,304: 7.3xDecode, 131,072: 8.6xPrefill, 8,192: 0.9xPrefill, 16,384: 1.2xPrefill, 32,768: 1.7xPrefill, 65,536: 2.5xPrefill, 98,304: 3.1xPrefill, 131,072: 3.6xDecode: 8.6xPrefill: 3.6xtoken positionTerminus cost / V3.2 cost
How many times cheaper a token is on V3.2 than on V3.1-Terminus, by its position, read from the vector data of the paper's Figure 3. Prefill crosses 1x near 10,700 tokens; decode gains much more at long positions.

What happened to the price

DeepSeek's release note for V3.2-Exp says it plainly: "DeepSeek API prices drop 50%+, effective immediately". DeepSeek also kept V3.1-Terminus on a temporary endpoint until 15 October 2025, so users could compare the two. The archived pricing pages give the exact prices in USD per million tokens:

V3.1 (from 5 Sep 2025)V3.2-Exp (from 29 Sep 2025)Cut
Input, cache hit$0.07$0.02860%
Input, cache miss$0.56$0.2850%
Output$1.68$0.4275%
Input, cache hit: V3.1Input, cache hit: V3.1: $0.07$0.07V3.2-ExpV3.2-Exp: $0.028$0.028Input, cache miss: V3.1Input, cache miss: V3.1: $0.56$0.56V3.2-ExpV3.2-Exp: $0.28$0.28Output: V3.1Output: V3.1: $1.68$1.68V3.2-ExpV3.2-Exp: $0.42$0.42USD per 1M tokens, from the archived pricing pages (Sept 2025 vs Oct 2025).
DeepSeek API prices per million tokens for V3.1 and for V3.2-Exp, from the archived pricing pages before and after the launch on 29 September 2025.

The output price fell the most, and decode is where Figure 3 shows the biggest saving. That fits, but keep the two things apart. A price is a business decision; Figure 3 is DeepSeek's own cost estimate at an assumed rental price.

The price is flat across positions, while the cost saving depends on context length. Short chats save little; long agent sessions and long reasoning chains save a lot. DeepSeek kept the same prices for V3.2 and V3.2-Speciale in December 2025.

DSA saves compute and reads, not storage

It is easy to read "sparse attention" as "smaller cache". For DSA, that is wrong. Every token's MLA latent must stay in memory, because any of them might be selected by a later query. And the indexer adds its own key per token per layer.

plain text
KV cache per token (61 layers):
  V3, BF16 latent:            61 x 576 x 2 B = 70,272 B
  V3, FlashMLA FP8 row:       61 x 656 B     = 40,016 B
  V3.2, FP8 row + indexer:    61 x (656 + 132) B = 48,068 B (+20.1%)
  at 128K tokens: V3 FP8 5.24 GB, V3.2 6.30 GB per sequence

The indexer key is 128 FP8 bytes plus one 4-byte scale per layer: 132 bytes. Added to the 656-byte FP8 row of MLA, V3.2 stores 48,068 bytes per token. This number is not only ours. The DeepSeek-V4.1-Flash paper's Figure 1 shows a bar for "DeepSeek-V3.2" at exactly 48,068 bytes per token. Our sum from the code matches it to the byte.

V3, BF16 latentV3, BF16 latent: 70,272 B70,272 BV3 / V3.1, FP8 layoutV3 / V3.1, FP8 layout: 40,016 B40,016 BV3.2, FP8 + indexer keyV3.2, FP8 + indexer key: 48,068 B48,068 B61 layers. Indexer key: 128 FP8 bytes + 4-byte scale per layer = +20%.
Bytes of cache per token. V3's BF16 latent, the same latent in DeepSeek's FP8 serving layout, and V3.2, which adds a 132-byte indexer key per layer. DSA makes the cache 20% larger, not smaller.

So what does DSA save? Look again at the bytes table from the start. At 128K, a new token reads 1,055 MB of indexer keys and only 82 MB of selected latents. That is 1,137 MB in all, against 5,245 MB for dense. Reads fall 4.6 times.

But 93% of what DSA still reads is the indexer's key cache, and that part still grows with the context. Decode with DSA is still linear in LL; the slope is about 5 times smaller.

That remaining linear term, the full-length indexer pass, is exactly what DeepSeek attacked next. V4's compressed sparse attention shrinks the cache itself, and V4.1 goes further. Part 6 follows that path.

DeepSeek-V3.2 and V3.2-Speciale (December 2025)

On 1 December 2025 DeepSeek released V3.2 and V3.2-Speciale, both under the MIT license, and the arXiv paper followed the next day. The architecture is the same as V3.2-Exp: DSA on MLA, 671B parameters, 37B active, 128K context. The difference is post-training. The paper says post-training also "employs sparse attention in the same way as the sparse continued pre-training stage".

Specialist models for maths, coding, reasoning and agent tasks generate training data. Then one mixed reinforcement-learning stage with GRPO trains reasoning, agent skills and alignment together. DeepSeek says its RL budget now "exceeds 10% of the pre-training cost". Part 4 covers that training in detail.

The paper reports Speciale at gold-medal level in four 2025 competitions. It scored 35 of 42 points at the IMO and 492 of 600 at the IOI. It also reached gold at the Chinese Mathematical Olympiad and solved 10 of 12 problems at the ICPC World Finals. It scored 96.0 on AIME 2025.

Speciale had no tool calls. It was served only through a temporary API endpoint until 15 December 2025, at the same price as V3.2. The regular V3.2 became the model behind DeepSeek's deepseek-chat and deepseek-reasoner API names.

Research notes: sources, versions and loose ends
  • Which document. V3.2-Exp came with its own technical report as a PDF in the GitHub repository (29 September 2025), "DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention". The arXiv paper 2512.02556 (2 December 2025) repeats its DSA sections and extends them. All quotes and page numbers here are from the arXiv version.
  • NSA training tokens. The NSA paper says the 27B model was pre-trained on "260B tokens" in §1 (page 3) and on "270B tokens of 8k-length texts" in §4.1 (page 10). We do not know which is right.
  • NSA decode speed. The 11.6x for decoding is the memory-access ratio of Table 4 ("expected speedup"), not a timed kernel. The forward (9.0x) and backward (6.0x) numbers are timed, on A100 GPUs with Triton kernels.
  • Indexer size. From the V3.2-Exp model.py shapes: wq_b 1,536 x 8,192, wk 7,168 x 128, weights_proj 7,168 x 64, plus a LayerNorm of 128. About 14.0M weights per layer, 0.85B over 61 layers. The model card still calls V3.2 a 671B model.
  • The 48,068 bytes. The indexer key cache layout (128 FP8 bytes + one float32 scale per 128 numbers) comes from the k_cache and k_scale_cache buffers in model.py. The MLA row (512 FP8 bytes, 16 bytes of scales, 128 bytes of BF16 RoPE key) comes from FlashMLA's README at commit ba89a34. The sum matches the V4.1-Flash paper's Figure 1(b).
  • Figure 3 values. Read from the PDF's vector paths, not by eye. The strokes of the decode panel stop 2 tokens short of the axis ends; we clamp to the drawn range.
  • Our calculator's limits. It counts the attention core only, in multiply-adds. It ignores the indexer's projections, softmax, top-k sorting, and kernel efficiency. It treats FP8 and BF16 operations as equal unless stated. "Weights" uses 2 x 37B FLOPs per token.
  • The API cut. Verified on DeepSeek's release note (news250929): "API prices cut by 50%+". The exact prices come from Wayback Machine captures of the pricing page on 6 September 2025 and 4 October 2025.

Next: Part 6, V4 and V4.1. DSA left the cache as large as before and the indexer still reading every token. V4 compresses the cache itself with hybrid compressed and sparse attention, and reaches a million tokens of context.

References

  1. J. Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. ACL 2025.
  2. DeepSeek-AI. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. 2025.
  3. DeepSeek-AI. DeepSeek-V3.2-Exp technical report (PDF) and repository. 2025.
  4. DeepSeek-AI. DeepSeek-V2 (MLA) and DeepSeek-V3 Technical Report. 2024.
  5. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, Figure 1(b). 2026.
  6. Hugging Face: DeepSeek-V3.2-Exp inference code, DeepSeek-V3.2, DeepSeek-V3.2-Speciale.
  7. Kernels: FlashMLA, DeepGEMM, TileLang DSA examples, DeepSelect.
  8. DeepSeek API news: V3.2-Exp release, 29 Sep 2025 and V3.2 release, 1 Dec 2025. Pricing captures: 6 Sep 2025, 4 Oct 2025.
  9. vLLM recipe: DeepSeek-V3.2-Exp.
  10. Our code, in code/deepseek/: part5_cost.py (the calculator), part5_indexer.py (the from-scratch indexer), part5_fig3_read.py (Figure 3 from the PDF's vectors).