DeepSeek-V4 and V4.1: A Million Tokens of Context

How DeepSeek cut the memory per token of context from 389,120 bytes to 890: compressed attention, stable wide residuals, FP4 weights and hashed n-gram memory.

A coding agent opens a large repository, reads every file, and keeps the result in its working memory: about one million tokens. With DeepSeek-V3's attention, the stored context for that one conversation takes 73.7 GB. That is almost all of an 80 GB GPU, for one user, before the model weights.

DeepSeek-V4.1-Flash, released in September 2026, stores the same million tokens in 0.93 GB. Per token, it keeps 890 bytes where DeepSeek's first big model kept 389,120. This part explains how each of those bytes went away. It also covers what else changed on the way: a new residual connection, 4-bit expert weights, a faster drafter and a hash-table memory.

Part 5 ended with V3.2, which made long-context attention cheap to compute but not cheap to store. This part starts there.

The 2026 releases

DeepSeek published two model generations in 2026, and several updates of each. The weights of all of them are open under the MIT license.

DeepSeek releases in 2026 (all open weights, MIT license)Apr 24V4 previewV4-Pro 1.6T / 49B, V4-Flash 284B / 13BJun 27DSpark checkpointsV4 + speculative-decoding drafterJul 31V4-Flash-0731official V4-Flash, with DSparkAug 13V4-Pro-0813official V4-Pro, with DSparkAug 21V4-Flash-Vision-Expfirst V4 that reads imagesSep 10V4.1-Flash552B + 196B Engram, 8B / 16B activeOct 17V4.1-Flash-1017minor update
The 2026 releases. V4 came as a preview in April and as official releases in July and August. V4.1-Flash followed in September (dates from DeepSeek's release notes and Hugging Face).
ModelReleasedTotal parametersActive per tokenContextRouted experts (top-k)Layers
V3 / V3.2 (for comparison)Dec 2024 / Dec 2025671B37B128K256 (8)61
V4-FlashApr 2026284B13B1M256 (6)43
V4-ProApr 20261.6T49B1M384 (6)61
V4.1-FlashSep 2026552B backbone + 196B Engram8B in prefill, 16B in decode1M384 (6)40

The totals and active counts are the papers' own numbers (V4 paper pages 1, 24 and 25; V4.1 paper pages 1 and 7). The layer and expert counts come from the config.json files on Hugging Face.

Three upgrades carry V4: a hybrid attention (CSA and HCA), a new residual connection (mHC), and a new optimizer, Muon. This part covers the first two; the optimizer changes training, not the model's shape.

Why a million tokens needs a new attention

The KV cache, again

When a model writes a token, attention compares the new token with every earlier token. The earlier tokens' keys and values never change, so the model stores them instead of computing them again.

We measure it in bytes per token: how much memory each token of context costs. The simplest case, a model where every layer stores a key and a value for each of its KV heads, gives:

bytes per token=2×L×Hkv×dh×b\text{bytes per token} = 2 \times L \times H_{kv} \times d_h \times b

Read it left to right. The 22 counts one key and one value. LL is the number of layers. HkvH_{kv} is the number of key-value heads per layer. dhd_h is the length of each head's vector. bb is the bytes per stored number: 2 for BF16, 1 for FP8, half a byte for FP4.

A tiny example first. A 2-layer model with 1 KV head of 4 numbers in BF16 stores 2×2×1×4×2=322 \times 2 \times 1 \times 4 \times 2 = 32 bytes per token. A 10-token prompt costs 320 bytes.

Now the real models. DeepSeek LLM 67B (2023) has 95 layers and 8 KV heads of 128 numbers in BF16:

2×95×8×128×2=389,120 bytes2 \times 95 \times 8 \times 128 \times 2 = 389{,}120 \text{ bytes}

DeepSeek-V3 uses multi-head latent attention (MLA). Each layer stores one 512-number latent plus one 64-number position key, in BF16 (Part 2; attention, Part 2):

61×(512+64)×2=70,272 bytes61 \times (512 + 64) \times 2 = 70{,}272 \text{ bytes}

What 70,272 bytes means at one million tokens

Multiply by the length. At 128K tokens, V3's cache is 9.21 GB per conversation. At 1M tokens (1,048,5761{,}048{,}576), it is 73.69 GB. That is one conversation filling a GPU.

Memory is only the first problem. To write each new token, the model must also read the cache, and reading memory is slow compared with arithmetic.

V3.2 attacked the reading cost with DeepSeek Sparse Attention (Part 5). A small "lightning indexer" scores every earlier token, and the main attention reads only the best 2,048. But V3.2 still stores every token, and the indexer adds its own key per token. In DeepSeek's serving format that is 656 bytes of FP8 latent plus 132 bytes of indexer key per layer:

61×(656+132)=48,068 bytes61 \times (656 + 132) = 48{,}068 \text{ bytes}

So V3.2 at one million tokens would still need 50.4 GB (V3.2 itself only supports 128K). To make a million tokens routine, the model must store fewer entries, not only read fewer.

V4's hybrid attention: CSA and HCA

The idea: store summaries, not every token

Think of minutes of a long meeting. You could keep a recording of every sentence. Or you could write one line for every four sentences, and one paragraph for every hour. To answer a question, you scan the hourly paragraphs for the right hour. Then you read the few one-line notes that matter, plus the last few sentences word for word.

V4 does exactly this with two kinds of attention layer.

One query at position 40. What it can read in each kind of layerCSA (ratio 4): every 4 tokens become one entry, the indexer picks the top-k entriestop-ktop-ktop-kwindowHCA (ratio 128): one entry per 128 tokens, all of them read, no indexerone heavily compressed entry (stands for 128 tokens)windowDrawn small: the real window is 128 tokens, V4-Flash keeps top-512 CSA entries, V4-Pro top-1024.
What one query can read. In a CSA layer, groups of 4 tokens become entries and the indexer picks a few; in an HCA layer, 128 tokens become one entry and all entries are read. Both layers add a window of recent tokens. Drawn small; real sizes in the note.

CSA, step by step

How does the compressor turn 4 tokens into one entry? It computes two candidate KV vectors for every token, series aa and series bb, plus a score for each. Entry ii mixes the 4 tokens of its own group from series aa and the 4 tokens of the previous group from series bb:

CiComp=∑j∈own mSja⊙Cja  +∑j∈previous mSjb⊙CjbC^{\text{Comp}}_i = \sum_{j \in \text{own } m} S^a_j \odot C^a_j \; + \sum_{j \in \text{previous } m} S^b_j \odot C^b_j

Here CjaC^a_j and CjbC^b_j are the two candidate vectors of token jj, each 512 numbers long. SjaS^a_j and SjbS^b_j are weights from one softmax over all 2m=82m = 8 scores, with a learned bias for each slot. The symbol ⊙\odot means "multiply number by number", so each of the 512 channels gets its own weights.

A tiny example with m=2m = 2 and one channel. Own tokens have values 4 and 8 with scores 0 and ln⁡3\ln 3. The previous tokens have values 2 and 6 with scores 0 and 0. The softmax over the four scores turns e0,eln⁡3,e0,e0=1,3,1,1e^0, e^{\ln 3}, e^0, e^0 = 1, 3, 1, 1 into weights 16,12,16,16\tfrac16, \tfrac12, \tfrac16, \tfrac16. The entry is 16⋅4+12⋅8+16⋅2+16⋅6=6\tfrac16 \cdot 4 + \tfrac12 \cdot 8 + \tfrac16 \cdot 2 + \tfrac16 \cdot 6 = 6. The token with the highest score dominates, but every token contributes.

CSA compression with m = 4: each entry mixes 8 tokens, but there is one entry per 4 tokens0123456789101112131415previous 4 (series b)own 4 (series a)entry 1previous 4 (series b)own 4 (series a)entry 2previous 4 (series b)own 4 (series a)entry 3One softmax over the 8 scores decides how much each token adds. Neighbouring entries share 4 tokens,so a boundary between groups does not cut a phrase in half. HCA uses one series and no overlap.
CSA builds entry i from its own 4 tokens and the 4 tokens before them. Neighbouring entries overlap, but there is still one entry per 4 tokens.

Why read 8 tokens to make an entry that replaces 4? A hard cut every 4 tokens would split phrases at random: "New" could land in one group and "York" in the next. With the overlap, a phrase that crosses a boundary still reaches one entry whole. The paper notes that CSA "in fact compresses the sequence length to 1/m times": the overlap costs compute, not memory.

Two more details make the stored entries small. First, CSA and HCA use one shared KV head, as in multi-query attention. All 64 or 128 query heads read the same entry.

V4 goes one step further than classic multi-query attention: the same 512-number entry serves as both the key and the value (attention, Part 2).

Second, V4 stores the entry in mixed precision. The 448 numbers that carry content go in FP8, and the 64 numbers that carry position (RoPE) stay in BF16. With 8 bytes of scale factors, one entry takes 448+64×2+8=584448 + 64 \times 2 + 8 = 584 bytes (FlashMLA's documented layout).

HCA: read everything, because there is little to read

The window, and why it is needed

V4 adds two smaller tricks to the core attention. An attention sink is one learned score per head, added to the softmax's denominator. With it, a head can give almost no weight to anything when nothing is relevant. And the output of each head is rotated back by its own position, so the result carries relative, not absolute, position.

How the layers combine

Which attention each layer uses (read from compress_ratios in config.json)V4-Flash, 43 layers2 window-only, 21 CSA, 20 HCAlayer 0: SWAlayer 1: SWAlayer 2: CSAlayer 3: HCAlayer 4: CSAlayer 5: HCAlayer 6: CSAlayer 7: HCAlayer 8: CSAlayer 9: HCAlayer 10: CSAlayer 11: HCAlayer 12: CSAlayer 13: HCAlayer 14: CSAlayer 15: HCAlayer 16: CSAlayer 17: HCAlayer 18: CSAlayer 19: HCAlayer 20: CSAlayer 21: HCAlayer 22: CSAlayer 23: HCAlayer 24: CSAlayer 25: HCAlayer 26: CSAlayer 27: HCAlayer 28: CSAlayer 29: HCAlayer 30: CSAlayer 31: HCAlayer 32: CSAlayer 33: HCAlayer 34: CSAlayer 35: HCAlayer 36: CSAlayer 37: HCAlayer 38: CSAlayer 39: HCAlayer 40: CSAlayer 41: HCAlayer 42: CSAV4-Pro, 61 layers30 CSA, 31 HCAlayer 0: HCAlayer 1: HCAlayer 2: CSAlayer 3: HCAlayer 4: CSAlayer 5: HCAlayer 6: CSAlayer 7: HCAlayer 8: CSAlayer 9: HCAlayer 10: CSAlayer 11: HCAlayer 12: CSAlayer 13: HCAlayer 14: CSAlayer 15: HCAlayer 16: CSAlayer 17: HCAlayer 18: CSAlayer 19: HCAlayer 20: CSAlayer 21: HCAlayer 22: CSAlayer 23: HCAlayer 24: CSAlayer 25: HCAlayer 26: CSAlayer 27: HCAlayer 28: CSAlayer 29: HCAlayer 30: CSAlayer 31: HCAlayer 32: CSAlayer 33: HCAlayer 34: CSAlayer 35: HCAlayer 36: CSAlayer 37: HCAlayer 38: CSAlayer 39: HCAlayer 40: CSAlayer 41: HCAlayer 42: CSAlayer 43: HCAlayer 44: CSAlayer 45: HCAlayer 46: CSAlayer 47: HCAlayer 48: CSAlayer 49: HCAlayer 50: CSAlayer 51: HCAlayer 52: CSAlayer 53: HCAlayer 54: CSAlayer 55: HCAlayer 56: CSAlayer 57: HCAlayer 58: CSAlayer 59: HCAlayer 60: CSAV4.1-Flash, 40 layers: 20 encoder + 20 decoderKV stored in 4 layers onlylayer 0: SWAlayer 1: SWAlayer 2: Flayer 3: Ulayer 4: Ulayer 5: Ulayer 6: Ulayer 7: Ulayer 8: Flayer 9: Ulayer 10: Ulayer 11: Ulayer 12: Ulayer 13: Ulayer 14: Flayer 15: Ulayer 16: Ulayer 17: Ulayer 18: Ulayer 19: Ulayer 20: Flayer 21: Ulayer 22: Ulayer 23: Ulayer 24: Xlayer 25: Ulayer 26: Ulayer 27: Ulayer 28: Xlayer 29: Ulayer 30: Ulayer 31: Ulayer 32: Xlayer 33: Ulayer 34: Ulayer 35: Ulayer 36: Xlayer 37: Ulayer 38: Ulayer 39: Uwindow onlyCSAHCACSA2 FullCSA2 ReindexCSA2 Reuse
The layer plans, read from compress_ratios in the configs. V4-Flash starts with two window-only layers, V4-Pro with two HCA layers; then both alternate CSA and HCA. V4.1-Flash is covered further down.

Counting the bytes

Now we can write V4's bytes per token. Each layer stores entries of ee bytes, one per mm tokens, so it costs e/me / m bytes per token:

bytes per token=∑layersemain+eindexerm\text{bytes per token} = \sum_{\text{layers}} \frac{e_{\text{main}} + e_{\text{indexer}}}{m}

For a CSA layer, emain=584e_{\text{main}} = 584 bytes. The indexer key is 128 numbers in MXFP4: 64 bytes of 4-bit values plus one 1-byte scale per 32 values, so eindexer=68e_{\text{indexer}} = 68. For an HCA layer, eindexer=0e_{\text{indexer}} = 0. The window layers add nothing per token.

Worked by hand for V4-Flash: 21×(584+68)/4=3,42321 \times (584 + 68) / 4 = 3{,}423 bytes from the CSA layers, and 20×584/128=91.2520 \times 584 / 128 = 91.25 bytes from the HCA layers. The total is 3,514 bytes, and that is exactly the number DeepSeek prints for V4-Flash. The indexer layout of 68 bytes is our reading of the MXFP4 format. The exact match with the published total is good evidence that it is right.

Here is the part of our script that does this from the config:

python
V4_MAIN = 448 * 1 + 64 * 2 + 8               # 448 FP8 + 64 BF16 RoPE + 8-byte scale row = 584
FP4_INDEX = 128 // 2 + 128 // 32             # 128 MXFP4 values (64 B) + 4 one-byte scales = 68

def v4(name):
    c = cfg(name)
    L = c['num_hidden_layers']
    ratios = c['compress_ratios'][:L]          # the last entry belongs to the MTP layer
    csa, hca, swa = ratios.count(4), ratios.count(128), ratios.count(0)
    b_csa = csa * (V4_MAIN + FP4_INDEX) / 4
    b_hca = hca * V4_MAIN / 128
plain text
  DeepSeek-V4-Flash: 43 layers = 2 sliding-window only + 21 CSA (ratio 4) + 20 HCA (ratio 128)
     CSA: 21 x (584 + 68) B / 4  = 3,423.00 B
     HCA: 20 x 584 B / 128       = 91.25 B
  DeepSeek-V4-Pro: 61 layers = 0 sliding-window only + 30 CSA (ratio 4) + 31 HCA (ratio 128)
     CSA: 30 x (584 + 68) B / 4  = 4,890.00 B
     HCA: 31 x 584 B / 128       = 141.44 B
V4-Pro (not in the chart): 5,031 B per token; V3.2 / V4-Pro = 9.55x, V3.2 / V4-Flash = 13.68x  (V4 paper Figure 1: "9.5x smaller", "13.7x smaller")
Bytes per token, split by the layers that store themV4-FlashV4-Flash, 21 CSA layers: 3,423.0 BV4-Flash, 20 HCA layers: 91.2 B3,514 B21 CSA layers: 3,423 B + 20 HCA layers: 91 BV4-ProV4-Pro, 30 CSA layers: 4,890.0 BV4-Pro, 31 HCA layers: 141.4 B5,031 B30 CSA layers: 4,890 B + 31 HCA layers: 141 BV4.1-FlashV4.1-Flash, 3 encoder Full layers: 534.0 BV4.1-Flash, 1 decoder Full layer: 356.0 B890 B3 encoder Full layers: 534 B + 1 decoder Full layer: 356 B
Where the bytes come from. In V4 the CSA layers hold almost everything; twenty or more HCA layers add under 150 bytes per token together. V4.1-Flash stores KV in only four layers (explained below).

The ratios match the paper's Figure 1: 48,068 / 3,514 is 13.68, and 48,068 / 5,031 is 9.55. Notice what the split shows. The HCA layers are almost free, so the 128-token view of the whole context costs nearly nothing. The real budget is the CSA layers, and that is where V4.1 will cut next.

mHC: a wider residual stream that cannot explode

Residual connections

Every Transformer layer has a bypass around it. The layer's input skips past the layer and is added to its output.

That untouched path is the reason deep networks train at all. Unroll the rule over many layers and the first input is still there, unchanged, in the last layer: xL=xl+∑F(…)x_L = x_l + \sum F(\ldots). Gradients flow back the same way. The mHC paper calls this the identity mapping property.

Hyper-connections: four streams instead of one

In 2024, researchers proposed widening the stream: keep nn copies of it, and let each layer mix them.

Residual connectionHyper-connections, n = 4 streamsx (one stream)layer F+x + F(x): the input passes through untouchedX: 4 copies, each d wideB mixes streamsA: 4 streams to 1layer FC: 1 back to 4+new X = B X + C F(A X)
A residual connection adds the layer's output to an untouched input. Hyper-connections keep four copies of the stream; B mixes them on the bypass, A reads one input from them and C writes the output back.

Read the update rule symbol by symbol:

Xl+1=BlXl+Cl Fl(AlXl)X_{l+1} = B_l X_l + C_l \, F_l(A_l X_l)

XlX_l is the residual state before layer ll: nn rows of dd numbers. AlXlA_l X_l is one row of dd numbers, the input that layer FlF_l sees. ClFl(⋅)C_l F_l(\cdot) copies the layer's output into the nn streams with nn weights. BlXlB_l X_l is the bypass, now a mix of streams instead of a plain copy. With n=1n = 1 and B=C=A=1B = C = A = 1, the rule is the ordinary residual connection.

Why unconstrained mixing explodes

The bypass is now a matrix. Unroll it over many layers and the first input reaches layer LL multiplied by the product BL−1⋯Bl+1BlB_{L-1} \cdots B_{l+1} B_l. Products of matrices are dangerous.

A tiny example. Let every layer use B=(1.10.10.11.1)B = \begin{pmatrix} 1.1 & 0.1 \\ 0.1 & 1.1 \end{pmatrix}. Each row sums to 1.2, so a signal [1,1][1, 1] grows by 1.2 per layer. After 40 layers it is 1.240≈1,4701.2^{40} \approx 1{,}470:

plain text
1) A 2 x 2 matrix that is a little too big: H = [[1.1, 0.1], [0.1, 1.1]], every row sums to 1.2
   after  1 layers the signal [1, 1] becomes [1.2, 1.2]
   after 10 layers the signal [1, 1] becomes [6.2, 6.2]
   after 20 layers the signal [1, 1] becomes [38.3, 38.3]
   after 40 layers the signal [1, 1] becomes [1,469.8, 1,469.8]

Each matrix looks harmless, yet 20% too much per layer becomes a factor of a thousand over depth. Learned matrices differ from layer to layer, so real growth is a random walk, but the danger is the same.

We reproduced the effect with random matrices: 4 streams, 120 mixing layers and 5 random seeds. A 60-block model has an attention and an MoE layer per block, so 120 mixing layers. HC uses I+0.2 GI + 0.2\,G, a matrix near the identity with random noise GG. mHC uses the projected matrix described next.

plain text
3) n = 4 streams, 120 mixing layers (a 60-block model: attention + MoE per block), noise scale 0.2, 5 seeds
   layer   HC gain (median / max of seeds)   mHC gain (median / max)
       1            1.8 /          2.3         1.000 /  1.000
      10            4.9 /          8.8         1.000 /  1.000
      30            7.1 /         14.1         1.000 /  1.000
      60           50.1 /         98.1         1.000 /  1.000
      90          165.3 /        330.5         1.000 /  1.000
     120          439.0 /        661.7         1.000 /  1.000
   largest single-layer spectral norm: HC 1.824, mHC 1.000104
   NumPy Sinkhorn vs PyTorch log-domain Sinkhorn: largest difference 4.4e-16
Largest gain of the product of 120 mixing matrices (4 streams, 5 random seeds)1101001,0001306090120layerHC, worst seed: 662HC, median: 439mHC, all seeds: 1.000
Our toy, on a log scale. The gain of unconstrained mixing matrices grows to hundreds over 120 layers; the projected (mHC) matrices hold it at 1 in every seed.

Our toy is not a trained model: the noise is random, not learned. It shows only the mechanism: small errors per layer compound. The paper's 3,000 comes from a real 27B training run.

The constraint: doubly stochastic mixing

mHC keeps the four streams and the mixing, but restricts what BlB_l can be.

Here is the mean-keeping property on numbers from our script. Mix the streams [4,2][4, 2] (mean 3) with the matrix above:

(0.3660.6340.6340.366)(42)=(2.7323.268)\begin{pmatrix} 0.366 & 0.634 \\ 0.634 & 0.366 \end{pmatrix} \begin{pmatrix} 4 \\ 2 \end{pmatrix} = \begin{pmatrix} 2.732 \\ 3.268 \end{pmatrix}

The streams changed; their mean is still 3.000. Each output is a weighted average of the inputs, with weights that sum to 1, so no output can be bigger than the biggest input.

Getting a doubly stochastic matrix: Sinkhorn-Knopp

The network produces a raw matrix of any numbers. A 1967 algorithm turns it into a doubly stochastic one.

Run it by hand on exp⁡(raw)=(1311)\exp(\text{raw}) = \begin{pmatrix} 1 & 3 \\ 1 & 1 \end{pmatrix}. The column sums are 2 and 4. Dividing gives (0.50.750.50.25)\begin{pmatrix} 0.5 & 0.75 \\ 0.5 & 0.25 \end{pmatrix}, whose rows sum to 1.25 and 0.75. Dividing each row by its sum gives (0.40.60.6670.333)\begin{pmatrix} 0.4 & 0.6 \\ 0.667 & 0.333 \end{pmatrix}. Now the columns are off a little: 1.067 and 0.933. Keep going:

python
def sinkhorn(raw, iters=20):
    """Method 1 (NumPy). exp() makes every entry positive; then alternate column and row normalisation.
    Same order as the paper's Eq. 8/9: M(t) = T_r(T_c(M(t-1)))."""
    M = np.exp(raw)
    for _ in range(iters):
        M = M / M.sum(axis=0, keepdims=True)     # T_c: every column sums to 1
        M = M / M.sum(axis=1, keepdims=True)     # T_r: every row sums to 1
    return M
plain text
2) Sinkhorn-Knopp on raw = [[0, ln 3], [0, 0]], so exp(raw) = [[1, 3], [1, 1]]
   step 1 columns: [[0.500, 0.750], [0.500, 0.250]]  row sums [1.25, 0.75]
   step 1 rows   : [[0.400, 0.600], [0.667, 0.333]]  column sums [1.067, 0.933]
   step 2 columns: [[0.375, 0.643], [0.625, 0.357]]  row sums [1.018, 0.982]
   step 2 rows   : [[0.368, 0.632], [0.636, 0.364]]  column sums [1.005, 0.995]
   step 3 columns: [[0.367, 0.635], [0.633, 0.365]]  row sums [1.001, 0.999]
   step 3 rows   : [[0.366, 0.634], [0.634, 0.366]]  column sums [1.0, 1.0]
   after 20 steps: [[0.3660, 0.6340], [0.6340, 0.3660]], rows [1.0, 1.0], columns [1.0, 1.0]
   mixing the streams [4, 2]: [2.732, 3.268], mean before 3.000, mean after 3.000
Sinkhorn-Knopp: divide columns by their sums, then rows, and repeatexp(raw)1311rows 4, 2 cols 2, 4columns / sum0.50.750.50.25rows 1.25, 0.75rows / sum0.40.60.6670.333cols 1.067, 0.933after 20 steps0.3660.6340.6340.366rows 1, 1 cols 1, 1
Sinkhorn-Knopp on a 2 x 2 matrix. Three rounds already give sums within 0.001 of 1; V4 runs 20 rounds for every 4 x 4 matrix, at every layer, for every token.

The error falls about tenfold per round, so three rounds are already close. The script also checks this NumPy version against a PyTorch version written in log space, the safe form for large exponents: they agree to 4.4×10−164.4 \times 10^{-16}.

What mHC costs and what it buys

The constraint has a price in time and memory. Four streams mean four times more data on the bypass, and the 20 Sinkhorn rounds run at every layer.

FP4 expert weights

Most of a V4 model's parameters sit in its routed experts. In V4-Flash they are 43×256×3×4,096×2,048=277.043 \times 256 \times 3 \times 4{,}096 \times 2{,}048 = 277.0 billion of the 284 billion, by our count from the config. Shrinking the experts shrinks the model. V4 stores them in FP4. On today's chips an FP4 x FP8 product runs at FP8 speed, so the gain is memory, not speed (V4 paper, page 5).

Our script checks the size this gives:

plain text
V4-Flash routed experts: 43 layers x 256 x 3 x 4096 x 2048 = 277.0B parameters
  in FP4 with one 1-byte scale per 32 values: 147.2 GB;  the other ~7B in FP8: ~7 GB;  estimate 154 GB  (published files: 159.6 GB)
  the same 284B in BF16 would be 568 GB

Our estimate of 154 GB is close to the 159.6 GB of the published V4-Flash files. We do not count the shared experts, norms, scales of the FP8 parts and file overhead, so the small gap is expected. The point is the other line: in BF16 the same model would be 568 GB. Part 7 turns these sizes into GPU counts.

DSpark and the official V4 releases

What DSpark adds

DSpark is V4's new draft model for speculative decoding (Part 2 shows how V3 drafted with its MTP module). It replaces that one-token drafter.

DeepSeek first shipped DSpark checkpoints for the V4 preview in June. Then came the official releases. V4-Flash-0731 "is the official release of DeepSeek-V4-Flash, superseding the preview version", with "a speculative decoding module attached" (model card). V4-Pro-0813 is "built on the DeepSeek-V4-Pro (Preview) model structure, with a DSpark speculative decoding module attached". Its config adds DSpark fields such as dspark_block_size: 5. In V4.1-Flash, DSpark replaces MTP completely: the backbone trains without MTP, and the drafter trains afterwards with the backbone frozen.

V4.1-Flash: 890 bytes per token

The causal encoder-decoder

V4.1-Flash: the decoder gets its long-range KV from the encoderprompt tokenscausal encoder, layers 0 to 19run on every prompt tokendecoder, layers 20 to 39run on the newest tokens onlyencoder output H (layer 19)projected once into decoder KVPrefill of a long prompt runs only the 20 encoder layersfor most tokens: about half the work (8B active, not 16B).Each decoder layer still keeps its own 128-token window.
In the causal encoder-decoder, the decoder layers project their long-range KV from the output of the last encoder layer. A long prompt mostly needs only the encoder half during prefill.

CSA2: one cache, many layers

V4 stored a separate cache in every CSA and HCA layer. V4.1 asks a sharper question: do neighbouring layers need different caches at all?

The decoder adds one more trick, the Hierarchical Sparse Indexer. The first decoder layer scores every position, then keeps the 2,048 best blocks of 8 positions: a pool of 16,384 candidates. Later Reindex layers score only that pool. Their cost per query no longer grows with the length of the context.

The last cut is precision. V4 stored the main entry in FP8 with BF16 for the position part. V4.1 stores all of it in FP4.

Now count V4.1's bytes. Each Full layer stores a 288-byte entry plus a 68-byte indexer key, once per mm tokens:

python
def v41():
    c = cfg('DeepSeek-V4.1-Flash')
    L = c['num_hidden_layers']
    ratios = c['compress_ratios'][:L]
    src = c['kv_source_layer_ids']
    total = 0
    for l in src:
        m = ratios[l]
        b = (V41_MAIN_FP4 + FP4_INDEX) / m
        total += b
plain text
  V4.1-Flash: 40 layers, compress ratios [0, 1, 2], KV stored only in layers [2, 8, 14, 20]
     layer  2 (encoder, ratio 2): (288 + 68) B / 2 = 178.0 B
     layer  8 (encoder, ratio 2): (288 + 68) B / 2 = 178.0 B
     layer 14 (encoder, ratio 2): (288 + 68) B / 2 = 178.0 B
     layer 20 (decoder, ratio 1): (288 + 68) B / 1 = 356.0 B

Three encoder layers give 3×356/2=5343 \times 356 / 2 = 534 bytes, the decoder layer gives 356, and the total is 890 bytes: the paper's number exactly. Notice that the compression ratios went down, from 4 to 2 and even 1. V4.1 stores finer detail than V4's CSA layers. It still uses a quarter of the memory, because it stores it in 4 layers instead of 41.

Single-Pass mHC

V4.1 keeps mHC but changes one index in the formula, to save memory traffic at inference.

Engram: memory you look up, not compute

Some knowledge is a fixed fact about a short phrase. After "Alexander the", "Great" is very likely. After "import numpy as", "np" is almost certain. A Transformer stores such facts in its weights, and finds them by running several layers of attention and MoE. Engram asks: why compute a lookup?

Here is the hash from our toy, written like the official Engram demo code. It multiplies each token ID by a fixed random odd number, combines with XOR, and takes the remainder modulo a prime.

python
def rows_for(seq, t, n):
    """The suffix n-gram ending at position t -> one row number per hash head."""
    gram = seq[max(0, t - n + 1): t + 1]
    gram = [0] * (n - len(gram)) + gram                                  # pad at the start of the text
    mix = 0
    for k, tok in enumerate(reversed(gram)):                             # offset 0 = current token
        mix ^= int(tok) * int(mult[k])
    return [mix % p for p in PRIMES[n]], gram
plain text
1) tokenizer compression
   raw IDs      : [0, 1, 2, 3, 4, 5, 2, 6, 7, 8, 9, 6, 10, 1, 2, 3]   (11 distinct)
   canonical IDs: [0, 1, 2, 3, 4, 5, 2, 6, 7, 8, 2, 6, 9, 1, 2, 3]   (10 distinct; "The" and "the" now share one ID)

2) lookups for the last tokens of "... obeyed Alexander the Great"
   2-gram "the Great" -> canonical IDs [2, 3] -> rows [283, 142] in tables of size [1009, 1013]
   3-gram "Alexander the Great" -> canonical IDs [1, 2, 3] -> rows [83, 847] in tables of size [1019, 1021]
   memory vector e_t = concatenation of 4 rows x 4 = 16 numbers
   "Alexander the Great" at position 3 and at the end give the same rows: [83, 847] and [83, 847]: same memory, both times

3) 50,000 distinct 2-grams hashed into tables of 65,537 and 65,539 rows
   share a row with another 2-gram in head 1: 53.3%   in head 2: 53.7%
   share BOTH rows with the same other 2-gram:  0.000%

Step 1 is the tokenizer compression from the paper. IDs that differ only in case or spacing map to one canonical ID, so "The" and "the" share their memory. Step 2 shows the key property: the same phrase always lands on the same rows, at any position, with no search. Step 3 shows why there are several heads. We packed 50,000 2-grams into tables of about 65,000 rows, much tighter than a real table. More than half share a row in one head, but none share both rows with the same partner.

Engram at one position: hash the last tokens, read rows, gate, addobeyedAlexandertheGreat2-gram: the Great3-gram: Alexander the Greathash, head 1hash, head 2hash, head 1hash, head 2one row from each table (prime-sized, about 16M rows each in V4.1)concatenate: egate: does e fit h?add to the streamhh = hidden state at this position
Engram at one position. Hash the last two and three tokens with two heads each, read one row from each table, join the rows, gate the result with the hidden state, and add it to the stream.

The retrieved vector is a guess made without context, so Engram checks it against the hidden state before use. The gate is a scaled dot product passed through a sigmoid:

αt=σ ⁣(RMSNorm(ht)⊤ RMSNorm(kt)d)\alpha_t = \sigma\!\left(\frac{\text{RMSNorm}(h_t)^\top \, \text{RMSNorm}(k_t)}{\sqrt{d}}\right)

hth_t is the hidden state at position tt, which already carries the context. ktk_t is a key projected from the retrieved memory. RMSNorm scales each vector so that its numbers have an average square of 1. σ\sigma squashes the result into the range 0 to 1. The output added to the stream is αt\alpha_t times a value projected from the memory. Our toy with d=16d = 16:

plain text
4) gate alpha = sigmoid(RMSNorm(h) . RMSNorm(k) / sqrt(16))
   hidden state agrees with the memory (h = k)   : alpha = 0.982
   unrelated hidden state (random h)             : alpha = 0.451
   hidden state contradicts the memory (h = -k)  : alpha = 0.018

When the memory fits the context, it passes almost whole. When it contradicts the context, for example "Great" in "the Great Barrier Reef", the gate closes. With h=kh = k the dot product of the normalized vectors is exactly d=16d = 16, so the sigmoid sees 16/4=416 / 4 = 4 and gives 0.982.

Why 196B parameters are not "active"

plain text
Engram (V4.1-Flash): layers [1, 14], 3 n-gram orders x 8 hash heads per module
  table rows  = 384,006,168 + 384,016,682 = 768,022,850
  parameters  = 768,022,850 rows x 256 dims = 196.6B   (paper: "196B Engram parameters")
  per table   = 16,000,476 rows on average   (paper: "approximately 16M entries")
  read per token = 48 rows x 256 = 12,288 numbers = 12,288 bytes in FP8 (0.0000062% of the tables)

Here is why "active" is the wrong word. Per token, Engram reads 48 rows: 2 modules x 3 orders x 8 heads. That is 12,288 numbers, about 12 KB in FP8, out of 196.6B. A dense layer with 196.6B weights would need about 393 GFLOPs per token. So DeepSeek counts the 196B separately from both the 552B backbone and the 8B or 16B active.

Does a lookup really cost the same in a huge table? On our CPU, gathering 8,192 rows took the same time from a table of a thousand rows as from one of four million:

plain text
5) gather 8,192 rows of 64 FP16 numbers (CPU, best of 20 runs)
   table of     1,000 rows (    0.1 MB): 0.113 ms
   table of   100,000 rows (   12.8 MB): 0.118 ms
   table of 4,000,000 rows (  512.0 MB): 0.080 ms

The small differences are timing noise on a laptop CPU, not a trend. The table's size costs memory, not time, and that memory can be cheap CPU memory. The Engram paper reports that a 100B-parameter table in host memory costs "negligible overhead (< 3%)". Its own test model, Engram-27B, swapped part of its experts for a 5.7B-parameter table at equal size and compute. It reports MMLU +3.4, BBH +5.0 and long-context retrieval (Multi-Query NIAH) from 84.2 to 97.0.

Multimodality

V4.1-Flash is DeepSeek's first main model trained on images from the start. A vision encoder trained from scratch, DeepSeek-ViT (32 layers, 14-pixel patches), turns an image into features. A 3 x 3 "pixel-unshuffle" folds each 3 x 3 neighbourhood into one token, so an image costs nine times fewer tokens. The pre-training corpus is 45T tokens, 7 parts text-only to 1 part multimodal. Image and text tokens also keep separate load-balancing biases for the experts (Part 2), because they prefer different experts.

The headline chart

Now every number in DeepSeek's own chart has an explanation, and a line of arithmetic behind it.

Global KV cache per token (bytes, log scale)1,00010,000100,000DeepSeek LLM 67B2023-11 GQA, BF16DeepSeek LLM 67B: 389,120 bytes per token389,120 B408.02 GB at 1M tokensDeepSeek-V32024-12 MLA, BF16DeepSeek-V3: 70,272 bytes per token70,272 B73.69 GB at 1M tokensDeepSeek-V3.22025-12 MLA + DSA, FP8DeepSeek-V3.2: 48,068 bytes per token48,068 B50.40 GB at 1M tokensDeepSeek-V4-Flash2026-04 CSA + HCADeepSeek-V4-Flash: 3,514 bytes per token3,514 B3.68 GB at 1M tokensDeepSeek-V4.1-Flash2026-09 CSA2 shared, FP4DeepSeek-V4.1-Flash: 890 bytes per token890 B0.93 GB at 1M tokens
Our script's reconstruction of the chart, with V3 added. Every bar comes from config.json and the published cache layouts; four of the five match the paper's figure to the byte.
plain text
model                  date      bytes/token   paper Fig 1b   x smaller   GB at 128K   GB at 1M
DeepSeek LLM 67B       2023-11      389,120        389,120                   51.00     408.02
DeepSeek-V3            2024-12       70,272              -        5.5x        9.21      73.69
DeepSeek-V3.2          2025-12       48,068         48,068        1.5x        6.30      50.40
DeepSeek-V4-Flash      2026-04        3,514          3,514       13.7x        0.46       3.68
DeepSeek-V4.1-Flash    2026-09          890            890        3.9x        0.12       0.93

V4-Pro (not in the chart): 5,031 B per token; V3.2 / V4-Pro = 9.55x, V3.2 / V4-Flash = 13.68x  (V4 paper Figure 1: "9.5x smaller", "13.7x smaller")
67B / V3.2 = 8.1x  (V4.1 paper Figure 1b: "8.1x smaller")
67B / V4.1-Flash = 437.2x  (V4.1 paper: "437-fold");  V4-Flash / V4.1-Flash = 3.95x  ("approximately 4-fold")

V3 is not in DeepSeek's chart; we added it because it is the model most readers know. At one million tokens, 67B would need 408 GB per conversation, V3 73.7 GB, and V4.1-Flash 0.93 GB.

Research notes: assumptions and what we could not check
  • The main entry layouts (656 B for V3.2, 584 B for V4, 288 B for V4.1 FP4) are from the FlashMLA README at commit ba89a34. The 68-byte indexer key is our reading of MXFP4 (128 four-bit values plus one 8-bit scale per 32). The V4 reference code uses an FP4 block size of 32 (fp4_block_size = 32). This assumption reproduces the published 3,514 and 890 exactly, which is strong but indirect evidence.
  • V4-Pro's 5,031 bytes per token is our number; DeepSeek does not print it. It reproduces the paper's "9.5x smaller".
  • The sliding-window estimate (3.2 MB per request for V4-Flash) assumes the window uses the same 584-byte layout as the compressed entries.
  • The arXiv identifier of the V4 paper starts with 2606, but arXiv shows "Submitted on 26 Apr 2026".
  • DeepSeek-V4-Flash-Vision-Exp exists on Hugging Face; we did not read its config, and there is no paper for it.
  • The V4.1-Flash release note on the API site is dated 10 September 2026; the arXiv paper is dated 17 September 2026.
  • Our mHC toy uses random, not learned, matrices; our Engram toy uses tiny tables. Both show mechanisms, not model behaviour.

Next

Part 7 turns all these numbers into hardware. It counts the GPUs each model needs, and what fits on one machine or on a CPU with lots of RAM. It also covers the quantized files, and how vLLM, SGLang, llama.cpp and DeepSeek's own serving system run them.

References