The DeepSeek-V3 Architecture, Piece by Piece

How DeepSeek-V3 holds 671 billion parameters but uses 37 billion per token: many small experts, a fair router, a compressed memory and a second prediction head.

You ask DeepSeek-V3 a question, and it starts to write. For every word of the answer, the model reads about 37 billion of its numbers. The other 634 billion sit still in memory and wait for a different word.

That one fact is the key to the whole design. A model with the knowledge of 671 billion parameters does the work of a model about 18 times smaller, for each token. The same design runs inside R1 and V3.1, and with one addition inside V3.2, all listed in Part 1. Part 2 takes it apart, piece by piece, and counts every parameter from the real config.json.

A map of the model

The frame is the standard transformer block: attention, then a feed-forward network, each with an RMSNorm before it and a residual connection around it. Inside a Modern Language Model covers that frame, and Attention, Part 1 builds attention from scratch.

DeepSeek-V3 keeps this frame and changes what is inside it. The figure is the map for this whole page.

The whole model (61 layers)Inside one MoE layer (layers 3 to 60)token idsembedding129,280 x 7,168layers 0 to 2MLA + dense FFNlayers 3 to 60MLA + DeepSeekMoERMSNorm + output headscores for 129,280 tokensnext tokenMTP module (layer 61)drafts the token after nextused in training; at inference, optionalinput from the layer belowRMSNormMulti-head Latent Attentioncaches 576 numbers per token+ residualRMSNormDeepSeekMoErouter picks 8 of 256 + 1 shared+ residualoutput to the layer above
The map of DeepSeek-V3 drawn from its config.json. Left: the whole model, from token ids to the next token, with 3 dense layers, 58 MoE layers and one multi-token prediction module. Right: one MoE layer. The two highlighted boxes, MLA and DeepSeekMoE, are the two big changes this part explains.

The paper draws the same picture with more detail.

The exact sizes are in the paper's list of hyper-parameters, and they match the released config.json field for field.

Why mixture of experts

A dense model has a hard rule: every parameter works on every token. If you want a model that knows more, you add parameters, and every token pays for all of them. Knowledge and cost grow together.

But a token does not need all the knowledge at once. The word "mitochondria" needs biology. The token } in a Python file needs code. A model could keep many specialist sub-networks and run only the few that matter for each token.

The idea is old. Shazeer and colleagues trained a sparsely-gated MoE in 2017, and Google's GShard (Lepikhin et al., 2021) scaled it up. DeepSeek's change was about the shape of the experts, which the next section explains.

First, look at what MoE buys in DeepSeek-V3's own numbers. A dense V3 layer has one SwiGLU FFN of width 18,432. An MoE layer has 257 experts of width 2,048. Per token, 9 of them run: 8 routed plus 1 shared. And 9×2,048=18,4329 \times 2{,}048 = 18{,}432.

3×7,168×18,432⏟dense FFN=396,361,728=9×3×7,168×2,048⏟one expert\underbrace{3 \times 7{,}168 \times 18{,}432}_{\text{dense FFN}} = 396{,}361{,}728 = 9 \times \underbrace{3 \times 7{,}168 \times 2{,}048}_{\text{one expert}}

Read it left to right: the dense FFN has 3×d×w3 \times d \times w parameters with d=7,168d = 7{,}168 and w=18,432w = 18{,}432, which equals nine experts of width 2,048. So a token does exactly the same arithmetic in a dense layer and in an MoE layer. The MoE layer just holds 257 experts instead of the 9 a token uses.

Dense FFN (layers 0 to 2)DeepSeekMoE layer (layers 3 to 60)one SwiGLU FFN, width 18,432396.4M parameters, all used1 shared expert (green) + 256 routed experts, width 2,048 each11.32B parameters stored, 9 experts used per tokenWork per token (multiply-adds, same in both):dense: 396.36M MoE: 9 x 44.04M = 396.36MSame work for each token, 28.6 times the parameters to learn with.
A dense FFN and a DeepSeekMoE layer from DeepSeek-V3, drawn to the same budget. Both cost 396.36 million multiply-adds per token. The MoE layer stores 11.32 billion parameters, 28.6 times more, and each token uses 9 of its 257 experts (highlighted).

That is the trade in one picture: same work per token, 28.6 times the parameters to learn with. The price is memory (all 257 experts must sit somewhere) and a new problem: someone has to decide which 9.

DeepSeekMoE: many small experts, plus a shared one

Earlier MoE designs such as GShard used a few big experts and sent each token to the best 2. DeepSeek's January 2024 paper argued that this is the wrong shape, and changed it in two steps.

Step 1: cut each expert into smaller pieces

The second reason is combinations. A token's "team" of experts is a set, and more experts give many more possible sets.

Step 2: one expert that every token uses

If every expert is small and specialised, where does common knowledge go? Grammar, frequent words and basic facts are useful for almost every token.

The equations of one V3 MoE layer

Here is the whole layer as DeepSeek-V3 computes it.

Written out, with the symbols in words:

ht′=ut+∑i=1NsFFNi(s)(ut)+∑i=1Nrgi,t FFNi(r)(ut)h'_t = u_t + \sum_{i=1}^{N_s} \mathrm{FFN}^{(s)}_i(u_t) + \sum_{i=1}^{N_r} g_{i,t}\, \mathrm{FFN}^{(r)}_i(u_t) gi,t=gi,t′∑j=1Nrgj,t′,gi,t′={si,tsi,t∈Topk({sj,t},Kr)0otherwise,si,t=Sigmoid(ut⊤ei)g_{i,t} = \frac{g'_{i,t}}{\sum_{j=1}^{N_r} g'_{j,t}}, \qquad g'_{i,t} = \begin{cases} s_{i,t} & s_{i,t} \in \mathrm{Topk}(\{s_{j,t}\}, K_r) \\ 0 & \text{otherwise} \end{cases}, \qquad s_{i,t} = \mathrm{Sigmoid}(u_t^{\top} e_i)
  • utu_t is the hidden state of token tt as it enters the layer (7,168 numbers), and ht′h'_t is what leaves it.
  • Ns=1N_s = 1 shared expert and Nr=256N_r = 256 routed experts; FFNi\mathrm{FFN}_i is expert ii, a small SwiGLU of width 2,048.
  • eie_i is the centroid of routed expert ii, and si,ts_{i,t} is the affinity: how well token tt matches expert ii.
  • Kr=8K_r = 8 is how many routed experts each token uses; gi,tg_{i,t} is the gate, the weight of expert ii in the sum.

A tiny worked example. Suppose a token's four best affinities are 0.90, 0.60, 0.55 and 0.30, and Kr=4K_r = 4. Their sum is 2.35, so the gates are 0.90/2.35=0.3830.90/2.35 = 0.383, then 0.255, 0.234 and 0.128. They add up to 1. The layer output is the input, plus the shared expert, plus 0.383 times the first expert's output, and so on.

One detail is only in the code. The released model multiplies the normalised gates by routed_scaling_factor = 2.5 from the config. So in practice the 8 gates of a token add up to 2.5, not 1. The paper hints at this when it says V3 "multiplies additional scaling factors at the width bottlenecks" (page 22).

Routing in V3: top-8 of 256, on at most 4 machines

So far the router looks free to pick any 8 of 256 experts. In real training it is not, because of where the experts live.

During training, each layer's 256 experts are spread over 64 GPUs in 8 nodes, 4 experts per GPU. A token whose 8 experts sit on 8 different nodes sends its hidden state over the slow network 8 times.

Training layout of one MoE layer: 256 experts on 64 GPUs in 8 nodes (4 experts per GPU)node 0: experts 0 to 31GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 1: experts 32 to 63GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 2: experts 64 to 95GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 3: experts 96 to 127GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 4: experts 128 to 159GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 5: experts 160 to 191GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 6: experts 192 to 223GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7node 7: experts 224 to 255GPU0GPU1GPU2GPU3GPU4GPU5GPU6GPU7Inside a node, GPUs talk over NVLink (fast). Between nodes, over InfiniBand: 160 GB/s against 50 GB/s on DeepSeek's H800 cluster.Node-limited routing: the 8 experts of a token must sit on at most 4 nodes (blue: the 4 nodes of the token in the next figure).
Where the 256 routed experts of one MoE layer lived during DeepSeek-V3's training: 8 nodes of 8 GPUs, 4 experts per GPU, 32 experts per node. The highlighted nodes are the 4 that the example token below was allowed to use.

The V3 report gives the numbers that made this a problem.

How the 4 nodes are chosen

Here is the rule on a case small enough to check by hand. There are 12 experts on 4 nodes (3 per node), K=4K = 4 and M=2M = 2. So each node is scored by its best 2 experts.

Step 1: score each node by its 2 best experts. Step 2: keep 2 nodes. Step 3: top-4 inside them.node 0 (kept)e00.90e10.10e20.30node score 1.20node 1 (kept)e30.60e40.55e50.05node score 1.15node 2 (skipped)e60.20e70.15e80.25node score 0.45node 3 (skipped)e90.85e100.05e110.10node score 0.95e9 (0.85) is lostChosen: e0, e3, e4, e2 (highlighted). Gates = 0.90, 0.60, 0.55, 0.30 divided by their sum 2.35.
Node-limited routing by hand. Node scores are the sums of each node's 2 best affinities: 0.90 + 0.30 = 1.20, 0.60 + 0.55 = 1.15, 0.25 + 0.20 = 0.45 and 0.85 + 0.10 = 0.95. Nodes 0 and 1 win. The top 4 experts inside them are e0, e3, e4 and e2. Expert e9, the second-best overall, is lost because its node scores too low.

The interesting loss is e9. With an affinity of 0.85 it is the second-best expert for this token. But its node has only one good expert, so the node loses to nodes 0 and 1. The router takes e2 (0.30) instead. This is the price of the rule: a small loss in choice for a large saving in traffic. The router is trained under this rule from the start, so it learns to work within it.

The router, from scratch, against the library

Here is our NumPy version of the whole V3 router. Each line maps to an equation; the bias line is explained in the next section.

python
def route_v3(u, centroids, bias, n_group, topk_group, top_k, scale):
    """u: (T, d) token vectors; centroids: (E, d); bias: (E,). Returns expert ids (T, top_k) and gates (T, top_k)."""
    s = 1 / (1 + np.exp(-(u @ centroids.T)))            # Eq. 15: affinity s_i,t = sigmoid(u_t . e_i), shape (T, E)
    choice = s + bias                                    # Eq. 16: the bias is added for choosing only
    T, E = s.shape
    per = choice.reshape(T, n_group, E // n_group)       # experts grouped by node: (T, 8 nodes, 32 experts)
    k_per_node = top_k // topk_group                     # K_r / M = 8 / 4 = 2
    node_score = np.sort(per, axis=-1)[:, :, -k_per_node:].sum(-1)   # sum of the 2 best experts on each node
    nodes = np.argsort(-node_score, axis=-1)[:, :topk_group]          # keep the 4 best nodes
    allowed = np.zeros((T, n_group), bool)
    np.put_along_axis(allowed, nodes, True, axis=1)
    masked = np.where(np.repeat(allowed, E // n_group, axis=1), choice, -np.inf)
    ids = np.argsort(-masked, axis=-1)[:, :top_k]       # Eq. 14/16: top-8 among the allowed experts
    g = np.take_along_axis(s, ids, axis=1)               # gates come from s, NOT from s + b
    g = g / g.sum(-1, keepdims=True)                     # Eq. 13: normalise the 8 gates to sum to 1
    return ids, g * scale, dict(s=s, node_score=node_score, nodes=nodes)

The config names map one to one: n_group = 8 is the number of nodes, topk_group = 4 is MM, num_experts_per_tok = 8 is KrK_r and routed_scaling_factor = 2.5 is scale. The topk_method in V3's config is noaux_tc, the name for this bias-based, node-limited rule.

We then built transformers' own DeepseekV3TopkRouter with V3's routing shape: hidden size 7,168, 256 experts, 8 groups, 4 kept, top-8. We gave it random weights and a random bias, and fed 4,096 random tokens through both. We did the same for V2's older rule with DeepseekV2TopkRouter on a tiny config. The output of part2_router.py:

plain text
== 2. V3 router (256 experts, 8 nodes, 4 nodes kept, top-8, scale 2.5) vs transformers ==
  4096 random tokens: same 8 experts for every token: True; largest gate difference: 5.3e-07
  gates sum to 2.5000 (= 2.5, the routed_scaling_factor)
  nodes touched per token: max 4, mean 3.97 (limit 4)
  same tokens with no node limit: mean 5.30 nodes, 83% of tokens need more than 4

== 3. V2 router on a tiny random config (16 experts, 4 devices, 2 devices kept, top-4) vs transformers ==
  1000 random tokens: same 4 experts for every token: True; largest gate difference: 6.4e-07

Both routers pick the same experts for every token, and the gates agree to about one part in two million (float32 rounding). The last V3 line shows why the limit matters. With random weights and no limit, a token's 8 experts land on 5.3 nodes on average, and 83% of tokens would cross more than 4. A trained router is not random, so treat this as an illustration of the mechanism, not a measurement of V3.

One token, 256 experts in 8 nodes of 32. Shade = affinity s (sigmoid). Boxed = chosen.node 0expert 0: s = 0.559expert 1: s = 0.412expert 2: s = 0.584expert 3: s = 0.423expert 4: s = 0.385expert 5: s = 0.364expert 6: s = 0.466expert 7: s = 0.459expert 8: s = 0.451expert 9: s = 0.616expert 10: s = 0.316expert 11: s = 0.587expert 12: s = 0.552expert 13: s = 0.559expert 14: s = 0.510expert 15: s = 0.467expert 16: s = 0.444expert 17: s = 0.340expert 18: s = 0.482expert 19: s = 0.435expert 20: s = 0.324expert 21: s = 0.607expert 22: s = 0.384expert 23: s = 0.465expert 24: s = 0.577expert 25: s = 0.656expert 26: s = 0.492expert 27: s = 0.561expert 28: s = 0.477expert 29: s = 0.765expert 30: s = 0.592expert 31: s = 0.6991.443 keptnode 1expert 32: s = 0.479expert 33: s = 0.496expert 34: s = 0.572expert 35: s = 0.590expert 36: s = 0.496expert 37: s = 0.420expert 38: s = 0.347expert 39: s = 0.238expert 40: s = 0.683expert 41: s = 0.407expert 42: s = 0.401expert 43: s = 0.380expert 44: s = 0.695expert 45: s = 0.349expert 46: s = 0.711expert 47: s = 0.527expert 48: s = 0.353expert 49: s = 0.539expert 50: s = 0.332expert 51: s = 0.523expert 52: s = 0.467expert 53: s = 0.470expert 54: s = 0.343expert 55: s = 0.590expert 56: s = 0.483expert 57: s = 0.478expert 58: s = 0.671expert 59: s = 0.633expert 60: s = 0.467expert 61: s = 0.535expert 62: s = 0.529expert 63: s = 0.4441.393node 2expert 64: s = 0.537expert 65: s = 0.698expert 66: s = 0.572expert 67: s = 0.395expert 68: s = 0.624expert 69: s = 0.675expert 70: s = 0.497expert 71: s = 0.513expert 72: s = 0.608expert 73: s = 0.501expert 74: s = 0.404expert 75: s = 0.265expert 76: s = 0.391expert 77: s = 0.482expert 78: s = 0.582expert 79: s = 0.677expert 80: s = 0.383expert 81: s = 0.649expert 82: s = 0.428expert 83: s = 0.234expert 84: s = 0.473expert 85: s = 0.511expert 86: s = 0.496expert 87: s = 0.235expert 88: s = 0.680expert 89: s = 0.582expert 90: s = 0.349expert 91: s = 0.344expert 92: s = 0.467expert 93: s = 0.528expert 94: s = 0.263expert 95: s = 0.4881.396node 3expert 96: s = 0.304expert 97: s = 0.605expert 98: s = 0.367expert 99: s = 0.477expert 100: s = 0.623expert 101: s = 0.553expert 102: s = 0.769expert 103: s = 0.382expert 104: s = 0.399expert 105: s = 0.605expert 106: s = 0.449expert 107: s = 0.333expert 108: s = 0.784expert 109: s = 0.651expert 110: s = 0.446expert 111: s = 0.369expert 112: s = 0.733expert 113: s = 0.488expert 114: s = 0.576expert 115: s = 0.466expert 116: s = 0.382expert 117: s = 0.629expert 118: s = 0.484expert 119: s = 0.567expert 120: s = 0.662expert 121: s = 0.592expert 122: s = 0.544expert 123: s = 0.616expert 124: s = 0.572expert 125: s = 0.372expert 126: s = 0.542expert 127: s = 0.3491.570 keptnode 4expert 128: s = 0.586expert 129: s = 0.371expert 130: s = 0.420expert 131: s = 0.693expert 132: s = 0.555expert 133: s = 0.436expert 134: s = 0.575expert 135: s = 0.573expert 136: s = 0.483expert 137: s = 0.668expert 138: s = 0.694expert 139: s = 0.479expert 140: s = 0.548expert 141: s = 0.397expert 142: s = 0.770expert 143: s = 0.636expert 144: s = 0.747expert 145: s = 0.581expert 146: s = 0.692expert 147: s = 0.511expert 148: s = 0.395expert 149: s = 0.546expert 150: s = 0.319expert 151: s = 0.576expert 152: s = 0.336expert 153: s = 0.496expert 154: s = 0.372expert 155: s = 0.421expert 156: s = 0.478expert 157: s = 0.593expert 158: s = 0.242expert 159: s = 0.5611.529 keptnode 5expert 160: s = 0.657expert 161: s = 0.280expert 162: s = 0.474expert 163: s = 0.638expert 164: s = 0.394expert 165: s = 0.634expert 166: s = 0.507expert 167: s = 0.522expert 168: s = 0.532expert 169: s = 0.467expert 170: s = 0.383expert 171: s = 0.321expert 172: s = 0.392expert 173: s = 0.369expert 174: s = 0.340expert 175: s = 0.603expert 176: s = 0.556expert 177: s = 0.455expert 178: s = 0.573expert 179: s = 0.325expert 180: s = 0.433expert 181: s = 0.408expert 182: s = 0.605expert 183: s = 0.260expert 184: s = 0.605expert 185: s = 0.598expert 186: s = 0.512expert 187: s = 0.644expert 188: s = 0.390expert 189: s = 0.576expert 190: s = 0.639expert 191: s = 0.4571.304node 6expert 192: s = 0.562expert 193: s = 0.413expert 194: s = 0.393expert 195: s = 0.486expert 196: s = 0.566expert 197: s = 0.449expert 198: s = 0.420expert 199: s = 0.663expert 200: s = 0.395expert 201: s = 0.518expert 202: s = 0.509expert 203: s = 0.329expert 204: s = 0.424expert 205: s = 0.654expert 206: s = 0.486expert 207: s = 0.613expert 208: s = 0.453expert 209: s = 0.428expert 210: s = 0.622expert 211: s = 0.575expert 212: s = 0.681expert 213: s = 0.443expert 214: s = 0.478expert 215: s = 0.481expert 216: s = 0.463expert 217: s = 0.689expert 218: s = 0.436expert 219: s = 0.411expert 220: s = 0.505expert 221: s = 0.618expert 222: s = 0.623expert 223: s = 0.4461.352node 7expert 224: s = 0.512expert 225: s = 0.557expert 226: s = 0.520expert 227: s = 0.448expert 228: s = 0.487expert 229: s = 0.477expert 230: s = 0.559expert 231: s = 0.497expert 232: s = 0.602expert 233: s = 0.686expert 234: s = 0.545expert 235: s = 0.681expert 236: s = 0.459expert 237: s = 0.473expert 238: s = 0.357expert 239: s = 0.445expert 240: s = 0.319expert 241: s = 0.483expert 242: s = 0.520expert 243: s = 0.433expert 244: s = 0.519expert 245: s = 0.625expert 246: s = 0.579expert 247: s = 0.320expert 248: s = 0.379expert 249: s = 0.657expert 250: s = 0.456expert 251: s = 0.697expert 252: s = 0.685expert 253: s = 0.541expert 254: s = 0.539expert 255: s = 0.6221.397 keptnode scoreGates of the 8 chosen experts (sum 2.5): e29 0.32, e31 0.29, e102 0.32, e108 0.33, e112 0.31, e142 0.32, e144 0.31, e251 0.29
One random token through a V3-shaped router (random weights). Each row is a node of 32 experts; darker cells have higher affinity. Nodes 0, 3, 4 and 7 have the highest sums of their 2 best affinities and are kept. The 8 boxed cells are the chosen experts, and their gates add up to 2.5.

Load balancing: keeping 256 experts busy

A learned router has a bad habit. Early in training, a few experts happen to be slightly better. The router sends them more tokens, so they train more and get better still. The others get few tokens and stay weak.

The classic fix: an auxiliary loss

The usual remedy adds a second term to the training loss that is small when the load is even. DeepSeek-V2 used this, and V3 keeps a very weak version, so it is worth understanding.

Here is V3's version, the sequence-wise balance loss:

LBal=α∑i=1NrfiPi,fi=NrKrT∑t=1T1(token t picks expert i),Pi=1T∑t=1Tsi,t∑jsj,t\mathcal{L}_{\mathrm{Bal}} = \alpha \sum_{i=1}^{N_r} f_i P_i, \qquad f_i = \frac{N_r}{K_r T} \sum_{t=1}^{T} \mathbb{1}\big(\text{token } t \text{ picks expert } i\big), \qquad P_i = \frac{1}{T} \sum_{t=1}^{T} \frac{s_{i,t}}{\sum_{j} s_{j,t}}
  • TT is the number of tokens in one sequence; NrN_r and KrK_r are 256 and 8 as before.
  • fif_i is expert ii's load, scaled so that the fair share is exactly 1.0.
  • PiP_i is expert ii's average normalised affinity: how much the router likes it, on average.
  • α\alpha is the balance factor, the strength of this term.

The trick is the product fiPif_i P_i. The load fif_i is a count, so training cannot push on it directly. The affinity PiP_i is smooth, so it can. An expert with a high load gets a large fif_i, which makes its PiP_i costly, so training lowers that expert's affinities. Over time it gets fewer tokens.

A tiny worked example, from part2_balance.py: 4 experts, each token picks 1, 4 tokens with the same affinities in both cases.

plain text
== Worked example of the balance loss: 4 experts, each token picks 1, 4 tokens ==
  collapsed (picks 0,0,0,3): f = [3.0, 0.0, 0.0, 1.0]  P = [0.48, 0.111, 0.185, 0.223]  sum f*P = 1.664
  balanced  (picks 0,1,2,3): f = [1.0, 1.0, 1.0, 1.0]  P = [0.48, 0.111, 0.185, 0.223]  sum f*P = 1.000

When expert 0 takes 3 of 4 tokens, its load is 3×4/4=3.03 \times 4/4 = 3.0, and the sum is 3.0×0.48+1.0×0.223=1.6643.0 \times 0.48 + 1.0 \times 0.223 = 1.664. When every expert takes one token, every fif_i is 1 and the sum is just ∑Pi=1.000\sum P_i = 1.000. So the collapsed case costs 66% more, and training is pushed away from it.

The paper sets the problem with this fix in one sentence.

The bias rule, written out

gi,t′={si,tsi,t+bi∈Topk({sj,t+bj},Kr)0otherwisebi←bi−γ⋅sign(loadi−load‾)g'_{i,t} = \begin{cases} s_{i,t} & s_{i,t} + b_i \in \mathrm{Topk}(\{s_{j,t} + b_j\}, K_r) \\ 0 & \text{otherwise} \end{cases} \qquad\quad b_i \leftarrow b_i - \gamma \cdot \mathrm{sign}\big(\mathrm{load}_i - \overline{\mathrm{load}}\big)
  • bib_i is expert ii's bias. It starts at 0 and is stored in the released weights as e_score_correction_bias.
  • loadi\mathrm{load}_i is the number of tokens expert ii got in the whole batch of this step; load‾\overline{\mathrm{load}} is the average over experts.
  • γ\gamma is the bias update speed. sign(x)\mathrm{sign}(x) is +1+1 if x>0x > 0, −1-1 if x<0x < 0 and 0 if x=0x = 0.

The left half is the choosing rule: rank experts by s+bs + b, but keep ss as the gate. The right half is the thermostat, written as one line. The paper describes it in words; the sign form is our compact way to write the same rule.

A tiny example with two experts, A and B. A token scores A at 0.62 and B at 0.60, so A wins. After a few steps in which A was overloaded, bA=−0.003b_A = -0.003 and bB=+0.003b_B = +0.003. Now the token sees 0.6170.617 for A and 0.6030.603 for B, so A still wins.

A token that scored A at 0.604 and B at 0.600 would now go to B (0.601 against 0.603). Only close calls flip. Those are tokens for which both experts were almost equally good.

A simulation: watch the bias even out the load

We cannot train V3, but we can simulate its router. part2_balance.py builds a V3-shaped router (256 experts, top-8) with random centroids. To make some experts "popular", it adds a fixed random offset to each expert's score, as a trained router might have. Each step, a fresh batch of 4,096 random tokens arrives. Here is the core loop:

python
def step(offset, bias):
    u = rng.normal(0, 1, (T, D))
    s = 1 / (1 + np.exp(-(u @ centroids.T + offset)))    # affinities s_i,t (sigmoid)
    ids = np.argpartition(-(s + bias), K, axis=1)[:, :K]  # top-8 of s + b
    load = np.bincount(ids.ravel(), minlength=E)         # tokens each expert received
    return s, ids, load

and the update, applied after each step exactly as the paper describes it, with γ=0.001\gamma = 0.001:

python
        if balance:                                       # V3's rule, applied at the end of every step
            bias -= GAMMA * np.sign(load - load.mean())

We measure imbalance with MaxVio, the measure used in the paper that introduced this method (Wang et al., 2024).

MaxVio=max⁡iloadi−load‾load‾\mathrm{MaxVio} = \frac{\max_i \mathrm{load}_i - \overline{\mathrm{load}}}{\overline{\mathrm{load}}}

It says how far the busiest expert is above the fair share, as a fraction of the fair share. MaxVio = 0 is perfect balance; MaxVio = 1 means the busiest expert does twice its share of work, so its GPU takes twice as long.

plain text
== A no balancing ==
  MaxVio: step 1 3.99, step 100 4.10, step 400 4.30
  tokens per expert at the last step: fair share 128, busiest 678, idlest 0, experts with no token 6
  busiest 8 experts take 13.0% of all assignments (fair: 3.1%)
  balance loss L_Bal (alpha = 1e-4): first step 1.114e-04, last step 1.114e-04

== B bias rule ==
  MaxVio: step 1 3.88, step 100 0.35, step 400 0.33
  tokens per expert at the last step: fair share 128, busiest 170, idlest 89, experts with no token 0
  busiest 8 experts take 3.8% of all assignments (fair: 3.1%)
  balance loss L_Bal (alpha = 1e-4): first step 1.115e-04, last step 1.002e-04
  bias range after 400 steps: -0.104 to +0.245; correlation with the popularity offset: -0.87

Without balancing, the busiest expert gets 678 tokens against a fair share of 128, and 6 experts get none. With the bias rule, MaxVio falls from 3.88 to 0.35 within 100 steps and stays there. The busiest expert drops to 170 and no expert is idle.

No balancing (last step)fair share 1280350700busiest 678, idlest 0With the bias rule, after 400 stepsfair share 1280350700busiest 170, idlest 89Each bar is one of 256 experts, sorted from busiest to idlest. Height = tokens it got in one batch of 4,096 tokens x 8 picks.
Tokens per expert in one batch, all 256 experts sorted from busiest to idlest, from our simulation. Top: no balancing; the busiest expert gets 678 tokens and 6 get none. Bottom: after 400 steps of the bias rule with gamma = 0.001; every expert gets between 89 and 170 tokens, near the fair share of 128.
0123451100200300400no balancing, 1: 3.99no balancing, 20: 4.02no balancing, 40: 4.30no balancing, 60: 4.19no balancing, 80: 4.13no balancing, 100: 4.10no balancing, 120: 4.45no balancing, 140: 4.02no balancing, 160: 4.45no balancing, 180: 4.51no balancing, 200: 4.33no balancing, 220: 4.20no balancing, 240: 4.03no balancing, 260: 4.27no balancing, 280: 4.26no balancing, 300: 4.16no balancing, 320: 4.21no balancing, 340: 4.27no balancing, 360: 4.27no balancing, 380: 4.18no balancing, 400: 4.30bias rule, 1: 3.88bias rule, 20: 3.21bias rule, 40: 2.41bias rule, 60: 1.65bias rule, 80: 0.64bias rule, 100: 0.35bias rule, 120: 0.32bias rule, 140: 0.31bias rule, 160: 0.24bias rule, 180: 0.30bias rule, 200: 0.38bias rule, 220: 0.32bias rule, 240: 0.28bias rule, 260: 0.24bias rule, 280: 0.23bias rule, 300: 0.25bias rule, 320: 0.27bias rule, 340: 0.27bias rule, 360: 0.26bias rule, 380: 0.24bias rule, 400: 0.33no balancing: 4.30bias rule: 0.33training step (one batch of 4,096 tokens each)MaxVio = (busiest - mean) / mean
MaxVio in our simulation over 400 steps. Without balancing it stays near 4: the busiest expert does about five times its fair share. With the bias rule it falls below 0.4 in about 100 steps.

Two lines of the output deserve a closer look. The biases end up strongly anti-correlated with the hidden popularity offsets (−0.87). The rule found the popular experts and pushed them down, without ever seeing the offsets. And MaxVio does not reach 0, because each batch is random. With 128 tokens expected per expert, chance alone spreads loads by about ±11 tokens. The busiest of 256 experts always sits a few of those spreads above the mean.

We also ran a harsher toy. We added a "rich get richer" feedback: after each step, every expert's popularity grows with its load, as if busy experts learned faster. This is our assumption, made to show collapse, not a model of real training.

plain text
== C feedback, no balancing ==
  MaxVio: step 1 4.29, step 100 31.00, step 400 31.00
  tokens per expert at the last step: fair share 128, busiest 4096, idlest 0, experts with no token 248
  busiest 8 experts take 100.0% of all assignments (fair: 3.1%)

== D feedback + bias rule ==
  MaxVio: step 1 4.27, step 100 20.03, step 400 0.34
  tokens per expert at the last step: fair share 128, busiest 171, idlest 4, experts with no token 0

Without balancing, the router collapses completely: 8 experts take every token, and 248 are dead. MaxVio reaches 31, its maximum, because (4,096−128)/128=31(4{,}096 - 128)/128 = 31. With the bias rule, the same feedback still causes a bad start (MaxVio 20 at step 100). But the biases catch up, and by step 400 every expert is back in use.

What balance costs in quality

The bias rule balances over the whole batch. The auxiliary loss in V2 balanced each sequence. That difference turns out to matter.

The report's ablation (Table 5, page 27) compares the two methods on larger models. At 228.7B total parameters and 578B training tokens, the auxiliary-loss-free model scores higher on most benchmarks. For example, HumanEval goes from 40.2 to 46.3 and GSM8K from 70.7 to 74.5. It is not better everywhere: MMLU drops from 68.3 to 67.2.

V3 still keeps a very weak sequence-wise loss as a safety net.

MLA: a small memory for long conversations

The second dashed box in Figure 2 is attention. Its problem is memory, not arithmetic.

With plain multi-head attention, every head in every layer stores one key and one value per token. Attention, Part 2 builds MHA, GQA, MQA and MLA from scratch and compares them; here we focus on V3's real numbers.

Compress, then cache the compressed version

In V3's sizes: the hidden state has 7,168 numbers. WDKVW^{DKV} maps it to dc=512d_c = 512 (kv_lora_rank). WUKW^{UK} and WUVW^{UV} map those 512 numbers back to 128 keys and 128 values of 128 numbers each. Queries get their own compression to 1,536 numbers (q_lora_rank). That does not shrink the cache; the paper says it saves activation memory in training.

MLA in DeepSeek-V3, one layer, one token (sizes from config.json)hidden state h7,168 numbersquery latent1,536 (q_lora_rank)queries, 128 heads128 content + 64 RoPE eachKV latent c512 (kv_lora_rank)RoPE key k_R64, shared by all headskeys, 128 heads128 content (+ k_R)values, 128 heads128 eachW_DQW_DKVW_KR + RoPEW_UQ, W_QRW_UKW_UVCached: 512 + 64 = 576 numbers per token per layer (the two highlighted boxes).Keys and values are never stored; with the absorb trick they are never even built.
MLA in one DeepSeek-V3 layer, with every size from config.json. The hidden state is compressed into a 512-number KV latent and a 64-number RoPE key. Only those two (highlighted) are cached. Keys and values for 128 heads are computed from the latent, and queries come from their own 1,536-number latent.

Absorption: never rebuild the keys at all

Why can the rebuild be skipped? An attention score is a dot product of a query qq with a key k=WUKck = W^{UK} c. Matrix products can be regrouped:

q⊤(WUKc)  =  (WUK⊤q)⊤cq^{\top} \big(W^{UK} c\big) \;=\; \big(W^{UK\top} q\big)^{\top} c
  • qq is one head's query (128 numbers) for the token being written.
  • cc is one cached latent (512 numbers) of an earlier token.
  • WUKW^{UK} is that head's slice of the key up-projection, a 128 × 512 matrix.

The left side rebuilds the key for every cached token, then compares. The right side moves the query into the 512-number latent space once, then compares it with every cached cc directly. Same number, very different cost. The value side works the same way: take the weighted sum of the latents first, then apply WUVW^{UV} once.

Two orders for the same attention score (one head, one cached token)Naive: rebuild the key, then comparecached c512key = W_UK c128, rebuiltquery q128score q . key1 numberAbsorbed: move W_UK to the query side once, compare with the cache directlyquery q128q_lat = W_UK^T q512, once per stepcached c512score q_lat . csame numberq . (W_UK c) = (W_UK^T q) . c : the same product, grouped differently. The value side works the same way with W_UV.
Two orders for the same attention score. Naive: rebuild a 128-number key from each cached latent, then compare it with the query. Absorbed: move the query into the 512-number latent space once per step, then compare it with each cached latent directly.

A tiny example with numbers you can check. Let WUK=(1201)W^{UK} = \begin{pmatrix} 1 & 2 \\ 0 & 1 \end{pmatrix}, c=(3,1)c = (3, 1) and q=(1,1)q = (1, 1). Naive: the key is (1⋅3+2⋅1,  0⋅3+1⋅1)=(5,1)(1 \cdot 3 + 2 \cdot 1,\; 0 \cdot 3 + 1 \cdot 1) = (5, 1), and the score is 5+1=65 + 1 = 6. Absorbed: WUK⊤q=(1,3)W^{UK\top} q = (1, 3), and the score is 1⋅3+3⋅1=61 \cdot 3 + 3 \cdot 1 = 6.

part2_kv.py runs both orders with V3's real per-head shapes, random weights and 64 cached tokens. The shapes are 128 heads, latent 512, content head 128, RoPE part 64 and value head 128. The core of both methods:

python
# Method 1: naive. Rebuild keys and values of every head for every cached token.
k_c = torch.einsum('hdc,tc->htd', W_uk, c_kv)                    # (128, T, 128)
v_c = torch.einsum('hdc,tc->htd', W_uv, c_kv)                    # (128, T, 128)
scores1 = (torch.einsum('hd,htd->ht', q_c, k_c) + q_r @ k_r.T) * scale
o1 = torch.einsum('ht,htd->hd', scores1.softmax(-1), v_c)        # (128, 128)

# Method 2: absorbed. Move W^UK onto the query side and W^UV after the weighted sum.
q_lat = torch.einsum('hd,hdc->hc', q_c, W_uk)                    # (128, 512): query seen in latent space
scores2 = (q_lat @ c_kv.T + q_r @ k_r.T) * scale                 # score straight against the cache
x = scores2.softmax(-1) @ c_kv                                   # (128, 512): weighted sum of latents
o2 = torch.einsum('hc,hdc->hd', x, W_uv)                         # apply W^UV once, at the end
plain text
== MLA, naive vs absorbed, V3 shapes, random weights ==
  cache: c_kv (64, 512) + k_r (64, 64) = 576 numbers per token
  naive  : rebuilt keys + values hold 2,097,152 numbers for 64 tokens
  absorbed: reads only the cache, 36,864 numbers for 64 tokens
  largest score difference 4.9e-15; largest output difference (relative) 1.5e-15
  multiply-adds per cached token per layer: naive 16,818,176, absorbed 139,264

The two methods agree to 15 decimal places (float64), so absorption is exact, not an approximation. The cost line is the reason V3's inference code uses it (attn_impl = "absorb"). Rebuilding keys and values costs about 16.8 million multiply-adds per cached token per layer. Working in latent space costs 139,264, about 120 times less.

Why position needs its own small key

There is one catch, and it explains the strange "+ 64" in V3's cache.

That gives the full cache entry per token per layer:

MLA cache per token=(dc+dhR)×l×b=(512+64)×61×2=70,272 bytes\text{MLA cache per token} = (d_c + d^R_h) \times l \times b = (512 + 64) \times 61 \times 2 = 70{,}272 \text{ bytes}
  • dc=512d_c = 512 is the latent size and dhR=64d^R_h = 64 the shared RoPE key size, both from the config.
  • l=61l = 61 is the number of layers, and b=2b = 2 bytes per number in BF16.

How small is 70,272 bytes?

Here are the numbers for V3's shapes, from part2_kv.py. "MHA, V3 head shapes" is what V3 would store if it kept full keys (192 numbers: 128 content + 64 RoPE) and values (128) for every head.

plain text
== KV cache per token, 61 layers ==
  MLA stores d_c + d_R = 512 + 64 = 576 numbers per layer; x 61 layers x 2 bytes = 70,272 bytes
  MHA, V3 head shapes (K 192, V 128 per head)           4,997,120 B/token    4880.0 KiB   71.11x MLA   128K tokens:  610.00 GiB
  MHA, textbook (2 x 128 heads x 128)                   3,997,696 B/token    3904.0 KiB   56.89x MLA   128K tokens:  488.00 GiB
  GQA, 8 KV groups x 128                                  249,856 B/token     244.0 KiB    3.56x MLA   128K tokens:   30.50 GiB
  MQA, 1 KV head x 128                                     31,232 B/token      30.5 KiB    0.44x MLA   128K tokens:    3.81 GiB
  MLA, BF16 (512 latent + 64 RoPE key)                     70,272 B/token      68.6 KiB    1.00x MLA   128K tokens:    8.58 GiB
  MLA, FP8 layout (512 B + 16 B scales + 128 B RoPE)       40,016 B/token      39.1 KiB    0.57x MLA   128K tokens:    4.88 GiB
MHA (V3 head shapes): 4,997,120 bytes per tokenMHA (V3 head shapes)4,880.0 KiB, 610.0 GiBMHA (textbook, 128 x 128): 3,997,696 bytes per tokenMHA (textbook, 128 x 128)3,904.0 KiB, 488.0 GiBGQA, 8 groups: 249,856 bytes per tokenGQA, 8 groups244.0 KiB, 30.5 GiBMLA, BF16: 70,272 bytes per tokenMLA, BF1668.6 KiB, 8.6 GiBMLA, FP8 layout: 40,016 bytes per tokenMLA, FP8 layout39.1 KiB, 4.9 GiBMQA, 1 head: 31,232 bytes per tokenMQA, 1 head30.5 KiB, 3.8 GiBPer token in KiB, then the cache for one 128K-token (131,072) conversation in GiB. All with 61 layers.
KV cache per token for attention types with DeepSeek-V3's 61 layers and 128 heads, and the cache for one 128K-token conversation. MLA in BF16 needs 68.6 KiB per token and 8.6 GiB at 128K; full MHA with V3's head shapes would need 4,880 KiB and 610 GiB.

Read the 128K column. One full-length conversation with plain MHA would need 610 GiB, more than eight 80 GB GPUs hold together (640 GB, or 596 GiB). With MLA it needs 8.6 GiB. That is the difference between serving one long chat per server and serving dozens.

DeepSeek's later paper on V3's hardware lists the same 70.272 KB for V3 (2505.09343, Table 1). The same table gives 327.680 KB for Qwen-2.5 72B and 516.096 KB for LLaMA-3.1 405B, both GQA models, all in BF16. The FP8 row uses the cache layout documented in DeepSeek's FlashMLA kernels. It keeps 512 bytes of FP8 latent, 16 bytes of scales and the 64-number RoPE key in BF16: 656 bytes per layer.

Research notes: where V2's "93.3% smaller KV cache" comes from

The V2 abstract says V2 "reduces the KV cache by 93.3%" against DeepSeek 67B. The arithmetic only works with mixed precisions. V2 caches (512+64)×60=34,560(512 + 64) \times 60 = 34{,}560 numbers per token, and the paper deploys it with the cache quantised to about 6 bits on average, so 25,920 bytes. DeepSeek 67B uses GQA with 8 KV heads: 2×8×128×95=194,5602 \times 8 \times 128 \times 95 = 194{,}560 numbers, 389,120 bytes in BF16. And 1−25,920/389,120=93.3%1 - 25{,}920 / 389{,}120 = 93.3\%. At equal BF16 precision the saving is 1−69,120/389,120=82.2%1 - 69{,}120 / 389{,}120 = 82.2\%. The paper does not spell this out; the derivation is from the research dossier for this series, and it matches to the decimal.

Multi-token prediction: learning to see two tokens ahead

The last new piece sits outside the main stack. It changes how V3 is trained, and it can make writing faster.

In symbols, for depth kk (V3 uses only k=1k = 1):

hi′k=Mk[RMSNorm(hik−1); RMSNorm(Emb(ti+k))],Pi+k+1k=OutHead(TRMk(hi′k))h'^{k}_i = M_k\big[\mathrm{RMSNorm}(h^{k-1}_i);\ \mathrm{RMSNorm}(\mathrm{Emb}(t_{i+k}))\big], \qquad P^{k}_{i+k+1} = \mathrm{OutHead}\big(\mathrm{TRM}_k(h'^{k}_i)\big) L=Lmain+λD∑k=1DLMTPk\mathcal{L} = \mathcal{L}_{\mathrm{main}} + \frac{\lambda}{D} \sum_{k=1}^{D} \mathcal{L}^{k}_{\mathrm{MTP}}
  • hik−1h^{k-1}_i is the hidden state at position ii from the level below; for k=1k = 1 it is the main model's output.
  • Emb(ti+k)\mathrm{Emb}(t_{i+k}) is the embedding of the real token kk steps ahead; [ ⋅ ; ⋅ ][\,\cdot\,;\,\cdot\,] joins two vectors of 7,168 into one of 14,336.
  • MkM_k maps 14,336 numbers back to 7,168; TRMk\mathrm{TRM}_k is one full transformer block (MLA plus an MoE layer in V3).
  • Pi+k+1kP^{k}_{i+k+1} is the predicted distribution for the token k+1k+1 steps ahead; LMTPk\mathcal{L}^{k}_{\mathrm{MTP}} is its cross-entropy loss.
  • D=1D = 1 is the number of MTP modules, and λ\lambda is their weight in the total loss.

Our toy run prints which targets each part learns, on a 6-token sentence:

plain text
== Which token each depth predicts (T = 6 tokens, D = 1) ==
  position 1 (The): main model predicts t2 = 'cat'  MTP module (also given the embedding of t2) predicts t3 = 'sat'
  position 2 (cat): main model predicts t3 = 'sat'  MTP module (also given the embedding of t3) predicts t4 = 'on'
  position 3 (sat): main model predicts t4 = 'on'   MTP module (also given the embedding of t4) predicts t5 = 'the'
  position 4 ( on): main model predicts t5 = 'the'  MTP module (also given the embedding of t5) predicts t6 = 'mat'
Training: at each position, the main model predicts the next token, the MTP module the one afterinput t1 = Themain modelh1MTP module+ Emb(t2)-> catpredicts t3 = satinput t2 = catmain modelh2MTP module+ Emb(t3)-> satpredicts t4 = oninput t3 = satmain modelh3MTP module+ Emb(t4)-> onpredicts t5 = theinput t4 = onmain modelh4MTP module+ Emb(t5)-> thepredicts t6 = matInference: keep the module as a drafter. The main model checks the draft in its next pass; DeepSeek reports 85% to 90% accepted.
Multi-token prediction on a short sentence. At each position the main model predicts the next token. The MTP module takes the main model's hidden state and the embedding of that next token, and predicts the token after it.

The weight λ\lambda was 0.3 for the first 10T training tokens and 0.1 for the last 4.8T (page 23). Take an illustrative main loss of 2.0 and MTP loss of 2.6; we chose these, they are not from the paper. The total is 2.0+0.3×2.6=2.782.0 + 0.3 \times 2.6 = 2.78 early in training and 2.0+0.1×2.6=2.262.0 + 0.1 \times 2.6 = 2.26 late. The extra signal is strong early and fades as training ends.

Why bother? The paper says MTP "densifies the training signals" and "may enable the model to pre-plan its representations". Its ablation (Table 4, page 26) adds one MTP module to a 228.7B-parameter MoE model trained on 540B tokens. Most scores go up, for example HumanEval from 44.5 to 53.7 and GSM8K from 72.3 to 74.0. A few go down slightly, such as MMLU from 67.5 to 66.6.

At inference: drop it, or use it as a drafter

With one draft token accepted with probability pp, each step yields 1+p1 + p tokens on average:

plain text
== Tokens per decoding step with one draft token ==
  acceptance 85%: 1 + 0.85 = 1.85 tokens per step (upper bound on the speed-up)
  acceptance 90%: 1 + 0.90 = 1.90 tokens per step (upper bound on the speed-up)
  reported by DeepSeek: 1.8x tokens per second; the gap is the cost of running the MTP layer and verifying

The reported 1.8× sits just under our 1.85 to 1.90 ceiling, which is what you would expect once the extra layer's cost is paid. In the V4 releases from July 2026, DeepSeek replaced this drafter with a new one called DSpark; Part 6 covers it.

The 671B and 37B, counted from config.json

Now we can count everything. part2_params.py reads the real V3 config.json and counts each weight matrix with a formula (Method 1). Then it builds DeepseekV3ForCausalLM from the same config on PyTorch's "meta" device, which creates every tensor's shape but no memory. It counts what transformers made (Method 2). The heart of Method 1 is one line for attention:

python
    q = d * nh * (nope + rope) if qr is None else d * qr + qr + qr * nh * (nope + rope)
    attn = q + d * (kvr + rope) + kvr + kvr * nh * (nope + vd) + nh * vd * d

Read it as the MLA flow figure, box by box. First the query down-projection (7,168 × 1,536) and its norm, then the query up-projection to 128 heads of 192. Next the KV down-projection to 512 + 64 and its norm, and the KV up-projection to 128 heads of 128 + 128. Last comes the output matrix. That is 187,107,328 parameters per layer. The output:

plain text
== Per-piece sizes ==
  MLA attention, one layer         187,107,328
  dense SwiGLU FFN, one layer      396,361,728
  one expert (SwiGLU)               44,040,192
  router, one MoE layer              1,835,264
  dense layers: 3, MoE layers: 58

== Method 1: our formula, total parameters ==
  embedding                         0.927B    0.14%
  output head                       0.927B    0.14%
  attention (MLA), all layers      11.414B    1.70%
  dense FFN, first layers           1.189B    0.18%
  routed experts                  653.909B   97.45%
  shared experts                    2.554B    0.38%
  routers (incl. bias)              0.106B    0.02%
  RMSNorm weights                   0.001B    0.00%
  TOTAL main model                671.026B   (671,026,419,200)

== Method 2: transformers DeepseekV3ForCausalLM on the meta device ==
  parameters + balancing bias    671.026B   (671,026,419,200)
  equal to Method 1: True

The two counts agree to the last parameter: 671,026,419,200. The paper's "671B" is exact, not a rounding of something larger. And 97.45% of the model is routed experts. Attention, the part most explanations of transformers spend their time on, is 1.7%.

Now count what one token touches: everything except the 248 routed experts it did not pick in each MoE layer.

plain text
== Active parameters for one token ==
  embedding                         0.927B
  output head                       0.927B
  attention (MLA), all layers      11.414B
  dense FFN, first layers           1.189B
  routed experts                   20.435B
  shared experts                    2.554B
  routers (incl. bias)              0.106B
  RMSNorm weights                   0.001B
  TOTAL active                     37.552B
  without the input embedding (a table lookup, not a matrix multiply):   36.626B
  share of all parameters used per token: 5.6%
All 671.0B parametersrouted experts: 653.91B (97.4%)attention (MLA): 11.41B (1.7%)shared experts: 2.55B (0.4%)dense FFN: 1.19B (0.2%)embedding, head, routers, norms: 1.96B (0.3%)routed experts 653.9B (97.4%) · attention (MLA) 11.4B (1.7%) · shared experts 2.6B (0.4%)dense FFN 1.2B (0.2%) · embedding, head, routers, norms 2.0B (0.3%)Used for one token: 37.6Brouted experts: 20.43B (54.4%)attention (MLA): 11.41B (30.4%)shared experts: 2.55B (6.8%)dense FFN: 1.19B (3.2%)embedding, head, routers, norms: 1.96B (5.2%)routed experts 20.4B (54.4%) · attention (MLA) 11.4B (30.4%) · shared experts 2.6B (6.8%)dense FFN 1.2B (3.2%) · embedding, head, routers, norms 2.0B (5.2%)Colours: blue routed experts, orange attention, green shared experts, yellow dense FFN, grey the rest.
Where DeepSeek-V3's parameters are, from our count of config.json. Top: all 671.0 billion; routed experts are 97.4 percent. Bottom: the 37.6 billion one token uses; routed experts are 54 percent, attention 30 percent.

Per token, the picture flips. Routed experts drop to 20.4B (8 of 256 in 58 layers), and attention becomes the second-largest share at 30%. So the paper's "37B" checks out: 37.55B if you count the input embedding, 36.63B if you do not. DeepSeek's weight notes say 36.7B "including 0.9B for Embedding and 0.9B for the output Head"; we could not reproduce that exact figure: with both counted, we get 37.55B, a gap of about 0.85B.

Why Hugging Face says 685B

plain text
== The MTP module (one extra layer, num_nextn_predict_layers = 1) ==
  enorm + hnorm + eh_proj (7168 x 14336) + one full MoE layer + final norm =   11.610B
  plus its stored copies of the embedding and output head            =    1.853B
  checkpoint on Hugging Face:  671.026B +   11.610B +    1.853B =  684.490B

The MTP module is a full MoE layer (11.5B of its 11.61B) plus the MkM_k projection and norms. In the checkpoint it also stores its own copies of the embedding and output head. That sums to 684.49B. DeepSeek's README rounds it to "685B, which includes 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights." The 685B on the model page is the same model plus its drafter.

Finally, a second real config as a check. The cached DeepSeek-V2-Lite config gives:

plain text
== Check on a second real config: DeepSeek-V2-Lite-Chat (published 15.7B total, 2.4B active) ==
  Method 1 total   15.706B, Method 2 total   15.706B, equal: True
  Method 1 active    2.661B, without the input embedding    2.451B

The total matches the paper's 15.7B. The active count matches its 2.4B when the input embedding is left out, which suggests that is the convention DeepSeek used for V2-Lite.

What this design costs

None of these ideas is free. Many experts mean a large memory footprint: the paper's own limits section says "the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams" (page 35). Expert parallelism needs fast networks and careful kernels. Node-limited routing gives up some choice of experts for less traffic. MLA saves memory but needs absorption tricks and special kernels to be fast.

The payoff is a model that does the work of a roughly 37B model per token, while it holds the knowledge of a 671B one. Its cache is small enough for 128K-token conversations. The same skeleton, with small changes, carries every DeepSeek model through V3.2.

Next, Part 3 follows this architecture through training. It covers the 14.8 trillion tokens, FP8 arithmetic and the DualPipe schedule that hides the all-to-all traffic described here. It also covers the cluster, and what the famous training cost does and does not include.

References

  • Dai et al. (2024), DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. arXiv:2401.06066
  • DeepSeek-AI (2024), DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434
  • DeepSeek-AI (2024), DeepSeek-V3 Technical Report. arXiv:2412.19437
  • Wang et al. (2024), Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. arXiv:2408.15664
  • DeepSeek-AI (2025), Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures. arXiv:2505.09343
  • Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538
  • DeepSeek-V3 model card, config.json and weight notes: huggingface.co/deepseek-ai/DeepSeek-V3, README_WEIGHTS.md
  • DeepSeek-V3 reference inference code: inference/model.py
  • FlashMLA (KV cache layouts): github.com/deepseek-ai/FlashMLA
  • Our scripts: code/deepseek/part2_params.py, part2_router.py, part2_balance.py, part2_kv.py, part2_mtp.py