How DeepSeek-V3 holds 671 billion parameters but uses 37 billion per token: many small experts, a fair router, a compressed memory and a second prediction head.
You ask DeepSeek-V3 a question, and it starts to write. For every word of the answer, the model reads about 37 billion of its numbers. The other 634 billion sit still in memory and wait for a different word.
That one fact is the key to the whole design. A model with the knowledge of 671 billion parameters does the work of a model about 18 times smaller, for each token. The same design runs inside R1 and V3.1, and with one addition inside V3.2, all listed in Part 1. Part 2 takes it apart, piece by piece, and counts every parameter from the real config.json.
The frame is the standard transformer block: attention, then a feed-forward network, each with an RMSNorm before it and a residual connection around it. Inside a Modern Language Model covers that frame, and Attention, Part 1 builds attention from scratch.
DeepSeek-V3 keeps this frame and changes what is inside it. The figure is the map for this whole page.
The map of DeepSeek-V3 drawn from its config.json. Left: the whole model, from token ids to the next token, with 3 dense layers, 58 MoE layers and one multi-token prediction module. Right: one MoE layer. The two highlighted boxes, MLA and DeepSeekMoE, are the two big changes this part explains.
The paper draws the same picture with more detail.
The exact sizes are in the paper's list of hyper-parameters, and they match the released config.json field for field.
A dense model has a hard rule: every parameter works on every token. If you want a model that knows more, you add parameters, and every token pays for all of them. Knowledge and cost grow together.
But a token does not need all the knowledge at once. The word "mitochondria" needs biology. The token } in a Python file needs code. A model could keep many specialist sub-networks and run only the few that matter for each token.
The idea is old. Shazeer and colleagues trained a sparsely-gated MoE in 2017, and Google's GShard (Lepikhin et al., 2021) scaled it up. DeepSeek's change was about the shape of the experts, which the next section explains.
First, look at what MoE buys in DeepSeek-V3's own numbers. A dense V3 layer has one SwiGLU FFN of width 18,432. An MoE layer has 257 experts of width 2,048. Per token, 9 of them run: 8 routed plus 1 shared. And 9×2,048=18,432.
Read it left to right: the dense FFN has 3×d×w parameters with d=7,168 and w=18,432, which equals nine experts of width 2,048. So a token does exactly the same arithmetic in a dense layer and in an MoE layer. The MoE layer just holds 257 experts instead of the 9 a token uses.
A dense FFN and a DeepSeekMoE layer from DeepSeek-V3, drawn to the same budget. Both cost 396.36 million multiply-adds per token. The MoE layer stores 11.32 billion parameters, 28.6 times more, and each token uses 9 of its 257 experts (highlighted).
That is the trade in one picture: same work per token, 28.6 times the parameters to learn with. The price is memory (all 257 experts must sit somewhere) and a new problem: someone has to decide which 9.
DeepSeekMoE: many small experts, plus a shared one#
Earlier MoE designs such as GShard used a few big experts and sent each token to the best 2. DeepSeek's January 2024 paper argued that this is the wrong shape, and changed it in two steps.
ut is the hidden state of token t as it enters the layer (7,168 numbers), and ht′ is what leaves it.
Ns=1 shared expert and Nr=256 routed experts; FFNi is expert i, a small SwiGLU of width 2,048.
ei is the centroid of routed expert i, and si,t is the affinity: how well token t matches expert i.
Kr=8 is how many routed experts each token uses; gi,t is the gate, the weight of expert i in the sum.
A tiny worked example. Suppose a token's four best affinities are 0.90, 0.60, 0.55 and 0.30, and Kr=4. Their sum is 2.35, so the gates are 0.90/2.35=0.383, then 0.255, 0.234 and 0.128. They add up to 1. The layer output is the input, plus the shared expert, plus 0.383 times the first expert's output, and so on.
One detail is only in the code. The released model multiplies the normalised gates by routed_scaling_factor = 2.5 from the config. So in practice the 8 gates of a token add up to 2.5, not 1. The paper hints at this when it says V3 "multiplies additional scaling factors at the width bottlenecks" (page 22).
Routing in V3: top-8 of 256, on at most 4 machines#
So far the router looks free to pick any 8 of 256 experts. In real training it is not, because of where the experts live.
During training, each layer's 256 experts are spread over 64 GPUs in 8 nodes, 4 experts per GPU. A token whose 8 experts sit on 8 different nodes sends its hidden state over the slow network 8 times.
Where the 256 routed experts of one MoE layer lived during DeepSeek-V3's training: 8 nodes of 8 GPUs, 4 experts per GPU, 32 experts per node. The highlighted nodes are the 4 that the example token below was allowed to use.
The V3 report gives the numbers that made this a problem.
Here is the rule on a case small enough to check by hand. There are 12 experts on 4 nodes (3 per node), K=4 and M=2. So each node is scored by its best 2 experts.
Node-limited routing by hand. Node scores are the sums of each node's 2 best affinities: 0.90 + 0.30 = 1.20, 0.60 + 0.55 = 1.15, 0.25 + 0.20 = 0.45 and 0.85 + 0.10 = 0.95. Nodes 0 and 1 win. The top 4 experts inside them are e0, e3, e4 and e2. Expert e9, the second-best overall, is lost because its node scores too low.
The interesting loss is e9. With an affinity of 0.85 it is the second-best expert for this token. But its node has only one good expert, so the node loses to nodes 0 and 1. The router takes e2 (0.30) instead. This is the price of the rule: a small loss in choice for a large saving in traffic. The router is trained under this rule from the start, so it learns to work within it.
Here is our NumPy version of the whole V3 router. Each line maps to an equation; the bias line is explained in the next section.
python
def route_v3(u, centroids, bias, n_group, topk_group, top_k, scale): """u: (T, d) token vectors; centroids: (E, d); bias: (E,). Returns expert ids (T, top_k) and gates (T, top_k).""" s = 1 / (1 + np.exp(-(u @ centroids.T))) # Eq. 15: affinity s_i,t = sigmoid(u_t . e_i), shape (T, E) choice = s + bias # Eq. 16: the bias is added for choosing only T, E = s.shape per = choice.reshape(T, n_group, E // n_group) # experts grouped by node: (T, 8 nodes, 32 experts) k_per_node = top_k // topk_group # K_r / M = 8 / 4 = 2 node_score = np.sort(per, axis=-1)[:, :, -k_per_node:].sum(-1) # sum of the 2 best experts on each node nodes = np.argsort(-node_score, axis=-1)[:, :topk_group] # keep the 4 best nodes allowed = np.zeros((T, n_group), bool) np.put_along_axis(allowed, nodes, True, axis=1) masked = np.where(np.repeat(allowed, E // n_group, axis=1), choice, -np.inf) ids = np.argsort(-masked, axis=-1)[:, :top_k] # Eq. 14/16: top-8 among the allowed experts g = np.take_along_axis(s, ids, axis=1) # gates come from s, NOT from s + b g = g / g.sum(-1, keepdims=True) # Eq. 13: normalise the 8 gates to sum to 1 return ids, g * scale, dict(s=s, node_score=node_score, nodes=nodes)
The config names map one to one: n_group = 8 is the number of nodes, topk_group = 4 is M, num_experts_per_tok = 8 is Kr and routed_scaling_factor = 2.5 is scale. The topk_method in V3's config is noaux_tc, the name for this bias-based, node-limited rule.
We then built transformers' own DeepseekV3TopkRouter with V3's routing shape: hidden size 7,168, 256 experts, 8 groups, 4 kept, top-8. We gave it random weights and a random bias, and fed 4,096 random tokens through both. We did the same for V2's older rule with DeepseekV2TopkRouter on a tiny config. The output of part2_router.py:
plain text
== 2. V3 router (256 experts, 8 nodes, 4 nodes kept, top-8, scale 2.5) vs transformers == 4096 random tokens: same 8 experts for every token: True; largest gate difference: 5.3e-07 gates sum to 2.5000 (= 2.5, the routed_scaling_factor) nodes touched per token: max 4, mean 3.97 (limit 4) same tokens with no node limit: mean 5.30 nodes, 83% of tokens need more than 4== 3. V2 router on a tiny random config (16 experts, 4 devices, 2 devices kept, top-4) vs transformers == 1000 random tokens: same 4 experts for every token: True; largest gate difference: 6.4e-07
Both routers pick the same experts for every token, and the gates agree to about one part in two million (float32 rounding). The last V3 line shows why the limit matters. With random weights and no limit, a token's 8 experts land on 5.3 nodes on average, and 83% of tokens would cross more than 4. A trained router is not random, so treat this as an illustration of the mechanism, not a measurement of V3.
One random token through a V3-shaped router (random weights). Each row is a node of 32 experts; darker cells have higher affinity. Nodes 0, 3, 4 and 7 have the highest sums of their 2 best affinities and are kept. The 8 boxed cells are the chosen experts, and their gates add up to 2.5.
A learned router has a bad habit. Early in training, a few experts happen to be slightly better. The router sends them more tokens, so they train more and get better still. The others get few tokens and stay weak.
The usual remedy adds a second term to the training loss that is small when the load is even. DeepSeek-V2 used this, and V3 keeps a very weak version, so it is worth understanding.
Here is V3's version, the sequence-wise balance loss:
LBal=αi=1∑NrfiPi,fi=KrTNrt=1∑T1(token t picks expert i),Pi=T1t=1∑T∑jsj,tsi,t
T is the number of tokens in one sequence; Nr and Kr are 256 and 8 as before.
fi is expert i's load, scaled so that the fair share is exactly 1.0.
Pi is expert i's average normalised affinity: how much the router likes it, on average.
α is the balance factor, the strength of this term.
The trick is the product fiPi. The load fi is a count, so training cannot push on it directly. The affinity Pi is smooth, so it can. An expert with a high load gets a large fi, which makes its Pi costly, so training lowers that expert's affinities. Over time it gets fewer tokens.
A tiny worked example, from part2_balance.py: 4 experts, each token picks 1, 4 tokens with the same affinities in both cases.
plain text
== Worked example of the balance loss: 4 experts, each token picks 1, 4 tokens == collapsed (picks 0,0,0,3): f = [3.0, 0.0, 0.0, 1.0] P = [0.48, 0.111, 0.185, 0.223] sum f*P = 1.664 balanced (picks 0,1,2,3): f = [1.0, 1.0, 1.0, 1.0] P = [0.48, 0.111, 0.185, 0.223] sum f*P = 1.000
When expert 0 takes 3 of 4 tokens, its load is 3×4/4=3.0, and the sum is 3.0×0.48+1.0×0.223=1.664. When every expert takes one token, every fi is 1 and the sum is just ∑Pi=1.000. So the collapsed case costs 66% more, and training is pushed away from it.
The paper sets the problem with this fix in one sentence.
bi is expert i's bias. It starts at 0 and is stored in the released weights as e_score_correction_bias.
loadi is the number of tokens expert i got in the whole batch of this step; load is the average over experts.
γ is the bias update speed. sign(x) is +1 if x>0, −1 if x<0 and 0 if x=0.
The left half is the choosing rule: rank experts by s+b, but keep s as the gate. The right half is the thermostat, written as one line. The paper describes it in words; the sign form is our compact way to write the same rule.
A tiny example with two experts, A and B. A token scores A at 0.62 and B at 0.60, so A wins. After a few steps in which A was overloaded, bA=−0.003 and bB=+0.003. Now the token sees 0.617 for A and 0.603 for B, so A still wins.
A token that scored A at 0.604 and B at 0.600 would now go to B (0.601 against 0.603). Only close calls flip. Those are tokens for which both experts were almost equally good.
We cannot train V3, but we can simulate its router. part2_balance.py builds a V3-shaped router (256 experts, top-8) with random centroids. To make some experts "popular", it adds a fixed random offset to each expert's score, as a trained router might have. Each step, a fresh batch of 4,096 random tokens arrives. Here is the core loop:
python
def step(offset, bias): u = rng.normal(0, 1, (T, D)) s = 1 / (1 + np.exp(-(u @ centroids.T + offset))) # affinities s_i,t (sigmoid) ids = np.argpartition(-(s + bias), K, axis=1)[:, :K] # top-8 of s + b load = np.bincount(ids.ravel(), minlength=E) # tokens each expert received return s, ids, load
and the update, applied after each step exactly as the paper describes it, with γ=0.001:
python
if balance: # V3's rule, applied at the end of every step bias -= GAMMA * np.sign(load - load.mean())
We measure imbalance with MaxVio, the measure used in the paper that introduced this method (Wang et al., 2024).
MaxVio=loadmaxiloadi−load
It says how far the busiest expert is above the fair share, as a fraction of the fair share. MaxVio = 0 is perfect balance; MaxVio = 1 means the busiest expert does twice its share of work, so its GPU takes twice as long.
plain text
== A no balancing == MaxVio: step 1 3.99, step 100 4.10, step 400 4.30 tokens per expert at the last step: fair share 128, busiest 678, idlest 0, experts with no token 6 busiest 8 experts take 13.0% of all assignments (fair: 3.1%) balance loss L_Bal (alpha = 1e-4): first step 1.114e-04, last step 1.114e-04== B bias rule == MaxVio: step 1 3.88, step 100 0.35, step 400 0.33 tokens per expert at the last step: fair share 128, busiest 170, idlest 89, experts with no token 0 busiest 8 experts take 3.8% of all assignments (fair: 3.1%) balance loss L_Bal (alpha = 1e-4): first step 1.115e-04, last step 1.002e-04 bias range after 400 steps: -0.104 to +0.245; correlation with the popularity offset: -0.87
Without balancing, the busiest expert gets 678 tokens against a fair share of 128, and 6 experts get none. With the bias rule, MaxVio falls from 3.88 to 0.35 within 100 steps and stays there. The busiest expert drops to 170 and no expert is idle.
Tokens per expert in one batch, all 256 experts sorted from busiest to idlest, from our simulation. Top: no balancing; the busiest expert gets 678 tokens and 6 get none. Bottom: after 400 steps of the bias rule with gamma = 0.001; every expert gets between 89 and 170 tokens, near the fair share of 128.MaxVio in our simulation over 400 steps. Without balancing it stays near 4: the busiest expert does about five times its fair share. With the bias rule it falls below 0.4 in about 100 steps.
Two lines of the output deserve a closer look. The biases end up strongly anti-correlated with the hidden popularity offsets (−0.87). The rule found the popular experts and pushed them down, without ever seeing the offsets. And MaxVio does not reach 0, because each batch is random. With 128 tokens expected per expert, chance alone spreads loads by about ±11 tokens. The busiest of 256 experts always sits a few of those spreads above the mean.
We also ran a harsher toy. We added a "rich get richer" feedback: after each step, every expert's popularity grows with its load, as if busy experts learned faster. This is our assumption, made to show collapse, not a model of real training.
plain text
== C feedback, no balancing == MaxVio: step 1 4.29, step 100 31.00, step 400 31.00 tokens per expert at the last step: fair share 128, busiest 4096, idlest 0, experts with no token 248 busiest 8 experts take 100.0% of all assignments (fair: 3.1%)== D feedback + bias rule == MaxVio: step 1 4.27, step 100 20.03, step 400 0.34 tokens per expert at the last step: fair share 128, busiest 171, idlest 4, experts with no token 0
Without balancing, the router collapses completely: 8 experts take every token, and 248 are dead. MaxVio reaches 31, its maximum, because (4,096−128)/128=31. With the bias rule, the same feedback still causes a bad start (MaxVio 20 at step 100). But the biases catch up, and by step 400 every expert is back in use.
The bias rule balances over the whole batch. The auxiliary loss in V2 balanced each sequence. That difference turns out to matter.
The report's ablation (Table 5, page 27) compares the two methods on larger models. At 228.7B total parameters and 578B training tokens, the auxiliary-loss-free model scores higher on most benchmarks. For example, HumanEval goes from 40.2 to 46.3 and GSM8K from 70.7 to 74.5. It is not better everywhere: MMLU drops from 68.3 to 67.2.
V3 still keeps a very weak sequence-wise loss as a safety net.
The second dashed box in Figure 2 is attention. Its problem is memory, not arithmetic.
With plain multi-head attention, every head in every layer stores one key and one value per token. Attention, Part 2 builds MHA, GQA, MQA and MLA from scratch and compares them; here we focus on V3's real numbers.
In V3's sizes: the hidden state has 7,168 numbers. WDKV maps it to dc=512 (kv_lora_rank). WUK and WUV map those 512 numbers back to 128 keys and 128 values of 128 numbers each. Queries get their own compression to 1,536 numbers (q_lora_rank). That does not shrink the cache; the paper says it saves activation memory in training.
MLA in one DeepSeek-V3 layer, with every size from config.json. The hidden state is compressed into a 512-number KV latent and a 64-number RoPE key. Only those two (highlighted) are cached. Keys and values for 128 heads are computed from the latent, and queries come from their own 1,536-number latent.
Why can the rebuild be skipped? An attention score is a dot product of a query q with a key k=WUKc. Matrix products can be regrouped:
q⊤(WUKc)=(WUK⊤q)⊤c
q is one head's query (128 numbers) for the token being written.
c is one cached latent (512 numbers) of an earlier token.
WUK is that head's slice of the key up-projection, a 128 × 512 matrix.
The left side rebuilds the key for every cached token, then compares. The right side moves the query into the 512-number latent space once, then compares it with every cached c directly. Same number, very different cost. The value side works the same way: take the weighted sum of the latents first, then apply WUV once.
Two orders for the same attention score. Naive: rebuild a 128-number key from each cached latent, then compare it with the query. Absorbed: move the query into the 512-number latent space once per step, then compare it with each cached latent directly.
A tiny example with numbers you can check. Let WUK=(1021), c=(3,1) and q=(1,1). Naive: the key is (1⋅3+2⋅1,0⋅3+1⋅1)=(5,1), and the score is 5+1=6. Absorbed: WUK⊤q=(1,3), and the score is 1⋅3+3⋅1=6.
part2_kv.py runs both orders with V3's real per-head shapes, random weights and 64 cached tokens. The shapes are 128 heads, latent 512, content head 128, RoPE part 64 and value head 128. The core of both methods:
python
# Method 1: naive. Rebuild keys and values of every head for every cached token.k_c = torch.einsum('hdc,tc->htd', W_uk, c_kv) # (128, T, 128)v_c = torch.einsum('hdc,tc->htd', W_uv, c_kv) # (128, T, 128)scores1 = (torch.einsum('hd,htd->ht', q_c, k_c) + q_r @ k_r.T) * scaleo1 = torch.einsum('ht,htd->hd', scores1.softmax(-1), v_c) # (128, 128)# Method 2: absorbed. Move W^UK onto the query side and W^UV after the weighted sum.q_lat = torch.einsum('hd,hdc->hc', q_c, W_uk) # (128, 512): query seen in latent spacescores2 = (q_lat @ c_kv.T + q_r @ k_r.T) * scale # score straight against the cachex = scores2.softmax(-1) @ c_kv # (128, 512): weighted sum of latentso2 = torch.einsum('hc,hdc->hd', x, W_uv) # apply W^UV once, at the end
plain text
== MLA, naive vs absorbed, V3 shapes, random weights == cache: c_kv (64, 512) + k_r (64, 64) = 576 numbers per token naive : rebuilt keys + values hold 2,097,152 numbers for 64 tokens absorbed: reads only the cache, 36,864 numbers for 64 tokens largest score difference 4.9e-15; largest output difference (relative) 1.5e-15 multiply-adds per cached token per layer: naive 16,818,176, absorbed 139,264
The two methods agree to 15 decimal places (float64), so absorption is exact, not an approximation. The cost line is the reason V3's inference code uses it (attn_impl = "absorb"). Rebuilding keys and values costs about 16.8 million multiply-adds per cached token per layer. Working in latent space costs 139,264, about 120 times less.
Here are the numbers for V3's shapes, from part2_kv.py. "MHA, V3 head shapes" is what V3 would store if it kept full keys (192 numbers: 128 content + 64 RoPE) and values (128) for every head.
plain text
== KV cache per token, 61 layers == MLA stores d_c + d_R = 512 + 64 = 576 numbers per layer; x 61 layers x 2 bytes = 70,272 bytes MHA, V3 head shapes (K 192, V 128 per head) 4,997,120 B/token 4880.0 KiB 71.11x MLA 128K tokens: 610.00 GiB MHA, textbook (2 x 128 heads x 128) 3,997,696 B/token 3904.0 KiB 56.89x MLA 128K tokens: 488.00 GiB GQA, 8 KV groups x 128 249,856 B/token 244.0 KiB 3.56x MLA 128K tokens: 30.50 GiB MQA, 1 KV head x 128 31,232 B/token 30.5 KiB 0.44x MLA 128K tokens: 3.81 GiB MLA, BF16 (512 latent + 64 RoPE key) 70,272 B/token 68.6 KiB 1.00x MLA 128K tokens: 8.58 GiB MLA, FP8 layout (512 B + 16 B scales + 128 B RoPE) 40,016 B/token 39.1 KiB 0.57x MLA 128K tokens: 4.88 GiB
KV cache per token for attention types with DeepSeek-V3's 61 layers and 128 heads, and the cache for one 128K-token conversation. MLA in BF16 needs 68.6 KiB per token and 8.6 GiB at 128K; full MHA with V3's head shapes would need 4,880 KiB and 610 GiB.
Read the 128K column. One full-length conversation with plain MHA would need 610 GiB, more than eight 80 GB GPUs hold together (640 GB, or 596 GiB). With MLA it needs 8.6 GiB. That is the difference between serving one long chat per server and serving dozens.
DeepSeek's later paper on V3's hardware lists the same 70.272 KB for V3 (2505.09343, Table 1). The same table gives 327.680 KB for Qwen-2.5 72B and 516.096 KB for LLaMA-3.1 405B, both GQA models, all in BF16. The FP8 row uses the cache layout documented in DeepSeek's FlashMLA kernels. It keeps 512 bytes of FP8 latent, 16 bytes of scales and the 64-number RoPE key in BF16: 656 bytes per layer.
Research notes: where V2's "93.3% smaller KV cache" comes from
The V2 abstract says V2 "reduces the KV cache by 93.3%" against DeepSeek 67B. The arithmetic only works with mixed precisions. V2 caches (512+64)×60=34,560 numbers per token, and the paper deploys it with the cache quantised to about 6 bits on average, so 25,920 bytes. DeepSeek 67B uses GQA with 8 KV heads: 2×8×128×95=194,560 numbers, 389,120 bytes in BF16. And 1−25,920/389,120=93.3%. At equal BF16 precision the saving is 1−69,120/389,120=82.2%. The paper does not spell this out; the derivation is from the research dossier for this series, and it matches to the decimal.
Multi-token prediction: learning to see two tokens ahead#
The last new piece sits outside the main stack. It changes how V3 is trained, and it can make writing faster.
hik−1 is the hidden state at position i from the level below; for k=1 it is the main model's output.
Emb(ti+k) is the embedding of the real token k steps ahead; [⋅;⋅] joins two vectors of 7,168 into one of 14,336.
Mk maps 14,336 numbers back to 7,168; TRMk is one full transformer block (MLA plus an MoE layer in V3).
Pi+k+1k is the predicted distribution for the token k+1 steps ahead; LMTPk is its cross-entropy loss.
D=1 is the number of MTP modules, and λ is their weight in the total loss.
Our toy run prints which targets each part learns, on a 6-token sentence:
plain text
== Which token each depth predicts (T = 6 tokens, D = 1) == position 1 (The): main model predicts t2 = 'cat' MTP module (also given the embedding of t2) predicts t3 = 'sat' position 2 (cat): main model predicts t3 = 'sat' MTP module (also given the embedding of t3) predicts t4 = 'on' position 3 (sat): main model predicts t4 = 'on' MTP module (also given the embedding of t4) predicts t5 = 'the' position 4 ( on): main model predicts t5 = 'the' MTP module (also given the embedding of t5) predicts t6 = 'mat'
Multi-token prediction on a short sentence. At each position the main model predicts the next token. The MTP module takes the main model's hidden state and the embedding of that next token, and predicts the token after it.
The weight λ was 0.3 for the first 10T training tokens and 0.1 for the last 4.8T (page 23). Take an illustrative main loss of 2.0 and MTP loss of 2.6; we chose these, they are not from the paper. The total is 2.0+0.3×2.6=2.78 early in training and 2.0+0.1×2.6=2.26 late. The extra signal is strong early and fades as training ends.
Why bother? The paper says MTP "densifies the training signals" and "may enable the model to pre-plan its representations". Its ablation (Table 4, page 26) adds one MTP module to a 228.7B-parameter MoE model trained on 540B tokens. Most scores go up, for example HumanEval from 44.5 to 53.7 and GSM8K from 72.3 to 74.0. A few go down slightly, such as MMLU from 67.5 to 66.6.
With one draft token accepted with probability p, each step yields 1+p tokens on average:
plain text
== Tokens per decoding step with one draft token == acceptance 85%: 1 + 0.85 = 1.85 tokens per step (upper bound on the speed-up) acceptance 90%: 1 + 0.90 = 1.90 tokens per step (upper bound on the speed-up) reported by DeepSeek: 1.8x tokens per second; the gap is the cost of running the MTP layer and verifying
The reported 1.8× sits just under our 1.85 to 1.90 ceiling, which is what you would expect once the extra layer's cost is paid. In the V4 releases from July 2026, DeepSeek replaced this drafter with a new one called DSpark; Part 6 covers it.
Now we can count everything. part2_params.py reads the real V3 config.json and counts each weight matrix with a formula (Method 1). Then it builds DeepseekV3ForCausalLM from the same config on PyTorch's "meta" device, which creates every tensor's shape but no memory. It counts what transformers made (Method 2). The heart of Method 1 is one line for attention:
python
q = d * nh * (nope + rope) if qr is None else d * qr + qr + qr * nh * (nope + rope) attn = q + d * (kvr + rope) + kvr + kvr * nh * (nope + vd) + nh * vd * d
Read it as the MLA flow figure, box by box. First the query down-projection (7,168 × 1,536) and its norm, then the query up-projection to 128 heads of 192. Next the KV down-projection to 512 + 64 and its norm, and the KV up-projection to 128 heads of 128 + 128. Last comes the output matrix. That is 187,107,328 parameters per layer. The output:
plain text
== Per-piece sizes == MLA attention, one layer 187,107,328 dense SwiGLU FFN, one layer 396,361,728 one expert (SwiGLU) 44,040,192 router, one MoE layer 1,835,264 dense layers: 3, MoE layers: 58== Method 1: our formula, total parameters == embedding 0.927B 0.14% output head 0.927B 0.14% attention (MLA), all layers 11.414B 1.70% dense FFN, first layers 1.189B 0.18% routed experts 653.909B 97.45% shared experts 2.554B 0.38% routers (incl. bias) 0.106B 0.02% RMSNorm weights 0.001B 0.00% TOTAL main model 671.026B (671,026,419,200)== Method 2: transformers DeepseekV3ForCausalLM on the meta device == parameters + balancing bias 671.026B (671,026,419,200) equal to Method 1: True
The two counts agree to the last parameter: 671,026,419,200. The paper's "671B" is exact, not a rounding of something larger. And 97.45% of the model is routed experts. Attention, the part most explanations of transformers spend their time on, is 1.7%.
Now count what one token touches: everything except the 248 routed experts it did not pick in each MoE layer.
plain text
== Active parameters for one token == embedding 0.927B output head 0.927B attention (MLA), all layers 11.414B dense FFN, first layers 1.189B routed experts 20.435B shared experts 2.554B routers (incl. bias) 0.106B RMSNorm weights 0.001B TOTAL active 37.552B without the input embedding (a table lookup, not a matrix multiply): 36.626B share of all parameters used per token: 5.6%
Where DeepSeek-V3's parameters are, from our count of config.json. Top: all 671.0 billion; routed experts are 97.4 percent. Bottom: the 37.6 billion one token uses; routed experts are 54 percent, attention 30 percent.
Per token, the picture flips. Routed experts drop to 20.4B (8 of 256 in 58 layers), and attention becomes the second-largest share at 30%. So the paper's "37B" checks out: 37.55B if you count the input embedding, 36.63B if you do not. DeepSeek's weight notes say 36.7B "including 0.9B for Embedding and 0.9B for the output Head"; we could not reproduce that exact figure: with both counted, we get 37.55B, a gap of about 0.85B.
== The MTP module (one extra layer, num_nextn_predict_layers = 1) == enorm + hnorm + eh_proj (7168 x 14336) + one full MoE layer + final norm = 11.610B plus its stored copies of the embedding and output head = 1.853B checkpoint on Hugging Face: 671.026B + 11.610B + 1.853B = 684.490B
The MTP module is a full MoE layer (11.5B of its 11.61B) plus the Mk projection and norms. In the checkpoint it also stores its own copies of the embedding and output head. That sums to 684.49B. DeepSeek's README rounds it to "685B, which includes 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights." The 685B on the model page is the same model plus its drafter.
Finally, a second real config as a check. The cached DeepSeek-V2-Lite config gives:
plain text
== Check on a second real config: DeepSeek-V2-Lite-Chat (published 15.7B total, 2.4B active) == Method 1 total 15.706B, Method 2 total 15.706B, equal: True Method 1 active 2.661B, without the input embedding 2.451B
The total matches the paper's 15.7B. The active count matches its 2.4B when the input embedding is left out, which suggests that is the convention DeepSeek used for V2-Lite.
None of these ideas is free. Many experts mean a large memory footprint: the paper's own limits section says "the recommended deployment unit for DeepSeek-V3 is relatively large, which might pose a burden for small-sized teams" (page 35). Expert parallelism needs fast networks and careful kernels. Node-limited routing gives up some choice of experts for less traffic. MLA saves memory but needs absorption tricks and special kernels to be fast.
The payoff is a model that does the work of a roughly 37B model per token, while it holds the knowledge of a 671B one. Its cache is small enough for 128K-token conversations. The same skeleton, with small changes, carries every DeepSeek model through V3.2.
Next, Part 3 follows this architecture through training. It covers the 14.8 trillion tokens, FP8 arithmetic and the DualPipe schedule that hides the all-to-all traffic described here. It also covers the cluster, and what the famous training cost does and does not include.