Running DeepSeek on Your Own Hardware, and How DeepSeek Serves It

How much memory each DeepSeek model needs, which GPUs, Macs and CPU servers run it at what reported speed, and how DeepSeek serves it at scale.

On one day in February 2025, DeepSeek served its chat models on an average of 226.75 server nodes, each with eight H800 GPUs. In that one day the fleet read 608 billion input tokens and wrote 168 billion output tokens. A few weeks later, a reviewer ran the same 671-billion-parameter model on one Mac Studio. It wrote about 17 to 18 tokens per second.

Both facts are true, and both come down to one question: where do the bytes go? A model is a very large file of numbers. Before it can answer, those numbers must sit in fast memory, and a few of them must be read for every word it writes. This part does that arithmetic for every major DeepSeek model, from the 1.5B distill on a laptop to the 1.6-trillion-parameter V4-Pro. Then it shows the real recipes people use, and how DeepSeek itself runs the service.

Part 6 ended with the V4 and V4.1 architectures and their tiny KV caches. Here those design choices meet real machines.

Memory arithmetic

Bytes per weight

The weights must sit somewhere the processor can reach quickly, so the first number is their size:

Mweights=N×b8M_{\text{weights}} = N \times \frac{b}{8}

Here MweightsM_{\text{weights}} is the memory in bytes, NN is the number of weights, and bb is the number of bits per weight. The division by 8 turns bits into bytes.

Now the real model. The script part7_fits.py starts with the rule and then reads the real file sizes:

python
for fmt, bits in [('BF16', 16), ('FP8', 8), ('INT4 / FP4', 4)]:
    log(f'  671B in {fmt:10s}: 671e9 x {bits:2d} / 8 = {671e9 * bits / 8 / GB:7,.1f} GB')
...
for m, f, gb, n, src in FILES:
    bits = gb * GB * 8 / n          # effective bits per weight of the real file
plain text
== 1. Bytes per weight: the rule ==
memory = parameters x bits per weight / 8
  671B in BF16      : 671e9 x 16 / 8 = 1,342.0 GB
  671B in FP8       : 671e9 x  8 / 8 =   671.0 GB
  671B in INT4 / FP4: 671e9 x  4 / 8 =   335.5 GB

== Real files: size and effective bits per weight ==
model                    format                  GB      GiB bits/weight
V3 / R1 (671B)           BF16 GGUF          1,342.3  1,250.1       16.00
V3 / R1 (671B)           GGUF Q8_0            713.3    664.3        8.50
V3 / R1 (671B)           official FP8         688.6    641.3        8.05
V3 / R1 (671B)           GGUF Q4_K_M          404.4    376.6        4.82
V3 / R1 (671B)           UD-Q2_K_XL           226.6    211.0        2.70
V3 / R1 (671B)           UD-IQ1_S             140.2    130.6        1.67
V3.2 (671B)              official FP8         689.5    642.1        8.06
V3.2 (671B)              UD-IQ1_S             184.1    171.5        2.19
V2 (236B)                official BF16        471.5    439.1       16.00
V2-Lite (16B)            official BF16         31.4     29.2       16.00
V4-Flash (284B)          official FP4+FP8     159.6    148.6        4.50
V4-Flash (284B)          UD-Q2_K_XL            96.8     90.2        2.73
V4-Pro (1.6T)            official FP4+FP8     864.7    805.3        4.32
V4.1-Flash (552B+196B)   official FP4+FP8     510.3    475.3        5.46
R1-Distill-Llama-70B     official BF16        141.1    131.4       16.00
R1-Distill-Llama-70B     GGUF Q4_K_M           42.5     39.6        4.82

The sizes are the sums of the real file listings on Hugging Face, checked on 10 November 2026. Three things in this table are worth a second look.

First, the official V3 file is 688.6 GB, not 671 GB. The repository holds 684.5 billion weights: the 671B main model plus a 14B module for multi-token prediction (MTP, explained below). About 3.9 billion of those weights are small tensors kept in BF16, and every 128 x 128 block of FP8 weights carries a scale. That is why the official file averages 8.05 bits, not exactly 8.

Second, a "4-bit" file is not 4.00 bits. The common GGUF format Q4_K_M averages 4.82 bits on V3, because some tensors keep more bits and every block stores its scales.

Third, the V4 family already ships its experts in FP4. The official V4-Pro file averages 4.32 bits per weight. So the usual trick of "download a 4-bit version" barely helps for V4: the official file is already about that small.

One 671B model, six file sizes (DeepSeek-V3 / R1)512 GB Mac8x H1008x H200BF16 GGUF (16.00 bits)BF16 GGUF (16.00 bits): 1,342.3 GB1,342.3 GBGGUF Q8_0 (8.50 bits)GGUF Q8_0 (8.50 bits): 713.3 GB713.3 GBofficial FP8 (8.05 bits)official FP8 (8.05 bits): 688.6 GB688.6 GBGGUF Q4_K_M (4.82 bits)GGUF Q4_K_M (4.82 bits): 404.4 GB404.4 GBUD-Q2_K_XL (2.70 bits)UD-Q2_K_XL (2.70 bits): 226.6 GB226.6 GBUD-IQ1_S (1.67 bits)UD-IQ1_S (1.67 bits): 140.2 GB140.2 GBGB on disk (decimal). Dashed lines: memory of common machines.
The same 671B DeepSeek-V3/R1 model as six files. The three smallest files fit a 512 GB Mac; the native FP8 file needs more memory than eight H100 GPUs have.

The GB versus GiB trap

In January 2025, the company Unsloth published a "1.58-bit" version of R1 and said it was 131 GB. The folder on Hugging Face adds up to 140.2 GB. Both numbers are right, because they use different units.

plain text
== 2. The GB versus GiB trap ==
R1 UD-IQ1_S ("131 GB" in the blog): 140.2 GB = 140,200,000,000 bytes = 130.6 GiB
R1 UD-Q2_K_XL ("212 GB" in the blog): 226.6 GB = 226,600,000,000 bytes = 211.0 GiB
average bits per weight of the "1.58-bit" file: 140.2e9 x 8 / 671e9 = 1.67

So the "131 GB" file is 131 GiB. On a machine where every gigabyte counts, a 7% mistake is the difference between "fits" and "does not fit". The output also shows a second trap: the "1.58-bit" label describes most expert weights, not the file. The file averages 1.67 bits, because Unsloth kept the first dense layers and other sensitive tensors at higher precision.

What all those bytes are

DeepSeek-V3 is a mixture of experts (Part 2 builds it from zero). Each of its 58 MoE layers holds 256 small feed-forward networks, the routed experts. A router picks 8 of them for each token. The same script counts where the weights live, straight from the model's config.json:

plain text
== 3. DeepSeek-V3: where the 671B weights live (from config.json) ==
one expert = 3 x 7168 x 2048 = 44,040,192 parameters
routed experts: 58 layers x 256 x 44,040,192 = 653.91B (97.4%)
everything else (attention, dense layers, shared experts, routers, embedding, head) = 17.12B (2.6%)
total = 671.03B
per token: 8 of 256 routed experts in each of 58 layers = 20.43B parameters
at Q4_K_M (4.82 bits): one expert = 26.5 MB; routed experts read per token = 12.3 GB out of 394 GB; the rest = 10.3 GB

This is the most important table in the part. Of 671 billion weights, 97.4% are routed experts. A token reads only 20.4 billion of them, plus the 17.1 billion that every token uses. That gives the "37B active" in the V3 paper.

DeepSeek-V3's 671B weights: stored versus used for one tokenstored (all weights must be somewhere)routed experts: 653.9B (97.4%)everything else: 17.1B (2.6%), always usedread for one token (37B)20.4B of experts (8 of 256 per layer) + 17.1B alwaysMemory size follows the top bar. Decode speed follows the bottom bar.
DeepSeek-V3 stores 653.9B weights of routed experts but one token reads only 20.4B of them, plus 17.1B weights that every token uses. Storage follows the top bar; work per token follows the bottom bar.

Hold on to the two bars. Memory size follows the top bar: every expert must be stored, because the next token may pick any of them. Speed follows the bottom bar: each new token reads about 37 billion weights, not 671 billion. A dense model of 671B would read all of them for every token.

The other memory bill: the KV cache

Weights are not the whole bill. While the model reads a prompt and writes an answer, it keeps notes about every token so far. These notes are the KV cache. LLM Inference Part 2 explains the cache from zero; here we only need its size.

The size is "bytes per token x number of tokens". Bytes per token depend on the attention design. DeepSeek changed that design three times, and each change shrank the cache.

V2 and V3 use Multi-head Latent Attention (MLA). Per layer, it stores one compressed vector of 512 numbers plus a 64-number position key, shared by all 128 heads. Attention Part 2 explains MLA, and Part 2 of this series shows it inside V3.

Btoken=(dc+dR)×L×sB_{\text{token}} = (d_c + d_R) \times L \times s

Read it left to right. BtokenB_{\text{token}} is the KV cache in bytes for one token. dcd_c is the size of the compressed vector (512). dRd_R is the size of the shared position key (64). LL is the number of layers. ss is the bytes per stored number (2 for BF16).

Tiny example: a toy model with dc=4d_c = 4, dR=2d_R = 2, 3 layers and BF16 stores (4+2)×3×2=36(4+2) \times 3 \times 2 = 36 bytes per token. A prompt of 10 tokens needs 360 bytes. For real V3: (512+64)×61×2=70,272(512 + 64) \times 61 \times 2 = 70{,}272 bytes per token, about 69 KiB.

Every model, from its config

The script reads each model's config.json and computes bytes per token with the formula that fits its attention type. The distills are normal Qwen and Llama models with grouped-query attention. Their formula is 2×L×hkv×dh×22 \times L \times h_{kv} \times d_h \times 2: keys and values, layers, key-value heads, head size and 2 bytes. For V3.2, V4 and V4.1, the script uses the FP8 and FP4 cache layouts from DeepSeek's FlashMLA kernels.

python
def mla_bf16(c):
    return (c['kv_lora_rank'] + c['qk_rope_head_dim']) * c['num_hidden_layers'] * 2

def v32(c):     # FlashMLA FP8 row 656 B (latent + scales + RoPE) + FP8 indexer key 128 B + 4 B scale
    return c['num_hidden_layers'] * (656 + c['index_head_dim'] + 4)

def v4(c):      # V4 row 584 B per compressed entry; FP4 indexer key 64 B + 4 B scale on CSA layers
    r = c['compress_ratios'][:c['num_hidden_layers']]
    csa = sum(1 for x in r if x == 4)
    hca = sum(1 for x in r if x == 128)
    return csa * (584 + 68) / 4 + hca * 584 / 128, csa, hca
plain text
== 4. KV cache per token, from each config.json ==
naive multi-head cache for V3 (for contrast): 128 heads x (128 + 64 + 128) x 61 x 2 B = 4,997,120 B per token
model                B/token       32K      128K        1M   how
V2-Lite               31,104     1.02G       n/a       n/a   MLA BF16: (512 + 64) x 27 layers x 2 B
V2                    69,120     2.26G     9.06G       n/a   MLA BF16: (512 + 64) x 60 layers x 2 B
V3 / R1 / V3.1        70,272     2.30G     9.21G       n/a   MLA BF16: (512 + 64) x 61 layers x 2 B
V3.2                  48,068     1.58G     6.30G       n/a   FP8 MLA row 656 B + indexer 132 B, x 61 layers
V4-Flash               3,514     0.12G     0.46G     3.68G   21 CSA layers x 652 B / 4 + 20 HCA layers x 584 B / 128
V4-Pro                 5,031     0.16G     0.66G     5.28G   30 CSA layers x 652 B / 4 + 31 HCA layers x 584 B / 128
V4.1-Flash               890     0.03G     0.12G     0.93G   published: 890 B per token (V4.1 report, abstract)
Distill-Qwen-1.5B     28,672     0.94G     3.76G       n/a   GQA BF16: 2 x 28 layers x 2 KV heads x 128 x 2 B
Distill-Qwen-7B       57,344     1.88G     7.52G       n/a   GQA BF16: 2 x 28 layers x 4 KV heads x 128 x 2 B
Distill-Llama-8B     131,072     4.29G    17.18G       n/a   GQA BF16: 2 x 32 layers x 8 KV heads x 128 x 2 B
Distill-Qwen-14B     196,608     6.44G    25.77G       n/a   GQA BF16: 2 x 48 layers x 8 KV heads x 128 x 2 B
Distill-Qwen-32B     262,144     8.59G    34.36G       n/a   GQA BF16: 2 x 64 layers x 8 KV heads x 128 x 2 B
Distill-Llama-70B    327,680    10.74G    42.95G       n/a   GQA BF16: 2 x 80 layers x 8 KV heads x 128 x 2 B
checks against published numbers:
  V3 70,272 B = ISCA 2025 Table 1 "70.272 KB"
  V3.2 48,068 B, V4-Flash 3,514 B: V4.1 report Figure 1(b) gives 48,068 and 3,514
  V4-Pro / V3.2 = 10.5% (V4 report: "10% of KV cache"); V4-Flash / V3.2 = 7.3% (report: "7%")
  V3 MLA versus naive multi-head: 71x smaller; Distill-Llama-8B needs 1.87x more per token than the 671B V3

"n/a" means the model does not support that context length. The checks at the bottom matter. Our numbers for V3, V3.2 and V4-Flash match the numbers DeepSeek published, byte for byte. V4-Pro has no published per-token figure, so 5,031 bytes is our estimate. It gives 10.5% of V3.2, and the V4 report says "10%". The estimate leaves out the small sliding-window cache, which has a fixed size of 128 tokens per layer and does not grow with the prompt.

KV cache per token (log scale)1 KB10 KB100 KB1 MBnaive multi-head (V3 shape)naive multi-head (V3 shape): 4,997,120 B4,997,120 BR1-Distill-Llama-70B (GQA)R1-Distill-Llama-70B (GQA): 327,680 B327,680 BR1-Distill-Llama-8B (GQA)R1-Distill-Llama-8B (GQA): 131,072 B131,072 BV3 / R1 / V3.1 (MLA, BF16)V3 / R1 / V3.1 (MLA, BF16): 70,272 B70,272 BV3.2 (FP8 MLA + indexer)V3.2 (FP8 MLA + indexer): 48,068 B48,068 BV4-Pro (CSA + HCA), estimateV4-Pro (CSA + HCA), estimate: 5,031 B5,031 BV4-Flash (CSA + HCA)V4-Flash (CSA + HCA): 3,514 B3,514 BV4.1-Flash (CSA2, FP4)V4.1-Flash (CSA2, FP4): 890 B890 Bbytes per token, log scale: each grid step is 10x
KV cache bytes per token on a log scale. A naive multi-head cache for V3's shape would need about 5 MB per token; MLA brings it to 70 KB, V3.2 to 48 KB, V4 to 3.5 to 5 KB, and V4.1-Flash to 890 bytes.

Two lessons sit in this table. The small 8B Llama distill needs 1.87 times more cache per token than the 671B V3. And V4.1-Flash keeps a one-million-token conversation in 0.93 GB, less than V3 needs for 16K tokens.

What fits where

Now we can answer the question people actually ask: will it run on my machine? The rule we use is an estimate, and it is strict on purpose. A setup fits if the weights plus the KV cache of one 32K-token conversation take at most 90% of the memory. The other 10% goes to activations, engine buffers and the operating system.

python
for label, w, kvm in ROWS:
    need = w + MODELS[kvm][0] * CTX / GB          # weights + KV cache of one 32K-token sequence
    row = [cap * USABLE >= need for _, cap in MACHINES]
plain text
== 5. What fits where: weights + KV for one 32,768-token sequence <= 90% of memory ==
(capacities taken as decimal GB; RAM sold as "512 GB" is really 512 GiB, so this errs on the safe side)
setup                           need GB  16GB lap  24GB GPU  32GB GPU 128GB box 512GB Mac   8x H100   8x H200  2x8 H100 8x MI300X
V3/R1 FP8 official                690.9         -         -         -         -         -         -       yes       yes       yes
V3/R1 Q4_K_M                      406.7         -         -         -         -       yes       yes       yes       yes       yes
R1 UD-Q2_K_XL                     228.9         -         -         -         -       yes       yes       yes       yes       yes
R1 UD-IQ1_S                       142.5         -         -         -         -       yes       yes       yes       yes       yes
V3.2 FP8 official                 691.1         -         -         -         -         -         -       yes       yes       yes
V2 BF16 official                  473.8         -         -         -         -         -       yes       yes       yes       yes
V2 4-bit (Ollama)                 135.3         -         -         -         -       yes       yes       yes       yes       yes
V2-Lite 4-bit (Ollama)              9.9       yes       yes       yes       yes       yes       yes       yes       yes       yes
V4-Pro-0813 official              892.9         -         -         -         -         -         -       yes       yes       yes
V4-Flash official                 159.7         -         -         -         -       yes       yes       yes       yes       yes
V4-Flash UD-Q2_K_XL                96.9         -         -         -       yes       yes       yes       yes       yes       yes
V4.1-Flash-1017 official          510.3         -         -         -         -         -       yes       yes       yes       yes
V4.1 UD-IQ1_S, Engram on SSD      122.4         -         -         -         -       yes       yes       yes       yes       yes
Distill-Llama-70B Q4_K_M           53.2         -         -         -       yes       yes       yes       yes       yes       yes
Distill-Qwen-32B Q4_K_M            28.5         -         -       yes       yes       yes       yes       yes       yes       yes
Distill-Qwen-14B Q4_K_M            15.4         -       yes       yes       yes       yes       yes       yes       yes       yes
Distill-Qwen-7B Q4_K_M              6.6       yes       yes       yes       yes       yes       yes       yes       yes       yes
Weights + one 32K-token KV cache within 90% of memory16GB lap24GB GPU32GB GPU128GB box512GB Mac8x H1008x H2002x8 H1008x MI300XV3/R1 FP8 official (691)fitsfitsfitsV3/R1 Q4_K_M (407)fitsfitsfitsfitsfitsR1 UD-Q2_K_XL (229)fitsfitsfitsfitsfitsR1 UD-IQ1_S (142)fitsfitsfitsfitsfitsV3.2 FP8 official (691)fitsfitsfitsV2 BF16 official (474)fitsfitsfitsfitsV2 4-bit (Ollama) (135)fitsfitsfitsfitsfitsV2-Lite 4-bit (Ollama) (10)fitsfitsfitsfitsfitsfitsfitsfitsfitsV4-Pro-0813 official (893)fitsfitsfitsV4-Flash official (160)fitsfitsfitsfitsfitsV4-Flash UD-Q2_K_XL (97)fitsfitsfitsfitsfitsfitsV4.1-Flash-1017 official (510)fitsfitsfitsfitsV4.1 UD-IQ1_S, Engram on SSD (122)fitsfitsfitsfitsfitsDistill-Llama-70B Q4_K_M (53)fitsfitsfitsfitsfitsfitsDistill-Qwen-32B Q4_K_M (28)fitsfitsfitsfitsfitsfitsfitsDistill-Qwen-14B Q4_K_M (15)fitsfitsfitsfitsfitsfitsfitsfitsDistill-Qwen-7B Q4_K_M (7)fitsfitsfitsfitsfitsfitsfitsfitsfitsNumber in brackets: GB needed. Filled cell: fits by memory alone (not a speed claim).
Which DeepSeek setups fit which machines by memory alone. The native FP8 V3 and the official V4-Pro need at least eight H200 GPUs; low-bit V3 files need a 512 GB machine; V4-Flash at 2 bits and the 70B distill fit a 128 GB box.

The grid answers "does it fit", not "is it fast" or "does the software support it". Three notes keep it honest.

The 8x H100 column is the famous gap. Eight H100 GPUs have 8 x 80 = 640 GB, and the native V3 file is 688.6 GB. That is why every guide says V3 in FP8 needs either 8 H200s (1,128 GB) or two H100 nodes. The column also says V4.1-Flash fits 8x H100 by memory. In practice, DeepSeek's own V4.1 engine targets newer NVIDIA SM100 GPUs and Huawei Ascend 950 chips, so memory is not the only gate.

The 128 GB column is where V4-Flash at 2 bits appears, and where V4.1-Flash with its Engram tables on SSD comes close. V4.1-Flash stores a large "Engram" memory of n-gram tables (Part 6). It reads only a few rows per token, so the tables can stay on an SSD. Unsloth's table splits each V4.1 file into "Engram" and "Rest".

For UD-IQ1_S, 110.6 GB is Engram and 122.4 GB is the rest, so the rest is what must sit in RAM or GPU memory. By our 90% rule, 122.4 GB is just too much for a 128 GB box. Unsloth itself lists 128 to 145 GB of RAM plus VRAM for its 1-bit V4.1 files.

The distills are a different kind of model. They are Qwen 2.5 and Llama 3 models, fine-tuned on R1's answers. They have no MLA and no experts. They fit laptops because they are small, not because of DeepSeek's architecture.

Data-centre recipes: GPUs, flags and why

Splitting one model over many GPUs

When a model does not fit one GPU, engines split it. There are three basic ways, and real engines mix them. LLM Inference Part 7 covers them in depth.

Three ways to spread one model over 4 GPUsTensor parallel (TP)GPU 0L1L2L3L4GPU 1L1L2L3L4GPU 2L1L2L3L4GPU 3L1L2L3L4every GPU holds 1/4 of every matrix;they talk at every layerData parallel (DP)GPU 0full model, requests AGPU 1full model, requests BGPU 2full model, requests CGPU 3full model, requests Devery GPU holds a full copy;each serves different requestsExpert parallel (EP)GPU 0experts 0-63GPU 1experts 64-127GPU 2experts 128-191GPU 3experts 192-255each GPU holds different experts;tokens travel to their expertsTP: box L1 on GPU 0 is one quarter of layer 1. Real engines mix all three: DP for attention, EP for experts.
Three ways to spread one model over four GPUs. Tensor parallelism slices every matrix; data parallelism copies the model and splits the requests; expert parallelism gives each GPU different experts.

DeepSeek's models suit a mix. The attention part (MLA) is small, so engines often copy it on every GPU with DP. The experts are 97% of the weights, so engines spread them with EP. SGLang calls this "DP attention".

Speculative decoding with MTP and DSpark

One more idea appears in every recipe. During training, V3 also learned to predict the token after the next one. That extra head is the MTP module. At serving time, it can propose a second token for free, and the main model checks it. LLM Inference Part 4 explains why checking is cheap.

V4 replaced the single MTP head with DSpark, a drafter that proposes several tokens at once (Part 6). The V4-Pro-0813 checkpoint ships with its DSpark module attached, which is one reason it is 892.7 GB, against 864.7 GB for the April preview.

The recipes, model by model

All flags below are copied from the engine docs or DeepSeek's model cards. Numbers are vendor claims unless the text says otherwise.

V3, R1, V3.1 on one node of 8x H200 or 8x MI300X. The native FP8 file fits, with about 440 GB left for KV cache and buffers. SGLang's launch line is short:

bash
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8
#   add --reasoning-parser deepseek-v3 for R1-style thinking output
#   add --attn-dp-size 8 for high-throughput, large-batch serving (DP attention)
#   add --speculative-algorithm EAGLE for MTP speculation
#       (defaults: --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4)

The SGLang page warns not to add --quantization fp8, because the checkpoint is already FP8. It reports MTP "up to 1.8x decoding speed at batch size 1 and 1.5x at batch size 32" on 8x H200. On 8x MI300X with SGLang 0.5.7 and 1K-token inputs and outputs, it reports 34.26 output tokens per second at one request with EAGLE. At 100 concurrent requests with --ep 8 --attn-dp-size 8, the total is 463.80 output tokens per second. The gain for one user is small, the gain for the whole server is large, exactly as the MTP box predicted.

vLLM's recipe uses the same hardware:

bash
vllm serve deepseek-ai/DeepSeek-R1-0528 --trust-remote-code --tensor-parallel-size 8 --enable-expert-parallel
#   or: --data-parallel-size 8 --enable-expert-parallel
#   AMD ROCm: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1

V3 on H100, H800, H20 or A100. The SGLang hardware table lists FP8 V3 on "2x8 H100/H800/H20", that is, two nodes with tensor parallelism across both. If you upcast to BF16 (1.34 TB), you need 2x8 H200, 4x8 H100 or 4x8 A100. There is also an INT8 path on 16 A100s, 32 L40S cards, a Xeon 6980P CPU server, or 4x Huawei Ascend A3. With 4-bit weights (W4A8, AWQ, NVFP4 and similar), one node of 8x H20 or 8x H100 is enough.

V3.2 and V3.2-Exp. DeepSeek's own README gives SGLang images for H200, AMD MI350 and Huawei Ascend, launched with --tp 8 --dp 8 --enable-dp-attention. The vLLM recipe recommends -dp 8 --enable-expert-parallel on 8x H200, 8x H20 or 8x B200, and says the sparse-attention kernels are "mainly optimized for TP=1 (DP=8, EP=8)". It uses an FP8 KV cache by default. Some H20 users found VLLM_USE_DEEP_GEMM=0 faster. Part 5 explains the sparse attention these kernels run.

V4-Pro and V4-Flash. DeepSeek's V4-Pro-0813 model card gives two launch lines. Each turns on DSpark, an FP8 KV cache and expert parallelism:

bash
# vLLM, one node of 4x GB300
vllm serve deepseek-ai/DeepSeek-V4-Pro-0813 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --data-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe \
  --attention-config '{"use_fp4_indexer_cache": true}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

# SGLang
sglang serve --model-path deepseek-ai/DeepSeek-V4-Pro-0813 --tp 4 --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK --mem-fraction-static 0.90 --chunked-prefill-size 4096 --swa-full-tokens-ratio 0.1

The vLLM recipe defaults V4-Pro to 8x H200 and caps the context at 800,000 tokens to keep room for the KV cache.

SGLang's cookbook puts V4-Flash on 4 B200, B300, GB200, GB300 or H200 GPUs. It also fits 2 RTX PRO 6000 cards (96 GB each) or 8 H100s. V4-Pro needs 8 B200s or B300s, 4 GB300s, or one 8x H200 node with FP4 weights. With FP8 weights on H200, or on H100, it needs two nodes. The cookbook says V4-Pro "does not fit on 8x 96 GB RTX PRO 6000", and that "a single 128 GB GB10 cannot hold any of these checkpoints". Neither recipe page publishes tokens-per-second results for V4 that we could read.

V4.1-Flash. DeepSeek's announcement says it "will work closely with the open-source community on V4.1-Flash inference support". The reference code splits the weights for 8-way tensor parallelism. The newest option is DeepSeek's own engine, DeepInfer, covered below.

Why MoE models suit CPU and GPU together

The speed rule for writing tokens

Writing one token means reading every active weight once from memory. On a single user's machine, the processor spends most of its time waiting for those bytes, not doing maths. LLM Inference Part 1 shows why.

This gives a simple ceiling for one user:

tokens per second≤bandwidthbytes read per token\text{tokens per second} \le \frac{\text{bandwidth}}{\text{bytes read per token}}

The bytes read per token are the active weights times the bytes per weight. It is a ceiling, not a forecast: real systems also read the KV cache, run attention, and never reach full bandwidth.

Tiny example: memory that delivers 100 GB/s, and a model that reads 10 GB per token, can write at most 10 tokens per second. Now the real numbers from part7_serving.py:

plain text
== 1. Decode ceiling: tokens/s <= bandwidth / bytes read per token ==
M3 Ultra, "over 800GB/s"                         V3/R1 Q4_K_M, 37B active                      22.3 GB/token -> at most  35.9 tok/s
M3 Ultra, "over 800GB/s"                         R1-Distill-Llama-70B Q4_K_M, 70.55B dense     42.5 GB/token -> at most  18.8 tok/s
2 sockets x 8 ch DDR5-4800 (2 x 8 x 38.4 GB/s)   V3/R1 Q4_K_M, 37B active                      22.3 GB/token -> at most  27.6 tok/s
2 sockets x 8 ch DDR5-4800 (2 x 8 x 38.4 GB/s)   R1-Distill-Llama-70B Q4_K_M, 70.55B dense     42.5 GB/token -> at most  14.5 tok/s
measured (third party): R1 4-bit on M3 Ultra 512 GB about 17-18 tok/s (short prompts);
V3-0324 Q4_K_M: 10.89 tok/s after a 4.8K prompt, 6.67 tok/s after a 12.7K prompt
share of the ceiling reached at 17.5 tok/s: 49%

Look at the first two lines. The 671B MoE model has a higher speed ceiling than the 70B dense distill, because it reads fewer bytes per token. That is the whole reason a 671B model is usable on one desk at all.

M3 Ultra (over 800 GB/s): ceiling versus reported speedceiling, V3/R1 Q4_K_M (37B active)ceiling, V3/R1 Q4_K_M (37B active): at most 35.9 tok/sat most 35.9 tok/smeasured, R1 4-bit, short promptmeasured, R1 4-bit, short prompt: about 17-18 tok/sabout 17-18 tok/smeasured, V3-0324 Q4_K_M, 4.8K promptmeasured, V3-0324 Q4_K_M, 4.8K prompt: 10.89 tok/s10.89 tok/smeasured, V3-0324 Q4_K_M, 12.7K promptmeasured, V3-0324 Q4_K_M, 12.7K prompt: 6.67 tok/s6.67 tok/sceiling, dense 70B distill Q4_K_Mceiling, dense 70B distill Q4_K_M: at most 18.8 tok/sat most 18.8 tok/soutput tokens per second (one user)
On an M3 Ultra, the bandwidth ceiling for V3/R1 at Q4_K_M is 35.9 tokens per second. Reports measure about 17-18 for short prompts and 6.67 to 10.89 after longer prompts. A dense 70B model has a lower ceiling, 18.8.

Experts can live in different places

The bandwidth rule has a second consequence. Only 8 of 256 experts are read per token. The other 248 can sit in slower, cheaper memory, as long as the chosen ones arrive in time. This turns memory into tiers, like a kitchen. The things you use every minute sit on the counter, the weekly things in the cupboard, the rare things in the basement.

Where the experts can live: three tiers, one tokenGPU memory (fast, small)attention, shared expert, KV cache, hot expertsRAM (slower, big)most routed expertsSSD (slowest, huge)cold experts or Engram tables, read on demandtokenA token reads only its chosen experts (filled boxes). The tier that holds each one sets how long the read takes.
Experts spread over three memory tiers. Attention, the shared expert, the KV cache and the most-used experts sit in GPU memory; most routed experts sit in RAM; cold experts or Engram tables can stay on an SSD. A token reads only its chosen experts.

The always-used part of V3 belongs on the counter. From the script: at Q4_K_M, everything except the routed experts is about 10.3 GB. Add a KV cache and buffers, and you get close to the "at least 14GB VRAM" that KTransformers asks for. The routed experts, 394 GB at Q4_K_M, go to RAM. For one token, the CPU reads about 12.3 GB of them.

The SSD tier is the newest. V4.1-Flash's Engram tables are made for it: the model reads only a few rows per token, so llama.cpp builds can map them from disk. Unsloth's guide uses --load-mode mmap --lazy-mode on for this. The same idea can be pushed onto experts, with a cache of recently used experts in RAM and the rest on an SSD. The catch is speed. An SSD delivers a few GB per second, far below RAM, so every miss costs a long wait.

Running on consumer machines

llama.cpp and GGUF files

llama.cpp has run DeepSeek V2 and V3 for a long time. Proper MLA support, with the small cache, arrived in pull request #12801, merged on 15 April 2025. V4 support landed in pull request #24162 on 29 June 2026. For V4.1-Flash, Unsloth's guide says to build from pull request #30817, which was closed without merging. As of 10 November 2026 we could not confirm V4.1 support in the main llama.cpp branch.

For the 1.58-bit R1 file, Unsloth reported (January 2025) "around 140 tokens per second for throughput and 14 tokens/s for single user inference" on two H100 80 GB GPUs. Unsloth staff wrote that one 24 GB RTX 4090 with the rest in RAM gives "at least 2-3 tokens/s". We read that second claim through a mirror, so treat it as unconfirmed.

KTransformers: experts on the CPU, the rest on the GPU

Its tutorial for V3 and R1 at Q4_K_M uses two Intel Xeon Gold 6454S CPUs with 32 cores each. The memory is DDR5-4800 with 8 channels per socket, and the GPU is one RTX 4090D with 24 GB. With one socket it needs 382 GB of RAM and at least 14 GB of GPU memory. With two sockets it needs 1 TB, because it copies the weights to both sockets' memory. The reported numbers, all from the KTransformers team for a 500-token prompt:

Setup (V3/R1 Q4_K_M)Prefill tok/sDecode tok/s
llama.cpp, same 2 x 32 cores10.314.51
KTransformers V0.2, 2 sockets, 8 experts82.9412.21
KTransformers V0.2, 2 sockets, 6 experts97.3213.69
KTransformers V0.3 preview (AMX), 8 / 6 experts255.26 / 286.55not given
KTransformers V0.2.1, 6 experts, short / 8K promptnot given16.8 / 13.9

KTransformers also runs V4-Flash on one RTX 5090 (32 GB) with an x86 CPU and at least 200 GB of RAM, and reports "20+ tok/s" decode. On a single Huawei Ascend die (61.3 GB) with 229 GB of host RAM, it reports 18.32 to 20.48 tokens per second. It notes that "Decode is host-memory-bandwidth bound". For R1-0528 with SGLang on a server with 8 L20 GPUs and a Xeon, it reports 87.58 output tokens per second at 8 parallel requests.

Apple Silicon

A Mac with Apple Silicon has one pool of unified memory shared by the CPU and GPU. A Mac Studio with an M3 Ultra can have up to 512 GB of it, at "over 800GB/s". Apple's own launch text says it can run "LLMs with over 600 billion parameters entirely in memory".

The reported speeds are good for short prompts and much worse for long ones. A reviewer measured R1 at 4 bits at "approximately 17-18 tokens per second". An independent llama.cpp-family test of V3-0324 Q4_K_M measured these numbers:

Prompt lengthPrompt reading (prefill)Writing (decode)
4.8K tokens64.02 tok/s, about 75 s10.89 tok/s
12.7K tokens58.82 tok/s, about 216 s6.67 tok/s

The waiting times are our arithmetic: 4,800 / 64.02 and 12,700 / 58.82. They show the weak spot. A Mac has the bandwidth for decode but much less raw compute than a data-centre GPU. So a long document takes minutes before the first word appears. Short demos hide this.

For V4-Flash, the project DwarfStar (ds4, by antirez) publishes its own Metal and CUDA measurements of a 2-bit file:

MachineContextPrefill tok/sDecode tok/s
M5 Max, 128 GB2,048790.1839.35
M5 Max, 128 GB65,536398.5027.64
DGX Spark, 128 GB2,048825.7618.05
DGX Spark, 128 GB65,536822.9813.84

V4-Flash has 13B active weights against V3's 37B, so a 128 GB laptop decodes it faster than a 512 GB Mac Studio decodes V3. For V4.1-Flash, ds4 says the 2-bit file runs on one 128 GB Mac or Spark with SSD streaming, and "Q4 needs SSD streaming or a 512 GB Mac". It publishes no speed for V4.1.

Reported decode speeds outside the data centreV4-Flash Q2, M5 Max 128 GB, 2K ctxV4-Flash Q2, M5 Max 128 GB, 2K ctx: 39.35 (ds4)39.35 (ds4)V4-Flash Q2, M5 Max 128 GB, 65K ctxV4-Flash Q2, M5 Max 128 GB, 65K ctx: 27.64 (ds4)27.64 (ds4)V4-Flash, 1x RTX 5090 + CPUV4-Flash, 1x RTX 5090 + CPU: "20+" (KTransformers)"20+" (KTransformers)V4-Flash Q2, DGX Spark, 2K ctxV4-Flash Q2, DGX Spark, 2K ctx: 18.05 (ds4)18.05 (ds4)R1 4-bit, M3 Ultra 512 GBR1 4-bit, M3 Ultra 512 GB: about 17-18 (review)about 17-18 (review)V3 Q4_K_M, 2x Xeon + 4090D, 6 expertsV3 Q4_K_M, 2x Xeon + 4090D, 6 experts: 13.69 (KTransformers)13.69 (KTransformers)V3 Q4_K_M, 2x Xeon + 4090D, 8 expertsV3 Q4_K_M, 2x Xeon + 4090D, 8 experts: 12.21 (KTransformers)12.21 (KTransformers)V3-0324 Q4_K_M, M3 Ultra, 4.8K promptV3-0324 Q4_K_M, M3 Ultra, 4.8K prompt: 10.89 (third party)10.89 (third party)V3 Q4_K_M, same Xeon box, llama.cppV3 Q4_K_M, same Xeon box, llama.cpp: 4.51 (KTransformers)4.51 (KTransformers)R1 1.58-bit, one RTX 4090 + RAMR1 1.58-bit, one RTX 4090 + RAM: "2-3" (Unsloth staff)"2-3" (Unsloth staff)output tokens per second, one user; source in bracketsTop four: V4-Flash (13B active). Rest: V3/R1 (37B active). Each from a different test.
Reported single-user decode speeds outside the data centre, from different tests and sources. V4-Flash (13B active) runs at 18 to 39 tokens per second on 128 GB machines; the 671B V3/R1 runs at about 11 to 18 on an M3 Ultra or a CPU server with one GPU.

Ollama for the distills

The R1 distills are the practical way to try DeepSeek-style reasoning on a laptop. Ollama's default tags are 4-bit files. The 1.5b tag is 1.1 GB, 7b is 4.7 GB and 8b is 5.2 GB. The 14b tag is 9.0 GB, 32b is 20 GB and 70b is 43 GB. The full deepseek-r1:671b tag is 404 GB.

The fits table gives a rough rule. A 16 GB laptop runs the 7B and 8B distills well, and a 24 GB GPU or a 32 GB Mac fits the 14B. The 32B at 4 bits needs about 28 GB with a 32K cache. The 70B needs 64 GB or more. We found no single trustworthy speed table for laptops, so measure your own.

How DeepSeek serves its own models

One model, two kinds of machine

A chat service does two different jobs. Reading a long prompt is one big parallel computation. Writing the answer is a long series of small steps, one token at a time, for thousands of users. On the same GPU, the two jobs get in each other's way: a big prefill pauses everybody's decode.

The February 2025 overview

On the last day of its "Open Source Week", DeepSeek published an overview of its V3/R1 serving system in the open-infra-index repository. It is a GitHub page, so we quote its text instead of a screenshot:

plain text
== 3. Where the 256 routed experts go ==
V3 paper, prefill unit (4 nodes)  : 256 + 32 redundant = 288 = 32 GPUs x 9 routed experts
Feb 2025, prefill unit (4 nodes)  : 256 + 32 redundant = 288 = 32 GPUs x 9 routed experts
Feb 2025, decode unit (18 nodes)  : 256 + 32 redundant = 288 = 144 GPUs x 2 routed experts
V3 paper, decode unit (40 nodes)  : 320 GPUs, one expert each: 256 routed + 64 for redundant and shared experts = 320
DeepSeek's serving split (Feb 2025, V3/R1 on H800)Prefill unit: 4 nodes, 32 GPUsreads the whole prompt at once99999999999999999999999999999999each GPU: 9 routed + 1 shared expert (EP32)KV cacheDecode unit: 18 nodes, 144 GPUswrites one token per step for every usereach GPU: 2 routed + 1 shared expert (EP144)Both units: 256 routed experts + 32 redundant copies = 288 slots (32 x 9 = 144 x 2).
DeepSeek's production split in February 2025: a 32-GPU prefill unit with 9 routed experts per GPU hands the KV cache to a 144-GPU decode unit with 2 routed experts per GPU.

Precision follows training: matrix multiplications and dispatch in FP8, core MLA and combine in BF16. And the fleet breathes with demand. DeepSeek runs inference on all nodes during the day and moves nodes to research and training at night.

How fast can expert parallelism be?

The ISCA 2025 paper does a back-of-the-envelope limit for this design. Each GPU holds one expert and handles about 32 tokens per step. Each token is sent to 9 experts (8 routed and 1 shared) in FP8 (1 byte per number) and returned in BF16 (2 bytes). The hidden size is about 7,000 numbers.

plain text
== 2. Expert-parallel communication limit (ISCA 2025, section 2.3.2) ==
CX7 400Gbps InfiniBand, 50 GB/s   : one all-to-all pair  120.96 us, x2 per layer x 61 layers =  14.76 ms ->    67.8 tok/s
GB200 NVL72, 900 GB/s             : one all-to-all pair    6.72 us, x2 per layer x 61 layers =   0.82 ms ->  1219.8 tok/s

The paper rounds 67.8 down to 67. In production, DeepSeek's users got an average of 20 to 22 tokens per second, about a third of this ideal.

One day of numbers, and the 545%

The overview ends with 24 hours of statistics, from 12:00 on 27 February 2025 to 12:00 the next day, Beijing time. V3 and R1 together peaked at 278 nodes and averaged 226.75 nodes of 8 H800s. 56.3% of input tokens hit DeepSeek's on-disk KV cache, so their prefill was skipped. Each node delivered about 73.7k input tokens per second in prefill (counting cache hits) or about 14.8k output tokens per second in decode.

Then it does a famous sum. If every token had been billed at R1's API prices, the day's revenue would have been $562,027 against a cost of $87,072. The overview calls this a "cost profit margin of 545%".

margin=R−CC\text{margin} = \frac{R - C}{C}

RR is the theoretical revenue for the day and CC is the cost. A tiny example: a lemonade stand that spends $10 and takes $30 has a margin of (30−10)/10=200%(30 - 10)/10 = 200\%. The real inputs, recomputed:

plain text
== 4. Feb 27-28 2025, 24 hours of V3 + R1 on H800 ==
cost    = 226.75 nodes x 8 GPUs x 24 h x $2 = $87,072
revenue = 342B hit x $0.14/M + 266B miss x $0.55/M + 168B out x $2.19/M
        = $47,880 + $146,300 + $367,920 = $562,100  (reported: $562,027)
margin  = (562,027 - 87,072) / 87,072 = 545%
cache-hit share of input: 56.25%
output share of the theoretical revenue: 65%
decode: 14.8k output tok/s per node = 1,850 per H800; at 20-22 tok/s per user, 673 to 740 streams per node

Our revenue is $73 higher than DeepSeek's because the token counts in the overview are rounded to billions. The 545% follows exactly from the published inputs.

One day of V3 + R1 (27-28 Feb 2025): cost versus theoretical revenuecost$87,072 (226.75 nodes x 8 x 24 h x $2)if all billedat R1 pricescache-hit input: $47,880cache-miss input: $146,300output: $367,920cache-hit input: $47,880cache-miss input: $146,300output: $367,920$562,027(562,027 - 87,072) / 87,072 = 5.45, the "545%". Real revenue is lower: web and app are free, V3 is cheaper, night discounts.
The 545% sum: a daily cost of $87,072 against a theoretical revenue of $562,027, of which $367,920 would come from output tokens at R1's price.

For anyone planning a service, the most useful number is per GPU. Decode gave about 1,850 output tokens per second per H800, while each user still saw 20 to 22 tokens per second. That is roughly 700 conversations being written at once on every 8-GPU node.

Others have rebuilt this design in the open. The SGLang team ran V3 on 96 H100 GPUs with prefill/decode disaggregation and large EP. It reported 52.3k input and 22.3k output tokens per second per node for 2,000-token inputs. That is up to 5 times the output throughput of plain TP16 on the same hardware. The vLLM team reported "2.2k tokens/s per H200 GPU" in multi-node wide-EP deployments in December 2025. Both are engine-team claims.

V4 and V4.1: the cache moves to disk

The V4 report says DeepSeek's inference framework "largely inherits" from V3's, with changes to KV cache management. The big change is that the cache now lives on disk between turns.

DeepInfer: the production engine, opened

In April 2025, DeepSeek wrote that its internal engine was "an early fork of vLLM" and that it would not release it. It would contribute pieces to vLLM and SGLang instead. In October 2026 it changed course and released DeepInfer under the Apache 2.0 license. The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services".

The open release is limited. It supports one model, V4.1-Flash, and one mode, offline inference. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It includes attention DP, MoE EP with EPLB, DSpark speculation, "SWA bounded replay" and batch-invariant inference. The README lists what a complete service still needs and the release leaves out: "A KV cache system", "VE/Prefill/Decode disaggregation", and load balancing across replicas. Its benchmark charts were measured on 4 SM100 GPUs for prefill and 64 for decode.

The open infrastructure libraries

DeepSeek released the building blocks of its serving and training stack during Open Source Week in February 2025, and kept updating them. Each solves one bottleneck from this part. All numbers below are DeepSeek's own.

LibraryWhat it doesReported numbers
FlashMLAAttention kernels for MLA, DSA, V4 and V4.1 cachesFeb 2025: 3,000 GB/s memory-bound, 580 TFLOPS compute-bound on H800; Apr 2025: up to 660 TFLOPS; Sep 2025 sparse kernels: 640 TFLOPS prefill, 410 decode on H800; Sep 2026 V4.1 on B200: 1,350 sparse prefill, 1,024 sparse decode; Ascend 950: 410 and 360 TFLOPS
DeepEPDispatch and combine for expert parallelism over NVLink and RDMAnormal kernels 153 to 158 GB/s inside a node, 43 to 58 GB/s across nodes; low-latency dispatch 163 to 194 us from EP8 to EP256
DeepGEMMFP8 (later FP4) matrix multiplication for dense and MoE layers"1350+ FP8 TFLOPS" at launch, "up to 1550 TFLOPS on H800" in April 2025; Mega MoE kernel (Apr 2026)
EPLBPlans which experts to copy and where to put themtwo policies: hierarchical for prefill, global for decode
3FSDistributed file system; holds the on-disk KV cache6.6 TiB/s aggregate read on 180 nodes; "40+ GiB/s peak throughput per client node for KVCache lookup"

FlashMLA's 2026 updates come with a warning for older models. The main branch dropped Hopper GPUs (H100, H800, H200) and the V3, V3.2 and V4.0 kernels. Those live on at commit ba89a34.

The DeepEP numbers explain how DeepSeek could afford 144-way expert parallelism in decode.

01002003004008163264128256combine, 8: 318combine, 16: 329combine, 32: 350combine, 64: 353combine, 128: 369combine, 256: 360dispatch, 8: 163dispatch, 16: 173dispatch, 32: 182dispatch, 64: 186dispatch, 128: 192dispatch, 256: 194combine: 360dispatch: 194expert-parallel group size (GPUs), log scalemicroseconds per call (H800, 128 tokens)
DeepEP's low-latency kernels on H800 with 128 tokens per batch: dispatch latency grows only from 163 to 194 microseconds and combine from 318 to 360 microseconds as the expert-parallel group grows from 8 to 256 GPUs.

Going from 8 to 256 GPUs multiplies the number of partners by 32, but the latency grows by less than a fifth. So a wider spread of experts costs little extra waiting, and each GPU can hold fewer experts.

V4 brought a new kernel that fuses dispatch, expert computation and combine into one.

Chinese hardware

DeepSeek trained and served V3 on NVIDIA H800s. Its move toward Chinese accelerators can be traced through primary sources, step by step.

WhenWhat DeepSeek published
Dec 2024, V3Huawei Ascend listed as a target through Huawei's MindIE engine, in BF16 at first
Aug 2025, V3.1Weights and activations use "UE8M0 FP8 scale" format, for "compatibility with microscaling data formats"
Sep 2025, V3.2-ExpKey kernels released in TileLang as well as CUDA; SGLang images for Ascend A2 and A3 on day one
Apr 2026, V4MegaMoE kernel validated on NVIDIA GPUs and Huawei Ascend NPUs
Sep 2026, V4.1FlashMLA "powering inference of the DeepSeek-V4.1 model on NVIDIA GPUs and Huawei Ascend NPUs"; Ascend versions of DeepGEMM and DeepEP
Oct 2026DeepInfer supports Huawei Ascend 950

What is not confirmed matters as much. DeepSeek's WeChat post on V3.1 said UE8M0 was for "soon-to-be-released next-generation domestic chips", as reported by Reuters; we did not check the original. Press reports said V4 was trained "entirely" on Ascend; the V4 report and model card do not say that. Whether DeepSeek's public API runs on Ascend, on NVIDIA, or on both is not public.

Outside DeepSeek, Huawei's CloudMatrix384 system serves R1 in INT8 on 384 Ascend NPUs and 192 Kunpeng CPUs. Its paper reports 6,688 prefill tokens per second per NPU. In decode it reports 1,943 tokens per second per NPU, at under 50 ms per token. The vLLM-Ascend project fits V3.2 at 8-bit weights on one Atlas 800 A3 node with 16 NPUs.

Which setup for which goal

All speeds are reported numbers from the sources above, on the hardware named. None of them is our own measurement.

GoalSetupMemoryReported speed
Try current DeepSeekAPI: deepseek-flash (V4.1-Flash-1017)nonen/a; peak price $0.006 / $0.30 / $1.20 per 1M tokens (cache-hit input / input / output), half off-peak
Try the biggest modelAPI: deepseek-v4-prononen/a; peak $0.044 / $1.32 / $3.96 per 1M
Learn reasoning on a laptopR1 distill 7B or 8B, Ollamaabout 5 GB + cachemeasure it yourself
Better laptop reasoningR1 distill 14B or 32B, 4-bit9 or 20 GB + cachemeasure it yourself
Modern DeepSeek at homeV4-Flash at 2 bits with ds4, 128 GB Mac or DGX Spark96.8 GB file39.35 tok/s (M5 Max), 18.05 (Spark), at 2K context
Modern DeepSeek, GPU + CPUV4-Flash with KTransformers, 1x RTX 509032 GB GPU + 200 GB RAM"20+ tok/s"
Full 671B V3/R1, cheapestUD-IQ1_S with llama.cpp, one 24 GB GPU + RAM140.2 GB file"at least 2-3 tokens/s" (unconfirmed)
Full 671B V3/R1, usableKTransformers, 1 GPU + Xeon (Gold 6454S)382 GB RAM (1 socket) or 1 TB (2) + 14 GB GPU12.21 tok/s decode (8 experts); 16.8 with 6 experts
Full 671B V3/R1 on a MacM3 Ultra 512 GB, 4-bit404 GB file17-18 tok/s short prompts; 6.67 to 10.89 after 5K to 13K prompts
V4.1-Flash at homeUnsloth GGUF with Engram on SSD, or ds4122 to 306 GB RAM + VRAM, plus disknone published
Serve V3/R1, one node8x H200 or 8x MI300X, FP8, SGLang or vLLM, DP attention + EP + MTP688.6 GB weightsMI300X: 34.26 tok/s at 1 request, 463.80 total at 100
Serve V3/R1 on H100two 8x H100 nodes, TP16, or 4-bit on one node688.6 GB / about 400 GBnot given
Serve V4-Pro8x H200 (FP4), 8x B200, 4x GB300, or 2 nodes of H100865 to 893 GB weightsnot published
Serve at large scalePD disaggregation + wide EP + EPLB, tens of nodesn/aDeepSeek: 73.7k in / 14.8k out tok/s per H800 node
Serve on Chinese hardwareAscend A3 with vLLM-Ascend or SGLang images; DeepInfer on Ascend 950V3.2 W8A8: 16 NPUsCloudMatrix384: 1,943 decode tok/s per NPU (Huawei)
Research notes: what we could not verify
  • The SGLang V4 cookbook shows throughput tables in an interactive widget that our tools could not read. The vLLM V4-Pro recipe publishes no tokens-per-second figures.
  • No tokens-per-second results for V4.1-Flash on consumer hardware were published by Unsloth or ds4 as of 10 November 2026.
  • Unsloth's January 2025 blog post and the TweakTown article (448 GB of RAM, under 200 W for R1 on an M3 Ultra) block automated readers. We quote Unsloth's speeds through secondary pages and leave out the TweakTown numbers.
  • The exact bytes of the V3.2 and V4 indexer caches in production engines are not documented. Our formulas assume an FP8 key with a 4-byte scale for V3.2 and an FP4 key with a 4-byte scale for V4. Both reproduce DeepSeek's published per-token totals exactly, which is strong but indirect evidence.
  • Huawei's 6,688 prefill tokens per second per NPU assumes ideal expert balancing, according to secondary summaries we could not confirm.
  • Which model answers deepseek-v4-pro API calls today is not fully clear from public pages: an announcement said V4-Pro would be routed to V4.1-Flash, then DeepSeek said it would keep serving V4-Pro.

Next

Part 8 leaves the general models and turns to the specialists. It covers Coder and Coder-V2, Math and Math-V2, the Prover models, the vision-language models, Janus, DeepSeek-OCR and the reward models. Many of them are small enough to run with the arithmetic from this part.

References