Running DeepSeek on Your Own Hardware, and How DeepSeek Serves It
How much memory each DeepSeek model needs, which GPUs, Macs and CPU servers run it at what reported speed, and how DeepSeek serves it at scale.
On one day in February 2025, DeepSeek served its chat models on an average of 226.75 server nodes, each with eight H800 GPUs. In that one day the fleet read 608 billion input tokens and wrote 168 billion output tokens. A few weeks later, a reviewer ran the same 671-billion-parameter model on one Mac Studio. It wrote about 17 to 18 tokens per second.
Both facts are true, and both come down to one question: where do the bytes go? A model is a very large file of numbers. Before it can answer, those numbers must sit in fast memory, and a few of them must be read for every word it writes. This part does that arithmetic for every major DeepSeek model, from the 1.5B distill on a laptop to the 1.6-trillion-parameter V4-Pro. Then it shows the real recipes people use, and how DeepSeek itself runs the service.
Part 6 ended with the V4 and V4.1 architectures and their tiny KV caches. Here those design choices meet real machines.
Memory arithmetic
Bytes per weight
The weights must sit somewhere the processor can reach quickly, so the first number is their size:
Here is the memory in bytes, is the number of weights, and is the number of bits per weight. The division by 8 turns bits into bytes.
Now the real model. The script part7_fits.py starts with the rule and then reads the real file sizes:
for fmt, bits in [('BF16', 16), ('FP8', 8), ('INT4 / FP4', 4)]:
log(f' 671B in {fmt:10s}: 671e9 x {bits:2d} / 8 = {671e9 * bits / 8 / GB:7,.1f} GB')
...
for m, f, gb, n, src in FILES:
bits = gb * GB * 8 / n # effective bits per weight of the real file== 1. Bytes per weight: the rule ==
memory = parameters x bits per weight / 8
671B in BF16 : 671e9 x 16 / 8 = 1,342.0 GB
671B in FP8 : 671e9 x 8 / 8 = 671.0 GB
671B in INT4 / FP4: 671e9 x 4 / 8 = 335.5 GB
== Real files: size and effective bits per weight ==
model format GB GiB bits/weight
V3 / R1 (671B) BF16 GGUF 1,342.3 1,250.1 16.00
V3 / R1 (671B) GGUF Q8_0 713.3 664.3 8.50
V3 / R1 (671B) official FP8 688.6 641.3 8.05
V3 / R1 (671B) GGUF Q4_K_M 404.4 376.6 4.82
V3 / R1 (671B) UD-Q2_K_XL 226.6 211.0 2.70
V3 / R1 (671B) UD-IQ1_S 140.2 130.6 1.67
V3.2 (671B) official FP8 689.5 642.1 8.06
V3.2 (671B) UD-IQ1_S 184.1 171.5 2.19
V2 (236B) official BF16 471.5 439.1 16.00
V2-Lite (16B) official BF16 31.4 29.2 16.00
V4-Flash (284B) official FP4+FP8 159.6 148.6 4.50
V4-Flash (284B) UD-Q2_K_XL 96.8 90.2 2.73
V4-Pro (1.6T) official FP4+FP8 864.7 805.3 4.32
V4.1-Flash (552B+196B) official FP4+FP8 510.3 475.3 5.46
R1-Distill-Llama-70B official BF16 141.1 131.4 16.00
R1-Distill-Llama-70B GGUF Q4_K_M 42.5 39.6 4.82The sizes are the sums of the real file listings on Hugging Face, checked on 10 November 2026. Three things in this table are worth a second look.
First, the official V3 file is 688.6 GB, not 671 GB. The repository holds 684.5 billion weights: the 671B main model plus a 14B module for multi-token prediction (MTP, explained below). About 3.9 billion of those weights are small tensors kept in BF16, and every 128 x 128 block of FP8 weights carries a scale. That is why the official file averages 8.05 bits, not exactly 8.
Second, a "4-bit" file is not 4.00 bits. The common GGUF format Q4_K_M averages 4.82 bits on V3, because some tensors keep more bits and every block stores its scales.
Third, the V4 family already ships its experts in FP4. The official V4-Pro file averages 4.32 bits per weight. So the usual trick of "download a 4-bit version" barely helps for V4: the official file is already about that small.
The GB versus GiB trap
In January 2025, the company Unsloth published a "1.58-bit" version of R1 and said it was 131 GB. The folder on Hugging Face adds up to 140.2 GB. Both numbers are right, because they use different units.
== 2. The GB versus GiB trap ==
R1 UD-IQ1_S ("131 GB" in the blog): 140.2 GB = 140,200,000,000 bytes = 130.6 GiB
R1 UD-Q2_K_XL ("212 GB" in the blog): 226.6 GB = 226,600,000,000 bytes = 211.0 GiB
average bits per weight of the "1.58-bit" file: 140.2e9 x 8 / 671e9 = 1.67So the "131 GB" file is 131 GiB. On a machine where every gigabyte counts, a 7% mistake is the difference between "fits" and "does not fit". The output also shows a second trap: the "1.58-bit" label describes most expert weights, not the file. The file averages 1.67 bits, because Unsloth kept the first dense layers and other sensitive tensors at higher precision.
What all those bytes are
DeepSeek-V3 is a mixture of experts (Part 2 builds it from zero). Each of its 58 MoE layers holds 256 small feed-forward networks, the routed experts. A router picks 8 of them for each token. The same script counts where the weights live, straight from the model's config.json:
== 3. DeepSeek-V3: where the 671B weights live (from config.json) ==
one expert = 3 x 7168 x 2048 = 44,040,192 parameters
routed experts: 58 layers x 256 x 44,040,192 = 653.91B (97.4%)
everything else (attention, dense layers, shared experts, routers, embedding, head) = 17.12B (2.6%)
total = 671.03B
per token: 8 of 256 routed experts in each of 58 layers = 20.43B parameters
at Q4_K_M (4.82 bits): one expert = 26.5 MB; routed experts read per token = 12.3 GB out of 394 GB; the rest = 10.3 GBThis is the most important table in the part. Of 671 billion weights, 97.4% are routed experts. A token reads only 20.4 billion of them, plus the 17.1 billion that every token uses. That gives the "37B active" in the V3 paper.
Hold on to the two bars. Memory size follows the top bar: every expert must be stored, because the next token may pick any of them. Speed follows the bottom bar: each new token reads about 37 billion weights, not 671 billion. A dense model of 671B would read all of them for every token.
The other memory bill: the KV cache
Weights are not the whole bill. While the model reads a prompt and writes an answer, it keeps notes about every token so far. These notes are the KV cache. LLM Inference Part 2 explains the cache from zero; here we only need its size.
The size is "bytes per token x number of tokens". Bytes per token depend on the attention design. DeepSeek changed that design three times, and each change shrank the cache.
V2 and V3 use Multi-head Latent Attention (MLA). Per layer, it stores one compressed vector of 512 numbers plus a 64-number position key, shared by all 128 heads. Attention Part 2 explains MLA, and Part 2 of this series shows it inside V3.
Read it left to right. is the KV cache in bytes for one token. is the size of the compressed vector (512). is the size of the shared position key (64). is the number of layers. is the bytes per stored number (2 for BF16).
Tiny example: a toy model with , , 3 layers and BF16 stores bytes per token. A prompt of 10 tokens needs 360 bytes. For real V3: bytes per token, about 69 KiB.
Every model, from its config
The script reads each model's config.json and computes bytes per token with the formula that fits its attention type. The distills are normal Qwen and Llama models with grouped-query attention. Their formula is : keys and values, layers, key-value heads, head size and 2 bytes. For V3.2, V4 and V4.1, the script uses the FP8 and FP4 cache layouts from DeepSeek's FlashMLA kernels.
def mla_bf16(c):
return (c['kv_lora_rank'] + c['qk_rope_head_dim']) * c['num_hidden_layers'] * 2
def v32(c): # FlashMLA FP8 row 656 B (latent + scales + RoPE) + FP8 indexer key 128 B + 4 B scale
return c['num_hidden_layers'] * (656 + c['index_head_dim'] + 4)
def v4(c): # V4 row 584 B per compressed entry; FP4 indexer key 64 B + 4 B scale on CSA layers
r = c['compress_ratios'][:c['num_hidden_layers']]
csa = sum(1 for x in r if x == 4)
hca = sum(1 for x in r if x == 128)
return csa * (584 + 68) / 4 + hca * 584 / 128, csa, hca== 4. KV cache per token, from each config.json ==
naive multi-head cache for V3 (for contrast): 128 heads x (128 + 64 + 128) x 61 x 2 B = 4,997,120 B per token
model B/token 32K 128K 1M how
V2-Lite 31,104 1.02G n/a n/a MLA BF16: (512 + 64) x 27 layers x 2 B
V2 69,120 2.26G 9.06G n/a MLA BF16: (512 + 64) x 60 layers x 2 B
V3 / R1 / V3.1 70,272 2.30G 9.21G n/a MLA BF16: (512 + 64) x 61 layers x 2 B
V3.2 48,068 1.58G 6.30G n/a FP8 MLA row 656 B + indexer 132 B, x 61 layers
V4-Flash 3,514 0.12G 0.46G 3.68G 21 CSA layers x 652 B / 4 + 20 HCA layers x 584 B / 128
V4-Pro 5,031 0.16G 0.66G 5.28G 30 CSA layers x 652 B / 4 + 31 HCA layers x 584 B / 128
V4.1-Flash 890 0.03G 0.12G 0.93G published: 890 B per token (V4.1 report, abstract)
Distill-Qwen-1.5B 28,672 0.94G 3.76G n/a GQA BF16: 2 x 28 layers x 2 KV heads x 128 x 2 B
Distill-Qwen-7B 57,344 1.88G 7.52G n/a GQA BF16: 2 x 28 layers x 4 KV heads x 128 x 2 B
Distill-Llama-8B 131,072 4.29G 17.18G n/a GQA BF16: 2 x 32 layers x 8 KV heads x 128 x 2 B
Distill-Qwen-14B 196,608 6.44G 25.77G n/a GQA BF16: 2 x 48 layers x 8 KV heads x 128 x 2 B
Distill-Qwen-32B 262,144 8.59G 34.36G n/a GQA BF16: 2 x 64 layers x 8 KV heads x 128 x 2 B
Distill-Llama-70B 327,680 10.74G 42.95G n/a GQA BF16: 2 x 80 layers x 8 KV heads x 128 x 2 B
checks against published numbers:
V3 70,272 B = ISCA 2025 Table 1 "70.272 KB"
V3.2 48,068 B, V4-Flash 3,514 B: V4.1 report Figure 1(b) gives 48,068 and 3,514
V4-Pro / V3.2 = 10.5% (V4 report: "10% of KV cache"); V4-Flash / V3.2 = 7.3% (report: "7%")
V3 MLA versus naive multi-head: 71x smaller; Distill-Llama-8B needs 1.87x more per token than the 671B V3"n/a" means the model does not support that context length. The checks at the bottom matter. Our numbers for V3, V3.2 and V4-Flash match the numbers DeepSeek published, byte for byte. V4-Pro has no published per-token figure, so 5,031 bytes is our estimate. It gives 10.5% of V3.2, and the V4 report says "10%". The estimate leaves out the small sliding-window cache, which has a fixed size of 128 tokens per layer and does not grow with the prompt.
Two lessons sit in this table. The small 8B Llama distill needs 1.87 times more cache per token than the 671B V3. And V4.1-Flash keeps a one-million-token conversation in 0.93 GB, less than V3 needs for 16K tokens.
What fits where
Now we can answer the question people actually ask: will it run on my machine? The rule we use is an estimate, and it is strict on purpose. A setup fits if the weights plus the KV cache of one 32K-token conversation take at most 90% of the memory. The other 10% goes to activations, engine buffers and the operating system.
for label, w, kvm in ROWS:
need = w + MODELS[kvm][0] * CTX / GB # weights + KV cache of one 32K-token sequence
row = [cap * USABLE >= need for _, cap in MACHINES]== 5. What fits where: weights + KV for one 32,768-token sequence <= 90% of memory ==
(capacities taken as decimal GB; RAM sold as "512 GB" is really 512 GiB, so this errs on the safe side)
setup need GB 16GB lap 24GB GPU 32GB GPU 128GB box 512GB Mac 8x H100 8x H200 2x8 H100 8x MI300X
V3/R1 FP8 official 690.9 - - - - - - yes yes yes
V3/R1 Q4_K_M 406.7 - - - - yes yes yes yes yes
R1 UD-Q2_K_XL 228.9 - - - - yes yes yes yes yes
R1 UD-IQ1_S 142.5 - - - - yes yes yes yes yes
V3.2 FP8 official 691.1 - - - - - - yes yes yes
V2 BF16 official 473.8 - - - - - yes yes yes yes
V2 4-bit (Ollama) 135.3 - - - - yes yes yes yes yes
V2-Lite 4-bit (Ollama) 9.9 yes yes yes yes yes yes yes yes yes
V4-Pro-0813 official 892.9 - - - - - - yes yes yes
V4-Flash official 159.7 - - - - yes yes yes yes yes
V4-Flash UD-Q2_K_XL 96.9 - - - yes yes yes yes yes yes
V4.1-Flash-1017 official 510.3 - - - - - yes yes yes yes
V4.1 UD-IQ1_S, Engram on SSD 122.4 - - - - yes yes yes yes yes
Distill-Llama-70B Q4_K_M 53.2 - - - yes yes yes yes yes yes
Distill-Qwen-32B Q4_K_M 28.5 - - yes yes yes yes yes yes yes
Distill-Qwen-14B Q4_K_M 15.4 - yes yes yes yes yes yes yes yes
Distill-Qwen-7B Q4_K_M 6.6 yes yes yes yes yes yes yes yes yesThe grid answers "does it fit", not "is it fast" or "does the software support it". Three notes keep it honest.
The 8x H100 column is the famous gap. Eight H100 GPUs have 8 x 80 = 640 GB, and the native V3 file is 688.6 GB. That is why every guide says V3 in FP8 needs either 8 H200s (1,128 GB) or two H100 nodes. The column also says V4.1-Flash fits 8x H100 by memory. In practice, DeepSeek's own V4.1 engine targets newer NVIDIA SM100 GPUs and Huawei Ascend 950 chips, so memory is not the only gate.
The 128 GB column is where V4-Flash at 2 bits appears, and where V4.1-Flash with its Engram tables on SSD comes close. V4.1-Flash stores a large "Engram" memory of n-gram tables (Part 6). It reads only a few rows per token, so the tables can stay on an SSD. Unsloth's table splits each V4.1 file into "Engram" and "Rest".
For UD-IQ1_S, 110.6 GB is Engram and 122.4 GB is the rest, so the rest is what must sit in RAM or GPU memory. By our 90% rule, 122.4 GB is just too much for a 128 GB box. Unsloth itself lists 128 to 145 GB of RAM plus VRAM for its 1-bit V4.1 files.
The distills are a different kind of model. They are Qwen 2.5 and Llama 3 models, fine-tuned on R1's answers. They have no MLA and no experts. They fit laptops because they are small, not because of DeepSeek's architecture.
Data-centre recipes: GPUs, flags and why
Splitting one model over many GPUs
When a model does not fit one GPU, engines split it. There are three basic ways, and real engines mix them. LLM Inference Part 7 covers them in depth.
DeepSeek's models suit a mix. The attention part (MLA) is small, so engines often copy it on every GPU with DP. The experts are 97% of the weights, so engines spread them with EP. SGLang calls this "DP attention".
Speculative decoding with MTP and DSpark
One more idea appears in every recipe. During training, V3 also learned to predict the token after the next one. That extra head is the MTP module. At serving time, it can propose a second token for free, and the main model checks it. LLM Inference Part 4 explains why checking is cheap.
V4 replaced the single MTP head with DSpark, a drafter that proposes several tokens at once (Part 6). The V4-Pro-0813 checkpoint ships with its DSpark module attached, which is one reason it is 892.7 GB, against 864.7 GB for the April preview.
The recipes, model by model
All flags below are copied from the engine docs or DeepSeek's model cards. Numbers are vendor claims unless the text says otherwise.
V3, R1, V3.1 on one node of 8x H200 or 8x MI300X. The native FP8 file fits, with about 440 GB left for KV cache and buffers. SGLang's launch line is short:
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8
# add --reasoning-parser deepseek-v3 for R1-style thinking output
# add --attn-dp-size 8 for high-throughput, large-batch serving (DP attention)
# add --speculative-algorithm EAGLE for MTP speculation
# (defaults: --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4)The SGLang page warns not to add --quantization fp8, because the checkpoint is already FP8. It reports MTP "up to 1.8x decoding speed at batch size 1 and 1.5x at batch size 32" on 8x H200. On 8x MI300X with SGLang 0.5.7 and 1K-token inputs and outputs, it reports 34.26 output tokens per second at one request with EAGLE. At 100 concurrent requests with --ep 8 --attn-dp-size 8, the total is 463.80 output tokens per second. The gain for one user is small, the gain for the whole server is large, exactly as the MTP box predicted.
vLLM's recipe uses the same hardware:
vllm serve deepseek-ai/DeepSeek-R1-0528 --trust-remote-code --tensor-parallel-size 8 --enable-expert-parallel
# or: --data-parallel-size 8 --enable-expert-parallel
# AMD ROCm: VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1V3 on H100, H800, H20 or A100. The SGLang hardware table lists FP8 V3 on "2x8 H100/H800/H20", that is, two nodes with tensor parallelism across both. If you upcast to BF16 (1.34 TB), you need 2x8 H200, 4x8 H100 or 4x8 A100. There is also an INT8 path on 16 A100s, 32 L40S cards, a Xeon 6980P CPU server, or 4x Huawei Ascend A3. With 4-bit weights (W4A8, AWQ, NVFP4 and similar), one node of 8x H20 or 8x H100 is enough.
V3.2 and V3.2-Exp. DeepSeek's own README gives SGLang images for H200, AMD MI350 and Huawei Ascend, launched with --tp 8 --dp 8 --enable-dp-attention. The vLLM recipe recommends -dp 8 --enable-expert-parallel on 8x H200, 8x H20 or 8x B200, and says the sparse-attention kernels are "mainly optimized for TP=1 (DP=8, EP=8)". It uses an FP8 KV cache by default. Some H20 users found VLLM_USE_DEEP_GEMM=0 faster. Part 5 explains the sparse attention these kernels run.
V4-Pro and V4-Flash. DeepSeek's V4-Pro-0813 model card gives two launch lines. Each turns on DSpark, an FP8 KV cache and expert parallelism:
# vLLM, one node of 4x GB300
vllm serve deepseek-ai/DeepSeek-V4-Pro-0813 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
--data-parallel-size 4 --enable-expert-parallel --moe-backend deep_gemm_mega_moe \
--attention-config '{"use_fp4_indexer_cache": true}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'
# SGLang
sglang serve --model-path deepseek-ai/DeepSeek-V4-Pro-0813 --tp 4 --moe-runner-backend flashinfer_mxfp4 \
--speculative-algorithm DSPARK --mem-fraction-static 0.90 --chunked-prefill-size 4096 --swa-full-tokens-ratio 0.1The vLLM recipe defaults V4-Pro to 8x H200 and caps the context at 800,000 tokens to keep room for the KV cache.
SGLang's cookbook puts V4-Flash on 4 B200, B300, GB200, GB300 or H200 GPUs. It also fits 2 RTX PRO 6000 cards (96 GB each) or 8 H100s. V4-Pro needs 8 B200s or B300s, 4 GB300s, or one 8x H200 node with FP4 weights. With FP8 weights on H200, or on H100, it needs two nodes. The cookbook says V4-Pro "does not fit on 8x 96 GB RTX PRO 6000", and that "a single 128 GB GB10 cannot hold any of these checkpoints". Neither recipe page publishes tokens-per-second results for V4 that we could read.
V4.1-Flash. DeepSeek's announcement says it "will work closely with the open-source community on V4.1-Flash inference support". The reference code splits the weights for 8-way tensor parallelism. The newest option is DeepSeek's own engine, DeepInfer, covered below.
Why MoE models suit CPU and GPU together
The speed rule for writing tokens
Writing one token means reading every active weight once from memory. On a single user's machine, the processor spends most of its time waiting for those bytes, not doing maths. LLM Inference Part 1 shows why.
This gives a simple ceiling for one user:
The bytes read per token are the active weights times the bytes per weight. It is a ceiling, not a forecast: real systems also read the KV cache, run attention, and never reach full bandwidth.
Tiny example: memory that delivers 100 GB/s, and a model that reads 10 GB per token, can write at most 10 tokens per second. Now the real numbers from part7_serving.py:
== 1. Decode ceiling: tokens/s <= bandwidth / bytes read per token ==
M3 Ultra, "over 800GB/s" V3/R1 Q4_K_M, 37B active 22.3 GB/token -> at most 35.9 tok/s
M3 Ultra, "over 800GB/s" R1-Distill-Llama-70B Q4_K_M, 70.55B dense 42.5 GB/token -> at most 18.8 tok/s
2 sockets x 8 ch DDR5-4800 (2 x 8 x 38.4 GB/s) V3/R1 Q4_K_M, 37B active 22.3 GB/token -> at most 27.6 tok/s
2 sockets x 8 ch DDR5-4800 (2 x 8 x 38.4 GB/s) R1-Distill-Llama-70B Q4_K_M, 70.55B dense 42.5 GB/token -> at most 14.5 tok/s
measured (third party): R1 4-bit on M3 Ultra 512 GB about 17-18 tok/s (short prompts);
V3-0324 Q4_K_M: 10.89 tok/s after a 4.8K prompt, 6.67 tok/s after a 12.7K prompt
share of the ceiling reached at 17.5 tok/s: 49%Look at the first two lines. The 671B MoE model has a higher speed ceiling than the 70B dense distill, because it reads fewer bytes per token. That is the whole reason a 671B model is usable on one desk at all.
Experts can live in different places
The bandwidth rule has a second consequence. Only 8 of 256 experts are read per token. The other 248 can sit in slower, cheaper memory, as long as the chosen ones arrive in time. This turns memory into tiers, like a kitchen. The things you use every minute sit on the counter, the weekly things in the cupboard, the rare things in the basement.
The always-used part of V3 belongs on the counter. From the script: at Q4_K_M, everything except the routed experts is about 10.3 GB. Add a KV cache and buffers, and you get close to the "at least 14GB VRAM" that KTransformers asks for. The routed experts, 394 GB at Q4_K_M, go to RAM. For one token, the CPU reads about 12.3 GB of them.
The SSD tier is the newest. V4.1-Flash's Engram tables are made for it: the model reads only a few rows per token, so llama.cpp builds can map them from disk. Unsloth's guide uses --load-mode mmap --lazy-mode on for this. The same idea can be pushed onto experts, with a cache of recently used experts in RAM and the rest on an SSD. The catch is speed. An SSD delivers a few GB per second, far below RAM, so every miss costs a long wait.
Running on consumer machines
llama.cpp and GGUF files
llama.cpp has run DeepSeek V2 and V3 for a long time. Proper MLA support, with the small cache, arrived in pull request #12801, merged on 15 April 2025. V4 support landed in pull request #24162 on 29 June 2026. For V4.1-Flash, Unsloth's guide says to build from pull request #30817, which was closed without merging. As of 10 November 2026 we could not confirm V4.1 support in the main llama.cpp branch.
For the 1.58-bit R1 file, Unsloth reported (January 2025) "around 140 tokens per second for throughput and 14 tokens/s for single user inference" on two H100 80 GB GPUs. Unsloth staff wrote that one 24 GB RTX 4090 with the rest in RAM gives "at least 2-3 tokens/s". We read that second claim through a mirror, so treat it as unconfirmed.
KTransformers: experts on the CPU, the rest on the GPU
Its tutorial for V3 and R1 at Q4_K_M uses two Intel Xeon Gold 6454S CPUs with 32 cores each. The memory is DDR5-4800 with 8 channels per socket, and the GPU is one RTX 4090D with 24 GB. With one socket it needs 382 GB of RAM and at least 14 GB of GPU memory. With two sockets it needs 1 TB, because it copies the weights to both sockets' memory. The reported numbers, all from the KTransformers team for a 500-token prompt:
| Setup (V3/R1 Q4_K_M) | Prefill tok/s | Decode tok/s |
|---|---|---|
| llama.cpp, same 2 x 32 cores | 10.31 | 4.51 |
| KTransformers V0.2, 2 sockets, 8 experts | 82.94 | 12.21 |
| KTransformers V0.2, 2 sockets, 6 experts | 97.32 | 13.69 |
| KTransformers V0.3 preview (AMX), 8 / 6 experts | 255.26 / 286.55 | not given |
| KTransformers V0.2.1, 6 experts, short / 8K prompt | not given | 16.8 / 13.9 |
KTransformers also runs V4-Flash on one RTX 5090 (32 GB) with an x86 CPU and at least 200 GB of RAM, and reports "20+ tok/s" decode. On a single Huawei Ascend die (61.3 GB) with 229 GB of host RAM, it reports 18.32 to 20.48 tokens per second. It notes that "Decode is host-memory-bandwidth bound". For R1-0528 with SGLang on a server with 8 L20 GPUs and a Xeon, it reports 87.58 output tokens per second at 8 parallel requests.
Apple Silicon
A Mac with Apple Silicon has one pool of unified memory shared by the CPU and GPU. A Mac Studio with an M3 Ultra can have up to 512 GB of it, at "over 800GB/s". Apple's own launch text says it can run "LLMs with over 600 billion parameters entirely in memory".
The reported speeds are good for short prompts and much worse for long ones. A reviewer measured R1 at 4 bits at "approximately 17-18 tokens per second". An independent llama.cpp-family test of V3-0324 Q4_K_M measured these numbers:
| Prompt length | Prompt reading (prefill) | Writing (decode) |
|---|---|---|
| 4.8K tokens | 64.02 tok/s, about 75 s | 10.89 tok/s |
| 12.7K tokens | 58.82 tok/s, about 216 s | 6.67 tok/s |
The waiting times are our arithmetic: 4,800 / 64.02 and 12,700 / 58.82. They show the weak spot. A Mac has the bandwidth for decode but much less raw compute than a data-centre GPU. So a long document takes minutes before the first word appears. Short demos hide this.
For V4-Flash, the project DwarfStar (ds4, by antirez) publishes its own Metal and CUDA measurements of a 2-bit file:
| Machine | Context | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| M5 Max, 128 GB | 2,048 | 790.18 | 39.35 |
| M5 Max, 128 GB | 65,536 | 398.50 | 27.64 |
| DGX Spark, 128 GB | 2,048 | 825.76 | 18.05 |
| DGX Spark, 128 GB | 65,536 | 822.98 | 13.84 |
V4-Flash has 13B active weights against V3's 37B, so a 128 GB laptop decodes it faster than a 512 GB Mac Studio decodes V3. For V4.1-Flash, ds4 says the 2-bit file runs on one 128 GB Mac or Spark with SSD streaming, and "Q4 needs SSD streaming or a 512 GB Mac". It publishes no speed for V4.1.
Ollama for the distills
The R1 distills are the practical way to try DeepSeek-style reasoning on a laptop. Ollama's default tags are 4-bit files. The 1.5b tag is 1.1 GB, 7b is 4.7 GB and 8b is 5.2 GB. The 14b tag is 9.0 GB, 32b is 20 GB and 70b is 43 GB. The full deepseek-r1:671b tag is 404 GB.
The fits table gives a rough rule. A 16 GB laptop runs the 7B and 8B distills well, and a 24 GB GPU or a 32 GB Mac fits the 14B. The 32B at 4 bits needs about 28 GB with a 32K cache. The 70B needs 64 GB or more. We found no single trustworthy speed table for laptops, so measure your own.
How DeepSeek serves its own models
One model, two kinds of machine
A chat service does two different jobs. Reading a long prompt is one big parallel computation. Writing the answer is a long series of small steps, one token at a time, for thousands of users. On the same GPU, the two jobs get in each other's way: a big prefill pauses everybody's decode.
The February 2025 overview
On the last day of its "Open Source Week", DeepSeek published an overview of its V3/R1 serving system in the open-infra-index repository. It is a GitHub page, so we quote its text instead of a screenshot:
== 3. Where the 256 routed experts go ==
V3 paper, prefill unit (4 nodes) : 256 + 32 redundant = 288 = 32 GPUs x 9 routed experts
Feb 2025, prefill unit (4 nodes) : 256 + 32 redundant = 288 = 32 GPUs x 9 routed experts
Feb 2025, decode unit (18 nodes) : 256 + 32 redundant = 288 = 144 GPUs x 2 routed experts
V3 paper, decode unit (40 nodes) : 320 GPUs, one expert each: 256 routed + 64 for redundant and shared experts = 320Precision follows training: matrix multiplications and dispatch in FP8, core MLA and combine in BF16. And the fleet breathes with demand. DeepSeek runs inference on all nodes during the day and moves nodes to research and training at night.
How fast can expert parallelism be?
The ISCA 2025 paper does a back-of-the-envelope limit for this design. Each GPU holds one expert and handles about 32 tokens per step. Each token is sent to 9 experts (8 routed and 1 shared) in FP8 (1 byte per number) and returned in BF16 (2 bytes). The hidden size is about 7,000 numbers.
== 2. Expert-parallel communication limit (ISCA 2025, section 2.3.2) ==
CX7 400Gbps InfiniBand, 50 GB/s : one all-to-all pair 120.96 us, x2 per layer x 61 layers = 14.76 ms -> 67.8 tok/s
GB200 NVL72, 900 GB/s : one all-to-all pair 6.72 us, x2 per layer x 61 layers = 0.82 ms -> 1219.8 tok/sThe paper rounds 67.8 down to 67. In production, DeepSeek's users got an average of 20 to 22 tokens per second, about a third of this ideal.
One day of numbers, and the 545%
The overview ends with 24 hours of statistics, from 12:00 on 27 February 2025 to 12:00 the next day, Beijing time. V3 and R1 together peaked at 278 nodes and averaged 226.75 nodes of 8 H800s. 56.3% of input tokens hit DeepSeek's on-disk KV cache, so their prefill was skipped. Each node delivered about 73.7k input tokens per second in prefill (counting cache hits) or about 14.8k output tokens per second in decode.
Then it does a famous sum. If every token had been billed at R1's API prices, the day's revenue would have been $562,027 against a cost of $87,072. The overview calls this a "cost profit margin of 545%".
is the theoretical revenue for the day and is the cost. A tiny example: a lemonade stand that spends $10 and takes $30 has a margin of . The real inputs, recomputed:
== 4. Feb 27-28 2025, 24 hours of V3 + R1 on H800 ==
cost = 226.75 nodes x 8 GPUs x 24 h x $2 = $87,072
revenue = 342B hit x $0.14/M + 266B miss x $0.55/M + 168B out x $2.19/M
= $47,880 + $146,300 + $367,920 = $562,100 (reported: $562,027)
margin = (562,027 - 87,072) / 87,072 = 545%
cache-hit share of input: 56.25%
output share of the theoretical revenue: 65%
decode: 14.8k output tok/s per node = 1,850 per H800; at 20-22 tok/s per user, 673 to 740 streams per nodeOur revenue is $73 higher than DeepSeek's because the token counts in the overview are rounded to billions. The 545% follows exactly from the published inputs.
For anyone planning a service, the most useful number is per GPU. Decode gave about 1,850 output tokens per second per H800, while each user still saw 20 to 22 tokens per second. That is roughly 700 conversations being written at once on every 8-GPU node.
Others have rebuilt this design in the open. The SGLang team ran V3 on 96 H100 GPUs with prefill/decode disaggregation and large EP. It reported 52.3k input and 22.3k output tokens per second per node for 2,000-token inputs. That is up to 5 times the output throughput of plain TP16 on the same hardware. The vLLM team reported "2.2k tokens/s per H200 GPU" in multi-node wide-EP deployments in December 2025. Both are engine-team claims.
V4 and V4.1: the cache moves to disk
The V4 report says DeepSeek's inference framework "largely inherits" from V3's, with changes to KV cache management. The big change is that the cache now lives on disk between turns.
DeepInfer: the production engine, opened
In April 2025, DeepSeek wrote that its internal engine was "an early fork of vLLM" and that it would not release it. It would contribute pieces to vLLM and SGLang instead. In October 2026 it changed course and released DeepInfer under the Apache 2.0 license. The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services".
The open release is limited. It supports one model, V4.1-Flash, and one mode, offline inference. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It includes attention DP, MoE EP with EPLB, DSpark speculation, "SWA bounded replay" and batch-invariant inference. The README lists what a complete service still needs and the release leaves out: "A KV cache system", "VE/Prefill/Decode disaggregation", and load balancing across replicas. Its benchmark charts were measured on 4 SM100 GPUs for prefill and 64 for decode.
The open infrastructure libraries
DeepSeek released the building blocks of its serving and training stack during Open Source Week in February 2025, and kept updating them. Each solves one bottleneck from this part. All numbers below are DeepSeek's own.
| Library | What it does | Reported numbers |
|---|---|---|
| FlashMLA | Attention kernels for MLA, DSA, V4 and V4.1 caches | Feb 2025: 3,000 GB/s memory-bound, 580 TFLOPS compute-bound on H800; Apr 2025: up to 660 TFLOPS; Sep 2025 sparse kernels: 640 TFLOPS prefill, 410 decode on H800; Sep 2026 V4.1 on B200: 1,350 sparse prefill, 1,024 sparse decode; Ascend 950: 410 and 360 TFLOPS |
| DeepEP | Dispatch and combine for expert parallelism over NVLink and RDMA | normal kernels 153 to 158 GB/s inside a node, 43 to 58 GB/s across nodes; low-latency dispatch 163 to 194 us from EP8 to EP256 |
| DeepGEMM | FP8 (later FP4) matrix multiplication for dense and MoE layers | "1350+ FP8 TFLOPS" at launch, "up to 1550 TFLOPS on H800" in April 2025; Mega MoE kernel (Apr 2026) |
| EPLB | Plans which experts to copy and where to put them | two policies: hierarchical for prefill, global for decode |
| 3FS | Distributed file system; holds the on-disk KV cache | 6.6 TiB/s aggregate read on 180 nodes; "40+ GiB/s peak throughput per client node for KVCache lookup" |
FlashMLA's 2026 updates come with a warning for older models. The main branch dropped Hopper GPUs (H100, H800, H200) and the V3, V3.2 and V4.0 kernels. Those live on at commit ba89a34.
The DeepEP numbers explain how DeepSeek could afford 144-way expert parallelism in decode.
Going from 8 to 256 GPUs multiplies the number of partners by 32, but the latency grows by less than a fifth. So a wider spread of experts costs little extra waiting, and each GPU can hold fewer experts.
V4 brought a new kernel that fuses dispatch, expert computation and combine into one.
Chinese hardware
DeepSeek trained and served V3 on NVIDIA H800s. Its move toward Chinese accelerators can be traced through primary sources, step by step.
| When | What DeepSeek published |
|---|---|
| Dec 2024, V3 | Huawei Ascend listed as a target through Huawei's MindIE engine, in BF16 at first |
| Aug 2025, V3.1 | Weights and activations use "UE8M0 FP8 scale" format, for "compatibility with microscaling data formats" |
| Sep 2025, V3.2-Exp | Key kernels released in TileLang as well as CUDA; SGLang images for Ascend A2 and A3 on day one |
| Apr 2026, V4 | MegaMoE kernel validated on NVIDIA GPUs and Huawei Ascend NPUs |
| Sep 2026, V4.1 | FlashMLA "powering inference of the DeepSeek-V4.1 model on NVIDIA GPUs and Huawei Ascend NPUs"; Ascend versions of DeepGEMM and DeepEP |
| Oct 2026 | DeepInfer supports Huawei Ascend 950 |
What is not confirmed matters as much. DeepSeek's WeChat post on V3.1 said UE8M0 was for "soon-to-be-released next-generation domestic chips", as reported by Reuters; we did not check the original. Press reports said V4 was trained "entirely" on Ascend; the V4 report and model card do not say that. Whether DeepSeek's public API runs on Ascend, on NVIDIA, or on both is not public.
Outside DeepSeek, Huawei's CloudMatrix384 system serves R1 in INT8 on 384 Ascend NPUs and 192 Kunpeng CPUs. Its paper reports 6,688 prefill tokens per second per NPU. In decode it reports 1,943 tokens per second per NPU, at under 50 ms per token. The vLLM-Ascend project fits V3.2 at 8-bit weights on one Atlas 800 A3 node with 16 NPUs.
Which setup for which goal
All speeds are reported numbers from the sources above, on the hardware named. None of them is our own measurement.
| Goal | Setup | Memory | Reported speed |
|---|---|---|---|
| Try current DeepSeek | API: deepseek-flash (V4.1-Flash-1017) | none | n/a; peak price $0.006 / $0.30 / $1.20 per 1M tokens (cache-hit input / input / output), half off-peak |
| Try the biggest model | API: deepseek-v4-pro | none | n/a; peak $0.044 / $1.32 / $3.96 per 1M |
| Learn reasoning on a laptop | R1 distill 7B or 8B, Ollama | about 5 GB + cache | measure it yourself |
| Better laptop reasoning | R1 distill 14B or 32B, 4-bit | 9 or 20 GB + cache | measure it yourself |
| Modern DeepSeek at home | V4-Flash at 2 bits with ds4, 128 GB Mac or DGX Spark | 96.8 GB file | 39.35 tok/s (M5 Max), 18.05 (Spark), at 2K context |
| Modern DeepSeek, GPU + CPU | V4-Flash with KTransformers, 1x RTX 5090 | 32 GB GPU + 200 GB RAM | "20+ tok/s" |
| Full 671B V3/R1, cheapest | UD-IQ1_S with llama.cpp, one 24 GB GPU + RAM | 140.2 GB file | "at least 2-3 tokens/s" (unconfirmed) |
| Full 671B V3/R1, usable | KTransformers, 1 GPU + Xeon (Gold 6454S) | 382 GB RAM (1 socket) or 1 TB (2) + 14 GB GPU | 12.21 tok/s decode (8 experts); 16.8 with 6 experts |
| Full 671B V3/R1 on a Mac | M3 Ultra 512 GB, 4-bit | 404 GB file | 17-18 tok/s short prompts; 6.67 to 10.89 after 5K to 13K prompts |
| V4.1-Flash at home | Unsloth GGUF with Engram on SSD, or ds4 | 122 to 306 GB RAM + VRAM, plus disk | none published |
| Serve V3/R1, one node | 8x H200 or 8x MI300X, FP8, SGLang or vLLM, DP attention + EP + MTP | 688.6 GB weights | MI300X: 34.26 tok/s at 1 request, 463.80 total at 100 |
| Serve V3/R1 on H100 | two 8x H100 nodes, TP16, or 4-bit on one node | 688.6 GB / about 400 GB | not given |
| Serve V4-Pro | 8x H200 (FP4), 8x B200, 4x GB300, or 2 nodes of H100 | 865 to 893 GB weights | not published |
| Serve at large scale | PD disaggregation + wide EP + EPLB, tens of nodes | n/a | DeepSeek: 73.7k in / 14.8k out tok/s per H800 node |
| Serve on Chinese hardware | Ascend A3 with vLLM-Ascend or SGLang images; DeepInfer on Ascend 950 | V3.2 W8A8: 16 NPUs | CloudMatrix384: 1,943 decode tok/s per NPU (Huawei) |
Research notes: what we could not verify
- The SGLang V4 cookbook shows throughput tables in an interactive widget that our tools could not read. The vLLM V4-Pro recipe publishes no tokens-per-second figures.
- No tokens-per-second results for V4.1-Flash on consumer hardware were published by Unsloth or ds4 as of 10 November 2026.
- Unsloth's January 2025 blog post and the TweakTown article (448 GB of RAM, under 200 W for R1 on an M3 Ultra) block automated readers. We quote Unsloth's speeds through secondary pages and leave out the TweakTown numbers.
- The exact bytes of the V3.2 and V4 indexer caches in production engines are not documented. Our formulas assume an FP8 key with a 4-byte scale for V3.2 and an FP4 key with a 4-byte scale for V4. Both reproduce DeepSeek's published per-token totals exactly, which is strong but indirect evidence.
- Huawei's 6,688 prefill tokens per second per NPU assumes ideal expert balancing, according to secondary summaries we could not confirm.
- Which model answers
deepseek-v4-proAPI calls today is not fully clear from public pages: an announcement said V4-Pro would be routed to V4.1-Flash, then DeepSeek said it would keep serving V4-Pro.
Next
Part 8 leaves the general models and turns to the specialists. It covers Coder and Coder-V2, Math and Math-V2, the Prover models, the vision-language models, Janus, DeepSeek-OCR and the reward models. Many of them are small enough to run with the arithmetic from this part.
References
- DeepSeek-AI (2024), DeepSeek-V3 Technical Report: arXiv:2412.19437
- Zhao et al. (2025), Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures: arXiv:2505.09343
- DeepSeek-AI (2026), DeepSeek-V4: arXiv:2606.19348; DeepSeek-V4.1-Flash: arXiv:2609.19969
- Huawei (2025), Serving Large Language Models on Huawei CloudMatrix384: arXiv:2506.12708
- DeepSeek, Day 6 inference system overview: open-infra-index; The Path to Open-Sourcing the DeepSeek Inference Engine; DeepInfer
- DeepSeek libraries: FlashMLA, DeepEP, DeepGEMM, EPLB, 3FS
- Model cards and file listings: DeepSeek-V3, DeepSeek-V4-Pro-0813, DeepSeek-V4.1-Flash, Unsloth R1 GGUF, Unsloth V4.1-Flash guide
- Engine recipes: SGLang DeepSeek V3, SGLang DeepSeek V4, vLLM DeepSeek V3, vLLM V3.2-Exp, vLLM V4-Pro, vLLM-Ascend V3.2
- Large-scale reproductions: SGLang, 96 H100s; vLLM wide-EP
- Consumer engines: llama.cpp MLA PR #12801, V4 PR #24162, KTransformers V3/R1 tutorial, KTransformers V4-Flash, ds4 performance, Ollama deepseek-r1 tags
- Apple Silicon reports: Apple Mac Studio announcement, MacRumors on the R1 review, M3 Ultra prompt and write speeds
- API prices: DeepSeek pricing