How DeepSeek-V3 Was Trained on 2,048 GPUs

How DeepSeek-V3 was trained: 14.8T tokens on 2,048 GPUs, a schedule that hides network time, 8-bit numbers, and what the $5.576M covers.

In 2024, a cluster of 2,048 GPUs read 14.8 trillion tokens of text, one batch at a time, for less than two months. At the end of it stood DeepSeek-V3, the base of R1, V3.1 and V3.2. The report gives one more number, and it travelled further than the model. The run took 2.788 million GPU hours, or $5.576M at $2 per hour.

Part 2 took the machine apart: 671B parameters, 37B of them active for each token, experts, MLA and multi-token prediction. This part is about how that machine was trained. We follow the run from the data to the bill. On the way we meet the hardware and the plan that splits one model over 2,048 GPUs. Then come a schedule that hides network time, and 8-bit numbers that do not break.

We also run two small experiments on a laptop CPU. One shows why DeepSeek scales 8-bit numbers in groups of 128. The other shows why it moves its running sums to 32-bit every 128 terms.

DeepSeek-V3, from random weights to a chat modelPre-training14.8T tokens4K-token sequences2,664K GPU hoursContext, phase 162.9B tokens32K-token sequences1,000 stepsContext, phase 262.9B tokens128K-token sequences1,000 stepsPost-trainingSFT + RLPart 45K GPU hoursboth phases together: 119K GPU hoursShare of the 2,788K GPU hourspre-training 95.6%context 4.3%post 0.2%pre-training: 2,664K (95.6%)context 119K (4.3%), post 5K (0.2%)GPU hours from V3 report Table 1 (p.5). Token counts of the two phases: our arithmetic from p.23.
The whole training of DeepSeek-V3. Pre-training on 4K-token sequences takes 95.6% of the GPU hours. Two short phases then stretch the context to 32K and 128K tokens, and post-training (Part 4) takes only 5K GPU hours.

Pre-training is next-token prediction with a cross-entropy loss; the training overview on this site walks through the loop. What matters here is the scale. V3 took about 243,000 steps (our estimate, below), and each step read up to 63 million tokens.

The data: 14.8 trillion tokens

DeepSeek says little about its data, but what it says is specific. The report changes three things compared with V2: more maths and code, more languages, and less repetition.

Fill-in-the-middle: one document, cut in three and reordered (PSM)Original filedef area(r):return 3.14159 * r * r# end of fileprefixmiddlesuffixWhat the model reads in training (10% of documents)<|fim_begin|>def area(r):<|fim_hole|># end of file<|fim_end|>return 3.14...<|eos|>The middle comes last, so plain next-token prediction learns to write it from both sides.Format and rate from V3 report §4.1 (p.22); our own example file.
Fill-in-the-middle in the PSM order of the V3 report. The file is cut in three, and the middle moves to the end, so ordinary next-token training learns to complete a gap. V3 does this to 10% of documents.

The tokenizer paragraph hides a small fix. Some new tokens join punctuation with a line break. That compresses text, but a prompt that ends without a line break then looks unusual to the model. So during training, DeepSeek randomly splits some joined tokens back into two, and the model learns both forms.

Research notes: what the R1 paper later added about the data

The Nature version of the R1 paper (arXiv 2501.12948 v2, Supplementary A.1, page 13) adds a disclosure about the V3 base: "we exclusively use plain web pages and e-books, without incorporating any synthetic data." It then admits that "some web pages contain a significant number of OpenAI-model-generated answers", so the base model may learn from other models indirectly. The same document gives the knowledge cutoff of V3-Base as July 2024 and describes 10-gram decontamination that removed about six million potential maths texts (page 40). None of this is in the V3 report itself.

The schedule: learning rate and batch size

A training run does not use one learning rate from start to end. V3's schedule has four parts, and the report gives each one in tokens.

The cosine part has a standard formula. The report names the curve but does not print it, so this is the usual form:

η(t)=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡πtTd)\eta(t) = \eta_{\min} + \frac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\left(1 + \cos\frac{\pi t}{T_{d}}\right)

Here η(t)\eta(t) is the learning rate after tt tokens of the decay phase. ηmax⁡=2.2×10−4\eta_{\max} = 2.2 \times 10^{-4} is where it starts and ηmin⁡=2.2×10−5\eta_{\min} = 2.2 \times 10^{-5} is where it ends. Td=4.3TT_d = 4.3\text{T} is the length of the decay. At t=0t = 0 the cosine is 1 and η=ηmax⁡\eta = \eta_{\max}; at t=Tdt = T_d it is −1-1 and η=ηmin⁡\eta = \eta_{\min}.

Check the middle by hand. At t=Td/2t = T_d / 2 the cosine is 0, so η=2.2×10−5+0.5×1.98×10−4=1.21×10−4\eta = 2.2 \times 10^{-5} + 0.5 \times 1.98 \times 10^{-4} = 1.21 \times 10^{-4}. Our script prints the same value at 12.15T tokens. The four parts also add up: 10 + 4.3 + 0.333 + 0.167 = 14.8T tokens.

Learning rate and batch size during pre-training0.00.51.01.52.00T5T10T14.8Tlearning rate (x 10⁻⁴)tokens seenconstant 2.2 x 10⁻⁴cosine0.22, then 0.07305,00010,00015,3600B200B469B600Bsequences per steptokens seen (first 600B)ramp shape not given;drawn straight
The two schedules of V3's pre-training, drawn from the numbers on pages 22 and 23. The learning rate is flat for 10T tokens, then brakes along a cosine. The batch grows five times over the first 469B tokens; the report does not give the shape of that ramp.

How big is a step? At the full batch, 15,360×4,096=62,914,56015{,}360 \times 4{,}096 = 62{,}914{,}560 tokens. If the ramp is a straight line (our assumption), the first 469B tokens take about 15,000 steps and the rest take about 227,800. That gives about 243,000 steps for the whole run.

plain text
2. Tokens per optimizer step
   start: 3,072 x 4,096 = 12,582,912 tokens   full: 15,360 x 4,096 = 62,914,560 tokens
   steps in the 469B-token ramp (linear ramp assumed): 14,997
   steps after the ramp: (14.8T - 469B) / 62,914,560 = 227,785
   total: about 242,782 steps

The same paragraph sets the other knobs. AdamW with β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95 and weight decay 0.1, gradient clipping at 1.0, initial weights with standard deviation 0.006. Two knobs from Part 2 also change during the run. The multi-token prediction loss weight drops from 0.3 to 0.1 after 10T tokens. The bias update speed of the aux-loss-free balancer is 0.001 until 14.3T tokens, then 0. So the balancing biases stop moving for the last 500B tokens.

Stretching the context to 128K tokens

Pre-training used 4K-token sequences, because attention costs grow with sequence length. A chat model must read much longer inputs. So after pre-training, DeepSeek ran two short phases on longer sequences.

A model trained at 4K positions has never seen the RoPE angles of position 100,000 (Part 2 explains RoPE).

The YaRN attention factor in the report is t=0.1ln⁡s+1\sqrt{t} = 0.1 \ln s + 1. With s=40s = 40, that is 0.1×3.689+1=1.3690.1 \times 3.689 + 1 = 1.369: attention scores get about 37% sharper to make up for the longer, flatter spread over many tokens.

These two phases cost 119K GPU hours for about 126B tokens. That is about 946K GPU hours per trillion tokens, 5.3 times the pre-training rate, because long sequences make attention expensive. The V3.1 model card, months later, gives a useful cross-check. It says the 32K phase "has been increased 10-fold to 630B tokens" and the 128K phase "extended by 3.3x to 209B tokens". Our numbers give 10.0 and 3.32 times the V3 phases.

The hardware: 2,048 H800 GPUs

What is an H800? DeepSeek's later hardware paper, shown at the ISCA 2025 conference (we call it the ISCA paper), answers it.

One node of the cluster (256 such nodes = 2,048 GPUs)Node: 8 x H800NVSwitch: any GPU to any GPU in the nodeGPU 0NIC 0GPU 1NIC 1GPU 2NIC 2GPU 3NIC 3GPU 4NIC 4GPU 5NIC 5GPU 6NIC 6GPU 7NIC 7InfiniBandto the other255 nodesNVLink in the node: 400 GB/s per GPU on the H800 (900 GB/s on the H100); about 160 GB/s usable one way.InfiniBand out of the node: one 400 Gb/s NIC per GPU, about 50 GB/s. Sources: ISCA paper p.7, V3 report p.13.
One H800 node. Eight GPUs share an NVSwitch, and each GPU has its own 400 Gb/s InfiniBand card to the rest of the cluster. The NVLink figure is 400 GB/s per GPU on the H800, against 900 GB/s on the H100.

This is the same habit as the Fire-Flyer 2 cluster in Part 1: accept a limited link, then hide it with software and scheduling.

The parallelism plan

One GPU cannot train V3. In BF16, the weights alone are about 1.3 TB; with gradients and optimizer state, our estimate below is about 9.6 TB. The work must be split. There are four standard ways to split it.

The report gives the three degrees but not the full layout. Here is our reading, using only arithmetic. 2,048 GPUs over 16 stages is 128 GPUs per stage, or 16 nodes. Expert parallelism spans 64 GPUs on 8 nodes, so each stage holds two expert-parallel groups. Each layer has 256 routed experts, and 256 / 64 = 4 experts per GPU; the ISCA paper confirms "4 experts per GPU" for this 8-node setup. The two groups hold the same experts and read different data, so they are data-parallel copies of each other.

How 2,048 GPUs share one model16-way pipeline: the 61 layers are cut into 16 stages, in orderS0S1S2S3S4S5S6S7S8S9S10S11S12S13S14S15each stage: about 4 layers, run by 128 GPUs = 16 nodesexpert-parallel group 1: 8 nodes, 64 GPUsnode 1node 2node 3node 4node 5node 6node 7node 8expert-parallel group 2: 8 nodes, 64 GPUsnode 9node 10node 11node 12node 13node 14node 15node 16Inside a group, the 256 routed experts of each layer are split 4 per GPU (64 x 4 = 256).The two groups hold the same experts and see different data; attention weights sit on all 128 GPUs.Data parallelism with ZeRO-1 splits the optimizer state over the copies. (Stage layout: our reading.)
Our reading of V3's layout. Sixteen pipeline stages of about four layers each. Each stage runs on 128 GPUs: two expert-parallel groups of 64 GPUs on 8 nodes. Inside a group, each GPU holds 4 of the 256 routed experts of every layer in its stage.

Does a stage fit in memory? Our script makes a rough budget. It assumes even stages and DualPipe's two stages per GPU (next section). It uses the formats the report states: BF16 weights, 32-bit gradients and master weights, BF16 AdamW moments.

plain text
7. Rough memory budget per GPU (DERIVED; our reading of the plan, not a number from the paper)
   parameters outside the routed experts: 17.5B   routed experts: 665.2B   (main model 671.0B + MTP module)
   per GPU: 2.18B non-expert + 1.30B routed-expert parameters = 3.48B (4 routed experts per layer per GPU)
   BF16 weights + FP32 gradients: 20.9 GB
   FP32 master weights + BF16 Adam moments (8 bytes per parameter): 27.8 GB without sharing, 5.3 GB with ZeRO-1
   total for the model state: 26.2 GB of 80 GB; the rest is for activations and buffers
   the whole model state on ONE GPU would be 9.6 TB

The surprise is how small the expert share is. Routed experts are 97% of the parameters, but a GPU holds only 4 of 256 per layer, so they shrink 64 times. What stays large is the attention and shared part, which every GPU of a stage keeps whole. ZeRO-1 cuts its optimizer state 128 ways. The 80 GB is our assumption: the papers do not state the H800's memory, and the H100 SXM has 80 GB.

DualPipe: hiding the network behind the compute

The bubble problem

A pipeline has a well-known flaw. At the start, stage 2 waits for stage 1 to finish the first micro-batch. At the end, stage 1 is done while the last micro-batch is still in stage 4. These idle periods are bubbles.

Our toy simulator shows it with 4 stages and 8 micro-batches. A forward pass takes 1 time unit and a backward pass takes 2 (backward does about twice the arithmetic). The simulator runs two classic orders: GPipe, all forwards first and then all backwards, and 1F1B, which alternates as soon as it can.

Toy pipeline: 4 stages, 8 micro-batches, forward = 1, backward = 2 time unitsGPipe: all forwards, then all backwardsGPU 0F0: 0 to 1F0F1: 1 to 2F1F2: 2 to 3F2F3: 3 to 4F3F4: 4 to 5F4F5: 5 to 6F5F6: 6 to 7F6F7: 7 to 8F7B0: 17 to 19B0B1: 19 to 21B1B2: 21 to 23B2B3: 23 to 25B3B4: 25 to 27B4B5: 27 to 29B5B6: 29 to 31B6B7: 31 to 33B7GPU 1F0: 1 to 2F0F1: 2 to 3F1F2: 3 to 4F2F3: 4 to 5F3F4: 5 to 6F4F5: 6 to 7F5F6: 7 to 8F6F7: 8 to 9F7B0: 15 to 17B0B1: 17 to 19B1B2: 19 to 21B2B3: 21 to 23B3B4: 23 to 25B4B5: 25 to 27B5B6: 27 to 29B6B7: 29 to 31B7GPU 2F0: 2 to 3F0F1: 3 to 4F1F2: 4 to 5F2F3: 5 to 6F3F4: 6 to 7F4F5: 7 to 8F5F6: 8 to 9F6F7: 9 to 10F7B0: 13 to 15B0B1: 15 to 17B1B2: 17 to 19B2B3: 19 to 21B3B4: 21 to 23B4B5: 23 to 25B5B6: 25 to 27B6B7: 27 to 29B7GPU 3F0: 3 to 4F0F1: 4 to 5F1F2: 5 to 6F2F3: 6 to 7F3F4: 7 to 8F4F5: 8 to 9F5F6: 9 to 10F6F7: 10 to 11F7B0: 11 to 13B0B1: 13 to 15B1B2: 15 to 17B2B3: 17 to 19B3B4: 19 to 21B4B5: 21 to 23B5B6: 23 to 25B6B7: 25 to 27B71F1B: one forward, one backward, in turnGPU 0F0: 0 to 1F0F1: 1 to 2F1F2: 2 to 3F2F3: 3 to 4F3B0: 10 to 12B0F4: 12 to 13F4B1: 13 to 15B1F5: 15 to 16F5B2: 16 to 18B2F6: 18 to 19F6B3: 19 to 21B3F7: 21 to 22F7B4: 22 to 24B4B5: 25 to 27B5B6: 28 to 30B6B7: 31 to 33B7GPU 1F0: 1 to 2F0F1: 2 to 3F1F2: 3 to 4F2B0: 8 to 10B0F3: 10 to 11F3B1: 11 to 13B1F4: 13 to 14F4B2: 14 to 16B2F5: 16 to 17F5B3: 17 to 19B3F6: 19 to 20F6B4: 20 to 22B4F7: 22 to 23F7B5: 23 to 25B5B6: 26 to 28B6B7: 29 to 31B7GPU 2F0: 2 to 3F0F1: 3 to 4F1B0: 6 to 8B0F2: 8 to 9F2B1: 9 to 11B1F3: 11 to 12F3B2: 12 to 14B2F4: 14 to 15F4B3: 15 to 17B3F5: 17 to 18F5B4: 18 to 20B4F6: 20 to 21F6B5: 21 to 23B5F7: 23 to 24F7B6: 24 to 26B6B7: 27 to 29B7GPU 3F0: 3 to 4F0B0: 4 to 6B0F1: 6 to 7F1B1: 7 to 9B1F2: 9 to 10F2B2: 10 to 12B2F3: 12 to 13F3B3: 13 to 15B3F4: 15 to 16F4B4: 16 to 18B4F5: 18 to 19F5B5: 19 to 21B5F6: 21 to 22F6B6: 22 to 24B6F7: 24 to 25F7B7: 25 to 27B7Dashed gaps = bubbles: each GPU waits 9 of 33 units (27%) in both schedules. Blue: forward. Orange: backward.
Our toy pipeline simulation with 4 stages and 8 micro-batches. In both orders, each GPU sits idle for 9 of 33 time units. 1F1B does not shrink the bubble; it starts backward passes earlier, so each GPU stores fewer activations at once.
plain text
6b. Toy simulation: 4 pipeline stages, 8 micro-batches, F = 1, B = 2
   GPipe  total time 33, idle time per device 9 = 27% of the run (formula (S-1)(F+B) = 9)
   1F1B   total time 33, idle time per device 9 = 27% of the run (formula (S-1)(F+B) = 9)
   the difference between them is memory: at its peak, stage 0 holds activations of 8 micro-batches in GPipe and 4 in 1F1B

The idle time follows a simple rule. The first micro-batch must cross every stage before the last stage starts, and the last one must cross back. So each stage waits for PP−1PP - 1 forward and backward passes in total:

bubble1F1B=(PP−1)(F+B)\text{bubble}_{\text{1F1B}} = (PP - 1)(F + B)

PPPP is the number of pipeline stages, FF the time of one forward chunk and BB the time of one backward chunk. With 4 stages, F=1F = 1 and B=2B = 2, that is 3×3=93 \times 3 = 9 units, exactly what the simulator measured. More micro-batches make the useful work longer, so the bubble becomes a smaller share, but it never goes away.

The communication problem

V3 has a second, bigger problem. Each MoE layer sends every token to its experts and back, across nodes. The report measures how bad that is.

Two forward/backward pairs when sending takes as long as computing (toy times)One after the othercomputeF1B1F2B2networkF1B1F2B212 unitsOverlapped: the network sends one chunk while the GPU computes the nextcomputeF1B1F2B2networkF1B1F2B28 unitsBlue and orange: computing a forward (F) or backward (B) chunk. Green: sending its results to other GPUs.With many pairs in a row, each pair costs 3 units instead of 6: the network hides under the compute.Toy numbers, chosen to match the 1:1 ratio the V3 report gives. Not a measurement.
A toy timing of the idea, with sending as slow as computing. Run one after the other, two forward and backward pairs take 12 units. Overlapped, the network sends one chunk while the GPU computes the next, and they take 8. With long runs, each pair costs half.

Feeding the pipeline from both ends

The name DualPipe comes from the full schedule. Micro-batches enter from both ends of the pipeline at once.

DualPipe feeds micro-batches from both ends (sketch with 4 GPUs)GPU 0stage 0stage 3GPU 1stage 1stage 2GPU 2stage 2stage 1GPU 3stage 3stage 0batches in →← batches inBlue: the model run left to right. Orange: a second copy run right to left.Each GPU holds two stages (2 x parameters), but always has work from one side or the other.GPU 0 holds the first and the last stage, so embedding and output head can live together.
A sketch of DualPipe's two directions on four GPUs. One copy of the model runs left to right, another right to left. Each GPU holds one stage of each copy, so it stores twice the parameters but rarely runs out of work.

Put toy numbers into Table 2. Take PP=16PP = 16, F=1F = 1, B=2B = 2 with W=1W = 1 of it for weights, and assume no gain from overlapping compute, so F&B=3F\&B = 3. Then 1F1B has 45 units of bubble, ZB1P has 15 and DualPipe 14. The formula gain looks small. The large gain is the communication that Figure 4 hides, and the bubble formulas do not show it.

DeepSeek open-sourced DualPipe in February 2025. The repository also has DualPipeV, a "cut-in-half" variant credited to Sea AI Lab that needs half the devices.

Sending tokens across nodes

Overlap only helps if the network part is fast enough to hide. The report's all-to-all design is a direct answer to the H800's links.

How one token reaches experts on another nodenode A (token is here)GPU 0GPU 1GPU 2GPU 3GPU 4GPU 5GPU 6GPU 7node B (3 of its experts)GPU 0GPU 1GPU 2GPU 3GPU 4GPU 5GPU 6GPU 7InfiniBand, onceNVLink inside the nodeThe token crosses InfiniBand once per node, to the GPU with the same number (GPU 3 → GPU 3),then NVLink copies it to the GPUs that hold its experts. At most 4 nodes per token. (V3 report p.13)
The path of one token. It crosses InfiniBand once per target node, to the GPU with the same number, then NVLink copies it to the GPUs that hold its experts.

An SM (streaming multiprocessor) is one of the independent processing units of an NVIDIA GPU. DeepSeek reserves 20 of them to move data and leaves the rest to compute, so the two run at the same time.

The custom kernels behind this became DeepEP, an open-source library for expert-parallel communication, released in February 2025.

Memory-saving tricks

Without tensor parallelism, each GPU must hold more of the model. So DeepSeek also cut memory in three small ways.

FP8: training with 8-bit numbers

This is the part of the report that other labs studied most closely. V3 does most of its matrix products in FP8, numbers that take one byte. It was, the report says, the first time FP8 training was validated "on an extremely large-scale model".

What a floating-point number is

A floating-point number is scientific notation in binary: a sign, an exponent and a mantissa. −1.5×23=−12-1.5 \times 2^{3} = -12 has sign minus, mantissa 1.5 and exponent 3. More exponent bits give a wider range, from tiny to huge. More mantissa bits give finer steps between neighbouring numbers.

Bits of each number format: sign, exponent (range), mantissa (precision)FP3232 bits (1 sign, 8 exponent, 23 mantissa); largest number about 3.4 x 10³⁸BF1616 bits (1 sign, 8 exponent, 7 mantissa); largest number about 3.4 x 10³⁸FP8 E5M28 bits (1 sign, 5 exponent, 2 mantissa); largest number 57,344FP8 E4M38 bits (1 sign, 4 exponent, 3 mantissa); largest number 448signexponent: how bigmantissa: how exactFP8 E4M3 has only 8 steps between 1 and 2; FP32 has about 8 million.
The bits of four number formats. Every bit of exponent buys range and every bit of mantissa buys precision. FP8 E4M3 can hold numbers up to 448, with only 8 steps between 1 and 2.

Why bother? An H800's Tensor Cores do FP8 products about twice as fast as BF16, and FP8 numbers take half the memory and half the network. The report says FP8 "theoretically doubles the computational speed compared with the original BF16 method".

Our script shows how coarse E4M3 is. These are real PyTorch casts to torch.float8_e4m3fn, run on a CPU:

plain text
A. FP8 E4M3 (4 exponent bits, 3 mantissa bits)
   largest number: 448   smallest normal: 0.015625   smallest above zero: 0.00195312
          1 -> stored as 1
        1.1 -> stored as 1.125
        0.3 -> stored as 0.3125
        300 -> stored as 288
      0.004 -> stored as 0.00390625
     0.0001 -> stored as 0
   between 1 and 2 there are only 8 steps: 1, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.875, 2

Two kinds of damage show up. Every number gets rounded to a step that is about 6% to 12% of its size: 300 becomes 288. And very small numbers fall off the bottom: 0.0001 becomes 0.

Why naive FP8 breaks

Here is the scaling rule as an equation:

s=448max⁡i∈G∣xi∣,qi=fp8(s xi),x^i=qiss = \frac{448}{\max_{i \in G} |x_i|}, \qquad q_i = \mathrm{fp8}(s \, x_i), \qquad \hat{x}_i = \frac{q_i}{s}

GG is the group that shares one scale: the whole tensor, a 1 × 128 tile or a 128 × 128 block. ss is the scale, chosen so that the largest value in the group lands on 448. qiq_i is the stored one-byte number, and x^i\hat{x}_i is what comes back when we divide the scale out.

Work it for a group of four numbers, once without and once with an outlier. Our script prints both:

plain text
A2. Worked example: scale, round, scale back
   group [0.5, -1.2, 2.0, 0.01]: scale = 448 / 2 = 224
      0.01 x scale = 2.24 -> FP8 2.25 -> back 0.010045  (error 0.4%)
   group [0.5, -1.2, 1000.0, 0.01]: scale = 448 / 1000 = 0.448
      0.01 x scale = 0.00448 -> FP8 0.0039062 -> back 0.0087193  (error 12.8%)

Same number, 0.01, same format. With a friendly group, it comes back with 0.4% error. With an outlier of 1,000 in its group, it is pushed to 0.00448, below FP8's normal range, where steps are 0.00195 wide. It comes back 12.8% wrong.

Who shares a scale? One outlier (red) in an 8 x 8 corner of a tensorone scale for the tensor1 x 128 tiles (activations)128 x 128 blocks (weights)Orange = numbers whose scale the outlier sets. A group here is 4 cells wide; in V3 it is 128 numbers.
Who shares a scale with an outlier. With one scale per tensor, every number does. With 1 × 128 tiles, only the numbers in the outlier's row-piece do; with 128 × 128 blocks, its whole block does.

Our experiment: one scale against small groups

Now the real test, on a full matrix. Our script builds a 128 × 1,024 activation matrix of random normal numbers. It quantizes it three ways: one scale, 1 × 128 tiles and 128 × 128 blocks. The tile function is short:

python
def per_tile(x, n=128):
    """One scale per row-piece of n channels (1 x n tiles): per token, per 128 channels."""
    r, c = x.shape
    t = x.reshape(r, c // n, n)
    scale = FMAX / t.abs().amax(dim=2, keepdim=True).clamp(min=1e-12)
    return to_fp8_and_back(t, scale).reshape(r, c)

The reshape cuts each row into pieces of 128. amax finds the largest absolute value in each piece. Each piece gets its own scale, then the shared function clamps, casts to FP8 and divides the scale back out.

First, one outlier of growing size, put at row 5, column 300. We measure the error of all the other numbers:

plain text
B. Activation matrix 128 x 1024, random normal (typical |x| about 0.8), one outlier at row 5, column 300
    outlier |      error of the ordinary numbers       |   ordinary numbers turned into 0  
            |       tensor    1x128 tile   128x128 blk |     tensor       tile      block
          0 |       2.65%         2.57%         2.63% |      0.00%      0.00%      0.00%
        100 |       2.67%         2.57%         2.63% |      0.01%      0.00%      0.00%
      1,000 |       2.65%         2.57%         2.63% |      0.19%      0.00%      0.03%
     10,000 |       2.72%         2.57%         2.65% |      1.78%      0.00%      0.24%
    100,000 |      12.56%         2.60%         5.07% |     17.28%      0.02%      2.14%
   for comparison: BF16 (7 stored mantissa bits) gives 0.17% on the same numbers

The result surprised us a little, and it is worth reading slowly. E4M3 has a floating exponent, so moderate outliers do almost no harm even with one scale: up to 10,000, the error stays near 2.7%. Only when the outlier is 100,000 times the typical value does one scale break: 12.56% error, and 17% of the numbers become zero. Tiles stay at 2.6% throughout. Blocks are in between, because the outlier's block is 128 rows tall.

Second, a harder and more realistic case. In gradients, whole rows (tokens) can differ in size by many orders of magnitude. Appendix B.2 of the report calls these "token-correlated outliers". We multiply each row by 10u10^{u}, with uu random between −S-S and 0:

plain text
C. Rows (tokens) of very different sizes: row i is multiplied by 10**u, u uniform in [-S, 0]
   (activation gradients look like this: some tokens carry far larger values than others)
   error = mean over rows of each row's relative error, so small rows count as much as large ones
     S |              error of G              |            error of G W^T           
       |     tensor         tile        block |     tensor         tile        block
     0 |      2.64%        2.56%        2.66% |      3.74%        3.67%        3.73%
     2 |      2.64%        2.58%        2.64% |      3.74%        3.70%        3.72%
     4 |      2.75%        2.57%        2.68% |      3.82%        3.70%        3.78%
     6 |     17.52%        2.57%       16.31% |     18.30%        3.68%       17.14%
     8 |     31.53%        2.56%       30.85% |     32.20%        3.69%       31.53%
Our FP8 test on random data (CPU, PyTorch float8_e4m3fn)A: one outlier in 128 x 1,024error of the other numbers0%10%20%30%none1001K10K100Ksize of the outlierone scale, outlier none: 2.65%one scale, outlier 100: 2.67%one scale, outlier 1K: 2.65%one scale, outlier 10K: 2.72%one scale, outlier 100K: 12.56%1x128 tiles, outlier none: 2.57%1x128 tiles, outlier 100: 2.57%1x128 tiles, outlier 1K: 2.57%1x128 tiles, outlier 10K: 2.57%1x128 tiles, outlier 100K: 2.60%128x128 blocks, outlier none: 2.63%128x128 blocks, outlier 100: 2.63%128x128 blocks, outlier 1K: 2.63%128x128 blocks, outlier 10K: 2.65%128x128 blocks, outlier 100K: 5.07%B: rows differ by up to 10^Smean error per row0%10%20%30%S = 0S = 2S = 4S = 6S = 8one scale, S = 0: 2.64%one scale, S = 2: 2.64%one scale, S = 4: 2.75%one scale, S = 6: 17.52%one scale, S = 8: 31.53%1x128 tiles, S = 0: 2.56%1x128 tiles, S = 2: 2.58%1x128 tiles, S = 4: 2.57%1x128 tiles, S = 6: 2.57%1x128 tiles, S = 8: 2.56%128x128 blocks, S = 0: 2.66%128x128 blocks, S = 2: 2.64%128x128 blocks, S = 4: 2.68%128x128 blocks, S = 6: 16.31%128x128 blocks, S = 8: 30.85%one scale1x128 tiles128x128 blocks
Our FP8 test on random data. A: one outlier in the matrix; one scale fails only at an outlier of 100,000. B: rows of very different sizes; one scale and 128 × 128 blocks fail once rows differ by a million times, while 1 × 128 tiles stay at 2.6%.

Once rows differ by a million times (S=6S = 6), both one scale and 128 × 128 blocks lose the small rows: about 17% error, rising to 31%. The 1 × 128 tiles do not care, because every token gets its own scale. The right half of the table shows the damage survives a matrix product with a weight matrix. This is why V3 uses tiles for activations and blocks only for weights, which are smooth.

The report learned the same lesson the hard way, in an appendix:

The second problem: adding up

A matrix product is a long sum. Each output is a dot product: multiply pairs, then add up thousands of products. Products of FP8 numbers are fine. The running sum needs more care.

We cannot run the H800's Tensor Cores on a laptop, so we built a model of the idea. Our accumulator keeps only 14 significant bits and cuts off the rest after every addition. We add 4,096 products of FP8-rounded random numbers, for 4,096 dot products, two ways:

python
for k in range(K):
    low = keep_bits(low + prod[:, k])                 # (1) everything in the narrow accumulator
    part = keep_bits(part + prod[:, k])               # (2) narrow accumulator for 128 terms...
    if (k + 1) % 128 == 0:
        total = (total + part).float().double()       # ...then moved into an FP32 total
        part.zero_()
plain text
D. SIMULATION: 4,096 dot products of length K = 4,096 (FP8-rounded random inputs)
   narrow accumulator for all 4,096 terms   max error 0.4638%   mean error 0.0643%
   narrow for 128 terms, then FP32          max error 0.0045%   mean error 0.0009%
   (error = |result - exact sum| / sum of |terms|, so it does not blow up when the sum is near zero)
SIMULATION: adding up 4,096 products with a 14-bit accumulatornarrow sum of all 4,096narrow sum of all 4,096: max 0.4638%worst 0.464% mean 0.0643%narrow for 128, then FP32narrow for 128, then FP32: max 0.0045%worst 0.004% mean 0.0009%Error = |result - exact sum| / (sum of |terms|), over 4,096 dot products. A model of the idea,not the H800 circuit: the V3 report measured "nearly 2%" worst-case on the real Tensor Cores.
Our simulation of a 14-bit accumulator. Adding all 4,096 products in it gives a worst-case error of 0.46%; moving the subtotal to FP32 every 128 products cuts it about 100 times. The real H800 measured nearly 2%, so this is a model of the effect, not of the chip.

Promotion every 128 terms cuts the worst error about 100 times in our model. Our worst case, 0.46%, is smaller than the report's "nearly 2%". The report does not describe the real circuit, and it does not behave like our simple cut-off. Only the direction of the result carries over.

What stays in higher precision

FP8 is not used everywhere. Figure 6 shows the plan for one linear layer.

The rest of the precision plan, from §3.3.3 (page 18), fits in a table:

ItemFormat in V3Why
Linear layer products (Fprop, Dgrad, Wgrad)FP8 E4M3 inputs, FP32 sumsMost of the arithmetic
Master weights, weight gradientsFP32Small updates must not round away
AdamW first and second momentsBF16Report: no "observable performance degradation"
Activations saved for backwardFP8, tile-scaledMemory
Input of the linear after attentionCustom E5M6Attention's backward needs it; precision-sensitive
SwiGLU input in MoEFP8, output recomputedMemory
Token dispatch to expertsFP8Halves network traffic
Combine back from expertsBF16Report: kept to "preserve training precision"
Embedding, output head, gate, norms, attentionBF16 or FP32Sensitive or cheap

Did it work? The report checked FP8 against BF16 before trusting it.

The FP8 kernels became DeepGEMM, released in February 2025. V3.1 later switched its scale format to UE8M0, a power-of-two scale that suits the newer "microscaling" hardware formats; its model card says so. The report itself says its tile-and-block design is "highly consistent with the idea of microscaling formats" that NVIDIA's Blackwell GPUs support in hardware.

No loss spikes, no rollbacks

The claim matters because FP8 was new at this scale, and outliers are what cause spikes. A stable run also supports the cost table: 2,664K hours for one clean pass.

V4 makes a useful contrast. Its report is open about trouble:

The cost: $5.576M, and what it leaves out

The numbers are consistent, and the check is a good habit:

plain text
4. The cost paragraph (V3 p.5), checked
   180K GPU hours per trillion tokens x 14.8 = 2,664K GPU hours (paper: 2,664K)
   180K GPU hours / 2,048 GPUs / 24 = 3.66 days per trillion tokens (paper: 3.7)
   pre-training        2,664K GPU hours =  54.2 days on 2,048 GPUs = $5.328M at $2/hour
   context extension     119K GPU hours =   2.4 days on 2,048 GPUs = $0.238M at $2/hour
   post-training           5K GPU hours =   0.1 days on 2,048 GPUs = $0.010M at $2/hour
   total               2,788K GPU hours =  56.7 days on 2,048 GPUs = $5.576M at $2/hour

Pre-training is 54 days on the full cluster, which matches "less than two months". The whole run is about eight weeks of wall-clock time on one large cluster.

What the $5.576M covers, and what it does notCounted (V3 report Table 1)the final pre-training run: 2,664K hcontext extension: 119K hpost-training of V3: 5K hpriced at an assumed $2 perH800 GPU hour (a rental rate)= 2,788K GPU hoursNot countedresearch and ablation runs (stated)buying the GPUs and the buildingspower, network, storage, staffcollecting and cleaning the datathe internal R1 model that madeV3's reasoning training datafailed or restarted runs, if anyLeft: what the paper states. Right: the paper names the first item; the rest follow from what a rental-price total is.
What the $5.576M counts and what it does not. Only the GPU hours of the final run, at an assumed rental price, are inside. The paper names research and ablations as excluded; hardware, staff, power and data work are outside any rental-price total by definition.

Two exclusions are less obvious. First, V3's reasoning training data came from "an internal DeepSeek-R1 model" (report, pages 28 and 29), and that model's cost is not in the 5K post-training hours. Second, the R1 paper later gave its own cost: $294K for R1-Zero and R1 on top of the V3 base. Part 4 covers that number and what it excludes.

A sanity check: does the compute add up?

Can we check 2,664K GPU hours from first principles? There is a well-known rule of thumb for the arithmetic of training.

The intuition. Every parameter that a token touches does one multiply and one add in the forward pass: 2 floating-point operations (FLOPs). The backward pass does about twice the forward work: one product for the gradient of the input, one for the gradient of the weight. So each parameter costs about 6 FLOPs per token.

C≈6 N DC \approx 6 \, N \, D

CC is the total compute in FLOPs. NN is the number of parameters each token uses: for an MoE model, the active parameters, not the total. DD is the number of training tokens. The rule ignores attention's own arithmetic, which grows with sequence length, and small extras.

A tiny example: a 1B-parameter dense model on 1B tokens needs about 6×109×109=6×10186 \times 10^9 \times 10^9 = 6 \times 10^{18} FLOPs. For V3, with 37B active parameters and 14.8T tokens:

C≈6×37×109×14.8×1012≈3.3×1024 FLOPsC \approx 6 \times 37 \times 10^{9} \times 14.8 \times 10^{12} \approx 3.3 \times 10^{24} \text{ FLOPs}

The ISCA paper gives a more careful number: about 250 GFLOPs per token at sequence length 4,096, which includes attention. That gives 3.7×10243.7 \times 10^{24} FLOPs.

Now divide by the time. 2,664K GPU hours is 9.59×1099.59 \times 10^{9} GPU-seconds:

plain text
5. Compute estimate (pre-training only)
   6 x N x D = 6 x 37e9 x 14.8e12 = 3.286e+24 FLOPs
   ISCA paper, 250 GFLOPs per token x 14.8e12 = 3.700e+24 FLOPs
   GPU-seconds: 2,664,000 x 3,600 = 9.5904e+09
   6ND : 343 TFLOPS per GPU on average = 35% of dense BF16 peak (990), 17% of dense FP8 peak (1979)
   ISCA: 386 TFLOPS per GPU on average = 39% of dense BF16 peak (990), 19% of dense FP8 peak (1979)

So each GPU did about 340 to 390 trillion useful operations per second, averaged over the run. The peaks come from NVIDIA's H100 SXM page, which lists 1,979 BF16 and 3,958 FP8 TFLOPS "with sparsity"; dense is half. We assume the H800 has the same Tensor Cores, as the ISCA paper says it is "similar to the H100" apart from FP64 and NVLink.

Is 19% of the FP8 peak low? Not for this job. Attention, normalisation, the gate and the output head run in BF16 or FP32. Scaling and promotion take time. Twenty of each GPU's SMs drive the network. And real models never reach the peak.

The useful conclusion is the reverse one: the claimed GPU hours are physically reasonable. They need no hidden extra compute, and they are not so small that they need a miracle.

How V3.1, V3.2 and V4 were trained differently

DeepSeek did not train every later model from scratch. The table collects what each source states; where a cell is empty, the source does not say.

ModelStarting pointNew training tokensContextPrecisionHardware and cost
V3 (Dec 2024)Random weights14.8T at 4K, plus 2 × 62.9B for long context128KFP8, 1 × 128 and 128 × 128 scales2,048 H800; 2.788M GPU hours
V3.1 (Aug 2025)The V3 base checkpoint630B at 32K and 209B at 128K128KFP8 with UE8M0 scalesNot stated
V3.2 (Dec 2025)V3.1-Terminus2.1B indexer warm-up, then 943.7B with sparse attention128KNot restatedNot stated
V4-Flash / V4-Pro (Apr 2026)Random weights32T / 33T4K growing to 1M during pre-trainingRouted experts in FP4Not stated
V4.1-Flash (Sep 2026)Random weights45T multimodal tokens64K, then 1MFP4 experts, FP8 elsewhere (card)Not stated

Sources: V3.1 model card; V3.2 report (arXiv 2512.02556, page 4); V4 report (arXiv 2606.19348, pages 5 and 25); V4.1-Flash report (arXiv 2609.19969, page 22).

Three changes stand out. First, V3.1 and V3.2 continue training V3's weights; they are not new pre-training runs. V3.2 added sparse attention (Part 5) in about 946B more tokens.

Second, V4 is a new run, and the recipe changed a lot. It trains on twice the data and uses the Muon optimizer for most weights instead of AdamW alone. It grows the sequence length to 1M tokens during pre-training. Its routed experts use FP4, 4-bit numbers. The report puts FP4 training in the post-training stage, as quantization-aware training, which can "fully reuse the existing FP8 training framework". V4.1-Flash keeps a fixed batch of 100.6 million tokens and reports training "with no instability".

Third, from V3.1 on, DeepSeek stopped publishing GPU hours. V3 remains the only DeepSeek model with a full public cost table. Any cost you read for V4 is somebody's estimate.

Next. V3 as described here is a base model with 5K GPU hours of post-training. The model that made DeepSeek famous a month later was R1, built on this base with reinforcement learning. Part 4 explains GRPO, R1-Zero, the four-stage R1 pipeline, the $294K figure and how later models changed post-training.

References