Inside a Modern Language Model: Data, Scale and Architecture

How a frontier language model is built before any chat training: the web data, the scaling rules, the layer design, experts, and the machines that train it.

In 2024, Meta trained a language model with 405 billion parameters on 15.6 trillion tokens of text. The run used up to 16,384 GPUs. In one 54-day stretch of it, the job stopped 466 times, most often because a piece of hardware failed. Each time, the system loaded the last saved copy of the model and carried on.

At the end there was a file of numbers that could continue almost any text. No chat training had happened yet. That file is the base model, and it is the subject of this part. What goes into it? How big should it be? What does each layer look like inside, and what does it cost to make?

This is Part 5 of How AI Got Here. Part 1 told the story of neural networks up to the Transformer and the first scaling laws. Part 2 covered classical machine learning, and Part 3 covered reinforcement learning. Part 4 showed how a base model becomes an assistant. This part goes back one step and opens the base model itself, as it is built in October 2026.

How the pre-trained model took its 2026 shape, 2017 to 2026Transformer, GPT, BERT2017 to 2020Scaling laws: Kaplan, Chinchilla2020 to 2022Block: RMSNorm, SwiGLU, RoPE2019 to 2021Mixture of experts at scale2017 to 2026Open web data: C4 to FineWeb2019 to 2024KV savings: MQA, GQA, MLA2019 to 2024Long context, FP8, multi-token2023 to 202620172019202120232025
The core of the 2017 design is still there. Most changes after 2020 make the same model cheaper to train, cheaper to serve, or able to read longer inputs.

What pre-training optimises

One loss, read off every token

Part 1 introduced the next-token loss. The model reads a text, and at every position it gives a probability to each possible next token. The loss is the negative log of the probability it gave to the token that really came next. Averaged over trillions of positions, that one number is the whole goal of pre-training.

The loss is hard to feel, so people often turn it into perplexity. Think of a quiz where the model must guess the next word. A perplexity of 7 means the model is as unsure as if it had to pick fairly between 7 equal choices at every step.

PPL=exp⁡ ⁣(−1T∑t=1Tlog⁡pθ(xt∣x<t))=eL\text{PPL} = \exp\!\Big( -\frac{1}{T} \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}) \Big) = e^{L}

TT is the number of tokens, xtx_t is the token at position tt and x<tx_{<t} are the tokens before it. pθp_\theta is the model with parameters θ\theta. The part inside the brackets is the average loss LL, so perplexity is just ee to the power of the loss.

Our script gives the scale. A new model that guesses at random over a vocabulary of 128,000 tokens has a loss of ln⁡128,000=11.76\ln 128{,}000 = 11.76 and a perplexity of 128,000. A loss of 4.0 is a perplexity of 54.60, and a loss of 2.0 is 7.39. A single token with probability 0.9 costs 0.105; one with probability 0.01 costs 4.605.

Scaling laws: how big, how long

Counting the compute

A FLOP is one floating-point operation, such as one multiply or one add. Training cost is counted in FLOPs, and there is a famous shortcut for it.

C≈6NDC \approx 6 N D

CC is the total training compute in FLOPs, NN is the number of parameters and DD is the number of training tokens. Each parameter takes part in one multiply and one add for each token on the forward pass, which is 2 FLOPs. The backward pass costs about twice the forward pass, which adds 4. So every parameter costs about 6 FLOPs per token.

A model with 1 billion parameters trained on 20 billion tokens costs 6×109×2×1010=1.2×10206 \times 10^9 \times 2 \times 10^{10} = 1.2 \times 10^{20} FLOPs. The rule leaves out the attention scores, which matter only for long texts. Jared Kaplan and colleagues used it in their 2020 paper, and almost every lab uses it today.

Here are some real runs, from our script:

ModelParameters NNTokens DD6ND6ND (FLOPs)Tokens per parameter
GPT-3 (2020)175B300B3.15×10233.15 \times 10^{23}1.7
Gopher (2021)280B300B5.04×10235.04 \times 10^{23}1.1
Chinchilla (2022)70B1.4T5.88×10235.88 \times 10^{23}20
Llama 3 8B (2024)8B15T7.2×10237.2 \times 10^{23}1,875
Llama 3 405B (2024)405B15.6T3.79×10253.79 \times 10^{25}38.5
DeepSeek-V3 (2024)37B active14.8T3.29×10243.29 \times 10^{24}400

The Llama 3 paper itself reports 3.8×10253.8 \times 10^{25} FLOPs for the 405B model, so the shortcut agrees. For DeepSeek-V3, NN is the number of parameters each token uses, not the total; the section on experts explains why.

Kaplan 2020: smooth power laws

Part 1 introduced the first scaling-law paper. Its main picture shows why it changed how labs plan.

Kaplan's team also asked how to split a growing budget. Their answer was to put most of it into parameters: the best model size grew as N∝C0.73N \propto C^{0.73}. GPT-3, Gopher and Megatron-Turing NLG followed that advice. They were very large and trained on 300 billion tokens or fewer.

Chinchilla 2022: the balance was wrong

In March 2022, Jordan Hoffmann and colleagues at DeepMind repeated the study with a different method. They trained over 400 models, and changed the learning-rate schedule to match each run's length, which Kaplan's runs had not done.

The paper fits one formula to all its runs. It splits the loss into three parts. One is a floor that no model can beat, one falls as the model grows, and one falls as the data grows.

L(N,D)=E+ANα+BDβL(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

EE is the irreducible loss, the randomness of language itself; no model can predict a coin toss in a story. A/NαA / N^{\alpha} is the penalty for a finite model and B/DβB / D^{\beta} the penalty for finite data. α\alpha and β\beta say how fast each penalty shrinks. The fitted values in the paper are E=1.69E = 1.69, A=406.4A = 406.4, B=410.7B = 410.7, α=0.34\alpha = 0.34 and β=0.28\beta = 0.28.

Put in Gopher, with N=280×109N = 280 \times 10^9 and D=300×109D = 300 \times 10^9. The model term is 0.0521 and the data term is 0.2511, for a loss of 1.9933. Gopher's loss was mostly a data problem. For Chinchilla, with N=70×109N = 70 \times 10^9 and D=1.4×1012D = 1.4 \times 10^{12}, the terms are 0.0835 and 0.1632, for a loss of 1.9366. The smaller model wins because its two penalties are better balanced.

To find the best split for a budget, put D=C/(6N)D = C / (6N) into the formula and find the NN with the lowest loss. Each budget gives a U-shaped curve. Too small a model wastes the budget on tokens it cannot use; too large a model sees too few tokens.

Iso-FLOP curves from the Chinchilla fit: each budget has a best model size1.92.12.32.52.7100M1B10B100B1Tparameters N (log scale); tokens D = C / (6N)loss10²¹ FLOPs10²²10²³5.76 × 10²³ (Gopher, Chinchilla)dot = lowest loss for that budget
Drawn from the paper's fitted formula L(N, D) = 1.69 + 406.4/N^0.34 + 410.7/D^0.28, not from its measured runs. Too small a model starves on parameters; too large a model starves on tokens.

Calculus gives the bottom of each U in closed form:

Nopt(C)=G(C6)βα+β,Dopt(C)=C6 Nopt(C),G=(αAβB)1α+βN_{\text{opt}}(C) = G \left( \frac{C}{6} \right)^{\frac{\beta}{\alpha + \beta}}, \qquad D_{\text{opt}}(C) = \frac{C}{6 \, N_{\text{opt}}(C)}, \qquad G = \left( \frac{\alpha A}{\beta B} \right)^{\frac{1}{\alpha + \beta}}

With the fitted values, the exponent is 0.452 for NN and 0.548 for DD. So as the budget grows, the best model grows a little slower than its data, and the two grow at almost the same rate. That is the paper's main finding, and it is very different from Kaplan's 0.73.

Over-training on purpose

Chinchilla answers one question: the lowest loss for a training budget. But a model is trained once and then used millions of times. Every user's token costs about 2N2N FLOPs to produce, so a smaller model is cheaper forever. If you expect heavy use, you should train a smaller model for longer than "optimal".

Our script puts numbers on that trade, with the Chinchilla fit as a rough guide (Meta's data and tokenizer differ). An 8B model on 15T tokens uses 7.2×10237.2 \times 10^{23} FLOPs and reaches a fitted loss of 1.9485. The compute-optimal model for the same budget has 35.6B parameters and a loss of 1.9226. The small model gives up only 0.0259 of loss, and it is 4.5 times cheaper per token to serve. The same 8B model with 20 tokens per parameter would reach only 2.1640.

The data wall, and a second axis

If models keep growing, and small ones keep reading more, the data must grow too. Epoch AI estimated in 2024 that the stock of good public human text is on the order of 300 trillion tokens. Their paper's figure is 320 trillion, with a wide range. They expect frontier runs to reach that size "between 2026 and 2032", and sooner if models are over-trained.

Llama 3 405B already read 15.6 trillion tokens, so the wall is not far off for text alone. That is one reason why labs now mix in images, audio and video, and why they write data with models. It is also why the second axis in Part 4, test-time compute, matters so much. A model that thinks longer can get better without any new pre-training data.

The data

Starting from the web

A crawl is raw HTML: menus, cookie banners, adverts, spam, copies of copies, and pages in hundreds of languages. Maybe a small share is text you would want a model to learn from. The art of pre-training data is throwing away the rest without throwing away the good.

From raw web pages to training tokensCommon Crawlraw HTML, monthlyExtract textdrop menus, ads, codeLanguage IDkeep the target languagesHeuristic filterslength, repeats, spamDeduplicateURL, MinHash, linesQuality classifierkeep the best pagesMix and weightweb, code, mathsTokenisetext becomes integer IDsFineWeb (2024): 96 Common Crawl snapshots in, 15 trillion tokens out. FineWeb-Edu keeps 1.3 trillion of them.Llama 3 mix (2024): about 50% general knowledge, 25% maths and reasoning, 17% code, 8% other languages.
Each step throws data away. Deduplication and the quality classifier (highlighted) changed results the most in the open studies of 2023 and 2024.

The first steps are mechanical. Text is pulled out of the HTML, and a small classifier guesses the language of each page. Simple rules then drop pages that are too short, too repetitive or full of bad words. C4, the "Colossal Clean Crawled Corpus" that Google built for its T5 model in 2019, used rules such as "keep only lines that end with a punctuation mark".

Deduplication

The web repeats itself. The same news story, licence text or product page appears thousands of times. A model trained on such copies memorises them and wastes compute on them. So every pipeline removes near-duplicates.

Near copies are measured by overlap. Cut each document into shingles, runs of a few words in a row, and compare the two sets:

J(A,B)=∣A∩B∣∣A∪B∣J(A, B) = \frac{|A \cap B|}{|A \cup B|}

AA and BB are the sets of shingles of two documents. ∣A∩B∣|A \cap B| counts the shingles they share, and ∣A∪B∣|A \cup B| counts all distinct shingles of both. JJ is the Jaccard similarity: 1 for identical sets, 0 for no overlap.

Take "the cat sat on the mat and looked out of the window" and the same sentence ending in "door". With 3-word shingles, each has 10. They share 9, and the union has 11, so J=9/11=0.8182J = 9/11 = 0.8182. They are near duplicates.

Comparing every pair of billions of pages is impossible. MinHash gives each document a short signature of hash values, so that two signatures agree at one spot with probability JJ. FineWeb uses 112 hashes in 14 buckets of 8. Two documents are flagged if all 8 hashes agree in any one bucket. The chance of that is 1−(1−J8)141 - (1 - J^8)^{14}. From our script, it is 0.0533 at J=0.5J = 0.5, 0.7716 at 0.75 and 0.9996 at 0.9: a soft threshold near 75%.

One of its results surprised the authors. They first deduplicated across all 96 snapshots at once, as you might expect to do.

Quality classifiers

Rules can remove obvious junk. They cannot tell a clear explanation from a dull one. For that, labs train a quality classifier: a small model that scores each page, so the pipeline keeps the best ones.

For FineWeb-Edu, Llama-3-70B-Instruct scored 460,000 web pages from 0 to 5 for educational value. A small classifier learned those scores and ran over all 15 trillion tokens, which took 6,000 H100 GPU hours. Pages that scored 3 or more were kept: 1.3 trillion tokens. A model trained on them matched the final MMLU score of another open set, Matrix, with almost ten times fewer tokens.

DataComp-LM (DCLM), from a large team of university and industry researchers in June 2024, ran the same kind of test as a contest. It released a pool of 240 trillion tokens from Common Crawl and fixed the training code, so only the data changed. Its best filter, also a small classifier, trained a 7B model to 64% on MMLU with 2.6 trillion tokens. Llama 3's paper describes classifiers of the same kind, trained on Llama 2's judgements.

Code, maths and data written by models

Code and maths pages are rare on the web, but they teach skills that help with reasoning in general. Pipelines use separate classifiers to find them. Llama 3's final mix had about 50% general knowledge, 25% maths and reasoning, 17% code and 8% text in other languages.

In June 2023, Microsoft's phi-1 paper, "Textbooks Are All You Need", made the case in its title. Its model had only 1.3 billion parameters. It was trained on 6 billion tokens of "textbook quality" web data and 1 billion tokens of textbooks and exercises written by GPT-3.5. It reached 50.6% on the HumanEval coding test, far above much larger models of the time. By 2025, rewriting web text with a model was a standard step, for example in the Kimi K2 report.

Mixtures, annealing and mid-training

Choosing the mix is itself an experiment. Llama 3 trained small models on candidate mixes, fitted scaling laws to them, and predicted how a large model would do. The end of training gets special care too, because the model learns most from what it reads last, while its learning rate falls.

Llama 3's paper says that "during pre-training on the final 40M tokens, we linearly annealed the learning rate to 0". During that phase, the mix was changed "to upsample data sources of very high quality". The final model is an average of several checkpoints from the annealing phase. Meta also used annealing as a test: train briefly on a new small data set while the rate falls, and see if scores rise.

This end phase has grown into a stage of its own, often called mid-training. As Part 4 described, Olmo 3 runs 100 billion tokens of maths, code and question answering after its main pre-training. Long-context training usually follows.

Tokenizers: from text to integers

Byte-pair encoding

A model reads integers, not letters. A tokenizer turns text into a list of token IDs from a fixed vocabulary. Whole words would make the vocabulary endless, and single letters would make texts very long. The standard answer sits between them, and it came from data compression.

Rico Sennrich, Barry Haddow and Alexandra Birch brought BPE to neural translation in 2016, and printed the whole algorithm in their paper.

We ran the same toy example. The first merge is "e" + "s", with a count of 9 (6 from newest, 3 from widest). Then "es" + "t", then "est" + "", all with 9. Then "l" + "o" and "lo" + "w", with 7. After 10 merges, the word "lowest", which was never in the data, splits into "low" and "est".

Byte-pair encoding: merge the most frequent pair, again and againwords and counts: low ×5, lower ×2, newest ×6, widest ×3 (</w> marks the end of a word)merge (count)lowlowernewestwideststartl o w </w>l o w e r </w>n e w e s t </w>w i d e s t </w>1: e + s (9)l o w </w>l o w e r </w>n e w es t </w>w i d es t </w>2: es + t (9)l o w </w>l o w e r </w>n e w est </w>w i d est </w>3: est + </w> (9)l o w </w>l o w e r </w>n e w est</w>w i d est</w>4: l + o (7)lo w </w>lo w e r </w>n e w est</w>w i d est</w>5: lo + w (7)low </w>low e r </w>n e w est</w>w i d est</w>... merges 6 and 7: n + e, ne + w ...8: new + est</w> (6)low </w>low e r </w>newest</w>w i d est</w>After 10 merges, "lowest" (never seen) becomes low + est</w>. The vocabulary grows by one symbol per merge.
The toy vocabulary of Sennrich, Haddow and Birch (2016), run by our script with the paper's own algorithm. Real vocabularies, built on bytes instead of letters, hold 32,000 to 262,000 tokens.

Vocabulary sizes, then and now

GPT-2 in 2019 ran BPE on bytes, not letters, so any text, in any script, can be encoded. Its vocabulary had 50,257 tokens. Llama 2 in 2023 used 32,000. Since then, vocabularies have grown: Llama 3 has 128,000 tokens (100,000 from OpenAI's tiktoken plus 28,000 for other languages). DeepSeek-V3 has 128,000, and Gemma 3 uses Gemini 2.0's tokenizer with 262,000 entries.

A bigger vocabulary means fewer tokens per text, so the model reads more text for the same compute. Meta reports that its new tokenizer raised the average from 3.17 to 3.94 English characters per token, compared with Llama 2's. The cost is a bigger embedding table and output layer, and rare tokens that the model sees too seldom to learn well. Languages that were poorly served by older tokenizers, where one word could cost five or more tokens, gain the most.

The 2026 decoder block

The Transformer of 2017 had an encoder and a decoder for translation. Almost every large language model since GPT has kept only the decoder: a stack of identical layers, each with attention and a feed-forward network. The layer of 2026 looks like the old one from far away. Up close, almost every part has been swapped.

One Transformer layer, 2017 and 20262017: "Attention Is All You Need"tokens in+ sine positions(added once, at the input)Multi-head attentionevery head has its own K, VAdd, then LayerNorm(post-norm)Feed-forwardReLU, 4d wideAdd, then LayerNormnext layerDense: every token uses every weight.Encoder and decoder, for translation.Base model: 65M parameters.2026: a typical decoder (Llama 3, DeepSeek-V3)tokens inRMSNorm(pre-norm, before the block)Attention with RoPEGQA or MLA: shared K, VAdd (residual)RMSNormSwiGLU feed-forward,often a mixture of expertsAdd (residual)next layerDecoder only; 128K to 1M+ tokens.Often sparse: a few experts per token.Llama 3 405B: 126 layers deep.
Same skeleton, new parts. The norm moved before each block, the positions moved into attention, the keys and values are shared, and the feed-forward got a gate and often a router.

Pre-norm and RMSNorm

Deep networks need their numbers kept at a steady size from layer to layer, or training blows up or stalls. Normalisation does this. The 2017 Transformer normalised after adding each block's output to its input ("post-norm"). GPT-2 moved the normalisation to the input of each block ("pre-norm"). In 2020, Ruibin Xiong and colleagues showed why that helps. With pre-norm, the gradients are well behaved at the start, so training needs less of a careful warm-up.

In 2019, Biao Zhang and Rico Sennrich proposed a cheaper normaliser. LayerNorm subtracts the mean and divides by the standard deviation. RMSNorm skips the mean and divides by the root mean square.

RMSNorm(x)i=xi1d∑j=1dxj2+ϵ  gi\text{RMSNorm}(x)_i = \frac{x_i}{\sqrt{\frac{1}{d} \sum_{j=1}^{d} x_j^2 + \epsilon}} \; g_i

xx is the vector for one token, with dd numbers. The square root is the root mean square: square every number, take the average, take the root. ϵ\epsilon (epsilon) is a tiny number that prevents division by zero, and gig_i is a learned scale for each position.

Take x=[1,2,3,4]x = [1, 2, 3, 4], with g=1g = 1. The mean of the squares is (1+4+9+16)/4=7.5(1 + 4 + 9 + 16)/4 = 7.5, and its root is 2.7386. RMSNorm gives [0.3651,0.7303,1.0954,1.4606][0.3651, 0.7303, 1.0954, 1.4606]. LayerNorm would first subtract the mean 2.5 and give [−1.3416,−0.4472,0.4472,1.3416][-1.3416, -0.4472, 0.4472, 1.3416]. RMSNorm keeps the direction of the vector and only fixes its size. The paper reports the same quality as LayerNorm and running times 7% to 64% shorter.

SwiGLU: a feed-forward network with a gate

The feed-forward network is where most of a dense model's parameters live. In 2017 it was two matrices with a ReLU between them. In 2020, Noam Shazeer tested a family of gated versions, and one, SwiGLU, became the standard.

FFNSwiGLU(x)=(SiLU(xW)⊙xV)W2,SiLU(z)=z1+e−z\text{FFN}_{\text{SwiGLU}}(x) = \big( \text{SiLU}(x W) \odot x V \big) W_2, \qquad \text{SiLU}(z) = \frac{z}{1 + e^{-z}}

xx is a token's vector. WW and VV are two matrices that widen it to dffd_{\text{ff}} numbers, and W2W_2 brings it back to dd. ⊙\odot means multiply element by element. SiLU (also called Swish) is a smooth version of ReLU. It is close to zz for large zz and close to 0 for very negative zz.

Take one element with xW=2xW = 2 and xV=3xV = 3. SiLU(2) is 1.7616, so the output is 1.7616×3=5.28481.7616 \times 3 = 5.2848. If xVxV were 0, the output would be 0 whatever the gate said. The two paths learn together which features to pass on.

SwiGLU has three matrices instead of two. To keep the parameter count the same, dffd_{\text{ff}} is set near 83d\frac{8}{3} d instead of 4d4d, since 3×d×83d=8d23 \times d \times \frac{8}{3}d = 8d^2 is what two matrices of d×4dd \times 4d cost. Llama 3 8B uses d=4,096d = 4{,}096 and dff=14,336d_{\text{ff}} = 14{,}336, which is 3.5 times wider. Its feed-forward weights alone come to 176,160,768 per layer.

RoPE: positions as rotations

Attention, on its own, sees a set of tokens with no order. The 2017 Transformer added a fixed wave pattern to each token's vector at the input. That works, but the model must learn to pull position out of the mixture again, and it handles texts longer than its training texts poorly.

In 2021, Jianlin Su and colleagues at Zhuiyi Technology proposed rotating the queries and keys instead. Think of a clock hand. A token at position mm has its hand turned by mm steps. Two hands turned by 3 and 1 steps make the same angle as two hands turned by 7 and 5 steps. Attention compares angles, so it sees only the distance.

For one pair of numbers, the rotation is a 2-by-2 matrix:

R(mθ)=(cos⁡mθ−sin⁡mθsin⁡mθcos⁡mθ),(R(mθ) q)⊤(R(nθ) k)=q⊤R((n−m)θ) kR(m\theta) = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix}, \qquad \big( R(m\theta)\, q \big)^{\top} \big( R(n\theta)\, k \big) = q^{\top} R\big( (n - m)\theta \big)\, k

qq and kk are a pair of numbers from a query and a key. mm and nn are the positions of their tokens, and θ\theta (theta) is the turning speed for this pair. The right side shows that the score depends on mm and nn only through n−mn - m. That works because turning one vector by aa and another by bb changes the angle between them by b−ab - a.

Our script checks it with q=(1,0.5)q = (1, 0.5), k=(0.8,−0.2)k = (0.8, -0.2) and θ=0.3\theta = 0.3. Without rotation, q⋅k=0.7q \cdot k = 0.7. With the query at position 3 and the key at 1, the score is 0.2389. At positions 7 and 5 it is 0.2389, and at 102 and 100 it is again 0.2389. At positions 5 and 1, four apart, it is −0.3056-0.3056.

RoPE: turn each pair of numbers by an angle that grows with positionpositions 3 and 1q at position 3k at position 1positions 7 and 5: both turned 4θ moreq at 7k at 5The angle between q and k is the same in both pictures, so the dot product is the same: 0.2389.Attention sees only the distance, 2 tokens, not where the pair sits. (θ = 0.3, from our script.)
A real head has 64 such pairs, each turning at its own speed, from one radian per token down to a full turn every 54,000 tokens (base 10,000) or 2.6 million tokens (base 500,000).

A real head has many pairs, each with its own speed: θi=b−2i/d\theta_i = b^{-2i/d} for pair ii of a head with dd numbers, with a base bb. Fast pairs track nearby words; slow pairs track far ones. With d=128d = 128 and the usual base of 10,000, the slowest pair turns once every 54,410 tokens. Llama 3 raised the base to 500,000, which makes the slowest turn last 2,559,196 tokens and helps with long texts.

Attention cost and FlashAttention

Attention compares every token with every other token. For nn tokens, that is a table of n×nn \times n scores per head and per layer.

FLOPsscores≈2 n2dper layer,memoryscores=n2 numbers per head\text{FLOPs}_{\text{scores}} \approx 2\, n^2 d \quad \text{per layer}, \qquad \text{memory}_{\text{scores}} = n^2 \text{ numbers per head}

nn is the sequence length and dd the model width. Our script shows the memory side for one head in 16-bit numbers. At 2,048 tokens the score table is 8 MiB. At 8,192 it is 128 MiB. At 131,072 tokens it is 32,768 MiB, or 32 GiB, for a single head of a single layer.

Most of the time of plain attention goes not into arithmetic but into writing the big score table to HBM and reading it back. In May 2022, Tri Dao and colleagues at Stanford fixed that without changing the result.

The paper reports a 3× speedup on GPT-2 training at 1K tokens. FlashAttention-2 and -3 followed for newer GPUs, and some version of it now runs inside nearly every training and serving system. The arithmetic is still n2n^2, which is why long-context models also use the sparser patterns of the attention series, such as Gemma 3's local layers.

Sharing keys and values: MQA, GQA and MLA

When a model writes text, it keeps the keys and values of every earlier token, so it does not compute them again. This store is the KV cache, and the inference series covers it in depth. Its size often decides how many users one GPU can serve.

KV bytes per token=2×L×Hkv×dh×b\text{KV bytes per token} = 2 \times L \times H_{kv} \times d_h \times b

The 2 counts keys and values. LL is the number of layers and HkvH_{kv} the number of key-value heads. dhd_h is the size of each head and bb the bytes per number (2 for 16-bit). Plain multi-head attention (MHA) has one key-value head per query head.

Take the shape of Llama 3 70B: 80 layers, 64 query heads of size 128. With MHA, one token would need 2×80×64×128×2=2,621,4402 \times 80 \times 64 \times 128 \times 2 = 2{,}621{,}440 bytes, or 2,560 KiB. A 128K-token chat would need 320 GiB, four times an 80 GB GPU. That is why nobody builds it that way.

Noam Shazeer proposed MQA in 2019. It shrinks the cache by the number of heads, but it can lose quality. In 2023, Joshua Ainslie and colleagues at Google proposed GQA as the middle way. They also showed that an existing MHA model can be converted with about 5% of its original training compute. Llama 3 70B uses 8 key-value heads for its 64 query heads: 320 KiB per token, and 40 GiB for 128K tokens. MQA, with one head, would need 40 KiB.

DeepSeek went further in May 2024 with multi-head latent attention in DeepSeek-V2.

DeepSeek-V3 stores 512 numbers for the latent plus 64 for a small RoPE key, per layer. With 61 layers, that is 61×576×2=70,27261 \times 576 \times 2 = 70{,}272 bytes, or 68.6 KiB per token. The same shape with full MHA, 128 heads of size 128, would need 3,904 KiB. MLA is 56.9 times smaller.

KV cache for one token, all layers, 16-bit numbersLlama 3 70B shape, MHA (64 KV heads)2,560 KiB320 GiBLlama 3 70B, GQA (8 KV heads, as built)320 KiB40 GiBLlama 3 70B shape, MQA (1 KV head)40 KiB5 GiBDeepSeek-V3 shape, MHA (128 heads)3,904 KiB488 GiBDeepSeek-V3, MLA (576 numbers a layer, as built)68.6 KiB8.6 GiBat 128K tokensGQA cuts the Llama cache 8 times. MLA cuts the DeepSeek cache 56.9 times, because it stores one small latentvector per layer and rebuilds the keys and values from it. An 80 GB GPU cannot even hold the MHA cache for one long chat.
Bytes per token = 2 (keys and values) × layers × KV heads × head size × 2 bytes. The MHA rows are what the same shapes would need without sharing. Numbers from our script.

Long context

A model trained on 8,000-token texts does not simply work on 128,000. Its slow RoPE pairs never turned that far, so long distances look like nothing it has seen. The fixes stretch the rotations.

In 2023, Shouyuan Chen and colleagues at Meta proposed position interpolation. It divides every position by a scale ss, so 128K positions fit into the old 8K range. From 8K to 128K, s=16s = 16. Later methods scale different pairs by different amounts, so fast pairs keep their precision. YaRN, from Bowen Peng and colleagues in 2023, needed "10x less tokens and 2.5x less training steps than previous methods".

Labs now do this as a stage of pre-training. Llama 3 went from 8K to 128K "in six stages". DeepSeek-V3 used YaRN to go from 4K to 32K and then to 128K. In February 2024, Google's Gemini 1.5 report showed "near-perfect retrieval (>99%) up to at least 10M tokens" in research tests. The models it shipped took one or two million tokens. Google has not published how it does this.

Mixture of experts

More parameters, same cost per token

A dense model uses every weight for every token. That ties two things together: what the model can know (its parameters) and what each token costs (its compute). A mixture of experts cuts that tie. The feed-forward block is replaced by many smaller blocks, the experts, and a small network chooses a few of them for each token.

Think of a hospital. It has many specialists, but each patient sees only two or three, chosen at the front desk. The hospital knows a great deal, and each visit stays short. The front desk is the router, and it learns its choices during training.

p=softmax(xWr),y=∑i∈TopK(p)pi∑j∈TopK(p)pj Ei(x)p = \text{softmax}(x W_r), \qquad y = \sum_{i \in \text{TopK}(p)} \frac{p_i}{\sum_{j \in \text{TopK}(p)} p_j} \, E_i(x)

xx is a token's vector and WrW_r the router's weight matrix, with one column per expert. pp holds one probability per expert. TopK(p)\text{TopK}(p) is the set of the kk experts with the highest probabilities. Ei(x)E_i(x) is the output of expert ii, and the fraction is its gate: its probability, rescaled so the chosen gates add up to 1.

Take 8 experts and a token whose router scores are 2.0, 1.0, 0.5, 0.1, −0.3, −1.0, 0.0 and 0.3. Softmax turns them into 0.4528, 0.1666, 0.1010, 0.0677, 0.0454, 0.0225, 0.0613 and 0.0827. With k=2k = 2, experts 0 and 1 win. Their gates are 0.4528/0.6194=0.73110.4528 / 0.6194 = 0.7311 and 0.2689. The other six experts do no work for this token.

A mixture-of-experts layer: a router picks 2 of 8 experts for each tokentoken xhidden vectorroutersoftmax(x W_r)E00.4528E10.1666E20.1010E30.0677E40.0454E50.0225E60.0613E70.0827FFN expert 0gate 0.7311FFN expert 1gate 0.2689output0.7311 E0(x) + 0.2689 E1(x)The six other experts do no work for this token. The two chosen scores, 0.4528 and 0.1666, are rescaled to add up to 1.Each token picks its own pair, so all experts are busy over a batch, but each token pays for only two.
One token of our script's router. Mixtral uses exactly this shape: 8 experts, top 2. DeepSeek-V3 picks 8 of 256 routed experts, plus 1 shared expert that every token uses.

Noam Shazeer and colleagues at Google made this work for language models in 2017, with a paper titled "Outrageously Large Neural Networks". It used thousands of experts between LSTM layers. The idea moved into Transformers with Google's GShard in 2020, then into the Switch Transformer.

Keeping the experts busy

A router left alone tends to collapse. A few experts get slightly better early, so the router sends them more tokens, so they get better still. The others starve. And in practice each expert lives on a different GPU, so an overloaded expert makes the others wait.

To keep the load even, Switch adds a small load-balancing loss to the training loss:

Laux=α N∑i=1Nfi Pi\mathcal{L}_{\text{aux}} = \alpha \, N \sum_{i=1}^{N} f_i \, P_i

NN is the number of experts. fif_i is the share of tokens in the batch that were actually sent to expert ii. PiP_i is the average router probability for expert ii over the batch. α\alpha is a small weight; the paper uses 10−210^{-2}. The sum is smallest when both shares are spread evenly.

With 8 experts and perfect balance, every fif_i and PiP_i is 1/81/8, so N∑fiPi=8×8×164=1N \sum f_i P_i = 8 \times 8 \times \frac{1}{64} = 1. If every token goes to expert 0 with probability 1, the sum is 8, the worst case. Our script also ran a random router that favours two experts, on 4,096 tokens. It sent 46.8% and 24.8% of tokens to them, and scored 1.7124, so the loss pushes it back toward 1.

Mixtral: an open MoE

In December 2023, Mistral AI released Mixtral 8x7B with open weights, and its paper followed in January 2024.

DeepSeek-V3: many small experts and one shared one

DeepSeek pushed the design in a different direction: many small experts instead of a few large ones, plus experts that every token uses.

DeepSeek-V3 has 61 layers. All but the first three use MoE, with 1 shared expert and 256 routed experts, of which each token uses 8. So each token passes through 9 of 257 expert networks. The model has 671 billion parameters, and 37 billion, 5.5%, are active for each token.

It also changed how balance is kept. A large auxiliary loss can hurt quality. So DeepSeek-V3 gives each expert a bias, which is added to its score only when the top 8 are chosen. After each step, the bias of an overloaded expert goes down and that of an idle one goes up. The report calls this "auxiliary-loss-free" balancing, and keeps only a tiny loss of the Switch kind as a backstop.

What the closed labs say

Google has said the most. The Gemini 1.5 report of 2024 calls Gemini 1.5 Pro "a sparse mixture-of-expert (MoE) Transformer-based model". The 2025 report on Gemini 2.5 says the same of the whole family, and adds what it is for.

The Gemini 3 Pro model card says the same: a "sparse mixture-of-experts (MoE)" Transformer "with native multimodal support for text, vision, and audio inputs", trained on Google's TPUs. Meta's Llama 4 models of April 2025 are MoE too. Llama 4 Scout has 17 billion active parameters, 16 experts and 109 billion in total; Maverick has 17 billion active, 128 experts and 400 billion in total. Kimi K2 has about a trillion parameters with 32 billion active.

OpenAI and Anthropic do not publish the architecture of their frontier models. Press reports about their sizes and designs exist, but none comes from the labs, so this part does not repeat them.

Native multimodality

Part 1 showed the Vision Transformer of 2020, which treats an image as a sequence of patches. That one idea is what lets a language model see.

Early open models used late fusion: a frozen image encoder, a small adapter and a language model, trained together only at the end. Google's first Gemini report, in December 2023, described its models as multimodal from the start of training. Meta's Chameleon paper of May 2024 described "early-fusion token-based mixed-modal models", and Llama 4 uses "early fusion to seamlessly integrate text and vision tokens".

One sequence, many kinds of tokenimage 224 × 224cut into 16 × 16 patchesaudioshort frames → vectorstextBPE tokens → vectors196 patch tokenslinear projectionaudio tokensan audio encodertext tokensembedding tableone Transformer reads it allsame attention, same layersthe token streamGemini and Llama 4 are described as natively multimodal: they meet images (and, for Gemini, sound) in pre-training,not only through an adapter bolted on afterwards. The exact encoders of the closed models are not public.
A 224 × 224 image with 16 × 16 patches gives 196 tokens, the setting of the 2020 Vision Transformer. Modern models use bigger images and compress the patch tokens.

Audio and video follow the same pattern. Sound is cut into short frames, an encoder turns them into vectors, and video is a stream of image patches plus its sound track. The Gemini 1.5 report tested retrieval on up to 9.7 million audio tokens. The exact encoders inside Gemini and other closed models are not public.

Training at scale

The machines

Training a frontier model needs thousands of accelerators working as one. Two families dominate: NVIDIA's GPUs and Google's TPUs.

An NVIDIA H100 does 989.4 trillion 16-bit FLOPs per second at its peak, on dense matrices. It carries 80 GB of HBM3 memory at 3.35 TB/s. Meta trained Llama 3 405B on up to 16,384 of them.

Google builds its own chips and joins them into pods. Gemini 1.5 was trained on several 4,096-chip pods of TPU v4, the report says. A TPU v5p chip does 459 trillion 16-bit FLOPs per second, and one pod holds 8,960 chips. Ironwood, announced in April 2025, puts 9,216 chips in a pod, with 4,614 trillion 8-bit FLOPs per second for each chip.

A blue circuit board with four large chips under copper cold plates, joined by grey liquid-cooling tubes
A TPU v3 board: four chips, each under a copper plate, cooled by liquid. Photo: Zinskauf, CC BY-SA 4.0, via Wikimedia Commons.
A green circuit board with four TPU v4 chips, joined by coloured liquid-cooling tubes
A TPU v4 board, the chip generation that trained Gemini 1.5. Image: Norman P. Jouppi and colleagues, from their 2023 paper on TPU v4, CC BY 4.0, via Wikimedia Commons.

No run reaches the peak. Llama 3 reports an MFU of 38% to 43% in 16-bit numbers. With those figures, our script estimates that 3.8×10253.8 \times 10^{25} FLOPs on 16,384 H100s would take 63 to 71 days of pure training. That is our estimate, not a number Meta published.

Splitting the work

A 405B model does not fit on one GPU, and one GPU would take centuries anyway. So the work is split in several ways at once.

Three ways to split one training run across many GPUsData parallelsame model, different dataGPU 0: full modelbatch slice 0GPU 1: full modelbatch slice 1GPU 2: full modelbatch slice 2all-reduce: average gradientsTensor parallelsplit each weight matrixGPU 0GPU 1GPU 2GPU 3one 4096 × 14336 matrix→ four 4096 × 3584 slicesexchange after each layerPipeline parallelsplit the layers into stagesGPU 01234GPU 11234GPU 21234GPU 31234blank slots = idle time,the "bubble": 3/7 hereLarge runs use all three at once, plus expert parallelism for MoE layers and ZeRO / FSDP to shard optimizer states.
With 4 stages and 4 micro-batches the pipeline is idle 3/7 of the time; with 32 micro-batches only 3/35 (8.6%). Numbers from our script.

Tensor parallelism comes from NVIDIA's Megatron-LM paper of 2019. It needs fast links, because GPUs must swap partial results inside every layer. So it usually stays within one server, where 8 GPUs are joined by NVLink.

Take Llama 3 8B's feed-forward matrix of 4,096 × 14,336. Four GPUs would each hold a 4,096 × 3,584 slice; eight would each hold 4,096 × 1,792. Pipeline parallelism has a different cost: at the start and end of each batch, some stages sit idle while they wait. Cut the batch into mm micro-batches that flow through pp stages, and the idle share is

bubble=p−1m+p−1\text{bubble} = \frac{p - 1}{m + p - 1}

With 4 stages and 4 micro-batches, 3/7=0.4293/7 = 0.429 of the time is wasted. With 32 micro-batches it falls to 3/35=0.0863/35 = 0.086. With 16 stages and 64 micro-batches it is 0.190. More micro-batches shrink the bubble but need more memory. MoE models add a fourth kind, expert parallelism: different experts on different GPUs, with tokens sent to wherever their experts live. DeepSeek-V3 spread each layer's routed experts over 64 GPUs.

The inference series shows the same splits used for serving.

Memory: sharding the optimizer

Training needs far more memory than the weights alone. In mixed-precision training with the Adam optimizer, each parameter needs 2 bytes for its 16-bit weight, and 2 for its 16-bit gradient. It also needs 12 bytes of 32-bit optimizer state: a master copy of the weight and Adam's two running averages. That is 16 bytes per parameter.

For a 70B model, 16 bytes per parameter is 1,120 GB: the full memory of 14 GPUs with 80 GB each, before any activations. For 405B it is 6,480 GB. The ZeRO paper of 2019, from Samyam Rajbhandari and colleagues at Microsoft, gave the fix. Its example is a 7.5B model, which needs 120 GB of states. Sharded over 64 GPUs, each holds only 1.875 GB.

Fewer bits per number

Halving the bits halves the memory traffic and roughly doubles the speed of the matrix units. In 2017, Paulius Micikevicius and colleagues at NVIDIA and Baidu showed that 16-bit training works. You keep a 32-bit master copy and scale the loss. The bf16 format, with its wide range, made the scaling unnecessary and became the default.

DeepSeek-V3 was the first very large open model trained mostly in FP8. Its report says it was the first to "validate its effectiveness on an extremely large-scale model". The trick is to scale small blocks of numbers separately, so one large value does not ruin the precision of its neighbours. The most sensitive parts, such as the embedding, the output head and the normalisation, stay in higher precision.

The optimizer and the schedule

Almost every large model is trained with AdamW, from Ilya Loshchilov and Frank Hutter (2017). It is like a hiker who keeps a running average of the slope and of how bumpy the slope has been. Steps are large where the ground is steady and small where it is noisy.

mt=β1mt−1+(1−β1)gt,vt=β2vt−1+(1−β2)gt2,wt=wt−1−η(m^tv^t+ϵ+λwt−1)m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2, \quad w_t = w_{t-1} - \eta \left( \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} + \lambda w_{t-1} \right)

gtg_t is the gradient at step tt. mtm_t is the running average of gradients and vtv_t of squared gradients, with decay rates β1\beta_1 and β2\beta_2. m^t\hat m_t and v^t\hat v_t are the same averages, corrected for starting at zero: divide by 1−β1t1 - \beta_1^t and 1−β2t1 - \beta_2^t. η\eta (eta) is the learning rate and λ\lambda (lambda) the weight decay, which pulls every weight a little toward 0.

Take one weight w=1.0w = 1.0 with gradient g=0.5g = 0.5, η=0.001\eta = 0.001, β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95 and λ=0.1\lambda = 0.1. Then m=0.05m = 0.05, v=0.0125v = 0.0125, m^=0.5\hat m = 0.5 and v^=0.25\hat v = 0.25. The Adam step is 0.001×0.5/0.5=0.0010.001 \times 0.5 / 0.5 = 0.001, the decay adds 0.0001, and the new weight is 0.9989. With g=50g = 50, the step is still 0.001. Adam's first step has the size of the learning rate whatever the gradient's size, which is part of why it is stable.

DeepSeek-V3 used β1=0.9\beta_1 = 0.9, β2=0.95\beta_2 = 0.95 and weight decay 0.1, the same values as our example. The learning rate also changes during training. It starts at zero, rises in a straight line during a warm-up, and then falls.

η(t)=ηmin⁡+12(ηmax⁡−ηmin⁡)(1+cos⁡π(t−tw)T−tw)for t≥tw\eta(t) = \eta_{\min} + \tfrac{1}{2} (\eta_{\max} - \eta_{\min}) \left( 1 + \cos \frac{\pi (t - t_w)}{T - t_w} \right) \quad \text{for } t \ge t_w

tt is the step, twt_w the length of the warm-up and TT the last step. ηmax⁡\eta_{\max} is the peak rate and ηmin⁡\eta_{\min} the final one. The cosine falls from 1 to −1-1, so the rate glides from the peak to the floor, slowly at first, then faster, then slowly again.

Llama 3 405B used a peak of 8×10−58 \times 10^{-5}, 8,000 warm-up steps, and a cosine to 8×10−78 \times 10^{-7} over 1,200,000 steps. Our script assumes that the 1.2 million steps include the warm-up. Then the rate is 4×10−54 \times 10^{-5} at step 4,000 and 4.04×10−54.04 \times 10^{-5} half-way through, at step 604,000.

Learning rate: linear warm-up, then cosine decay (Llama 3 405B settings)02e-54e-56e-58e-50300K600K900K1,200Kpeak 8e-5 after 8,000 warm-up stepshalf-way: 4.04e-5end: 8e-7training step
The warm-up is too short to see at this scale: 8,000 of 1,200,000 steps. Llama 3 then annealed the rate to 0 over the last 40 million tokens. Curve from our script.

Other shapes are common. DeepSeek-V3 held its peak rate flat until 10 trillion tokens and decayed it along a cosine over the next 4.3 trillion. It ended with two short flat steps. Chinchilla's lesson about schedules still holds: the decay must match the planned length of the run.

When training breaks

Large runs are fragile. Google's PaLM paper of 2022 is unusually frank about it. For its 540B model, the team "observed spikes in the loss roughly 20 times during training", although gradient clipping was on. Their fix was to restart "from a checkpoint roughly 100 steps before the spike started" and skip "roughly 200–500 data batches". The same batches, given to an earlier checkpoint, caused no spike. So the cause was the data and the model's state together, not bad data alone.

Since then, labs have added tricks such as extra normalisation inside attention and limits on the size of attention scores. DeepSeek-V3 reports that "we did not experience any irrecoverable loss spikes or perform any rollbacks". Kimi K2 used a new optimizer, MuonClip, and reports pre-training on 15.5 trillion tokens "with zero loss spike".

Hardware fails too. In one 54-day stretch of Llama 3 training, there were 466 interruptions. Of these, 419 were unexpected, and about 78% of those were traced to confirmed or suspected hardware problems. Google's Gemini 2.5 report describes the same problem at TPU scale.

What it costs

The most quoted cost in recent years comes from DeepSeek-V3, and it is often quoted wrongly.

Our script adds one more check. With C=6×37×109×14.8×1012=3.29×1024C = 6 \times 37 \times 10^9 \times 14.8 \times 10^{12} = 3.29 \times 10^{24} FLOPs over 2,664K GPU hours, each GPU did about 343 trillion useful FLOPs per second. That leaves out attention, and it is our estimate, not DeepSeek's. Closed labs do not publish their training costs at all.

Predicting more than one token

DeepSeek-V3 also trained with multi-token prediction. Besides the next token, a small extra module predicts the token after it.

Fabian Gloeckle and colleagues at Meta studied the idea in 2024. Their models trained with 4-token prediction were "up to 3× faster at inference". DeepSeek-V3 used a depth of 1, one extra token. It reports that the extra prediction is accepted 85% to 90% of the time, which gives about 1.8 times more tokens per second. The next section shows where that number comes from.

Built for serving

A model is trained once and served for months to millions of people, so its design is shaped by serving costs. GQA, MLA and MoE are all, in part, serving decisions. Three more tricks matter.

Speculative decoding

Writing one token at a time leaves a GPU mostly idle, because each step reads all the weights to make one token. Checking five guessed tokens costs about the same as making one. In 2022, Yaniv Leviathan and colleagues at Google showed a 2X-3X speed-up on T5-XXL "with identical outputs". If each guess is accepted with probability aa and the draft guesses γ\gamma tokens, each pass of the large model yields on average

E[tokens per pass]=1−aγ+11−a\mathbb{E}[\text{tokens per pass}] = \frac{1 - a^{\gamma + 1}}{1 - a}

aa is the acceptance rate and γ\gamma (gamma) is the draft length. The formula counts the accepted guesses plus the one token the large model always adds. With a=0.8a = 0.8 and γ=4\gamma = 4, that is 3.362 tokens per pass; with a=0.6a = 0.6, 2.306.

DeepSeek-V3 uses its MTP module as the draft, with γ=1\gamma = 1. At a=0.85a = 0.85 the formula gives 1.850 tokens per pass, and at 0.9 it gives 1.900. That agrees with the 1.8 times speed-up the report gives. The inference series covers the method in depth.

Quantisation

Part 1 introduced quantisation: storing weights in fewer bits. A 70B model needs 140 GB at 16 bits, 70 GB at 8 bits and 35 GB at 4 bits. DeepSeek-V3's 671B parameters take about 671 GB in FP8, the format it was trained in. Many open models are now released in 8-bit or 4-bit versions, and serving systems run them with special matrix kernels.

Distillation into small models

Most users talk to small models. Labs make them by distillation, defined in Part 4: a small student learns from a big teacher's outputs. In pre-training, the student learns from the teacher's full probability over the next token, not only from the one token in the text. That is a far richer signal: it says that "cat" was likely, "dog" was nearly as likely, and "car" was not.

Google's open Gemma models follow the same path: the Gemma 3 report says that "all Gemma 3 models are trained with knowledge distillation". Meta used the 405B model to improve its smaller Llama 3 models in post-training, and Qwen3 describes "strong-to-weak distillation". The big model's main job in a family is often to teach the small ones.

What labs publish, and what they do not

Everything in this part comes from papers, technical reports and model cards. They differ widely in how much they say.

The open-weight labs publish the most: DeepSeek, Meta's Llama papers, Qwen, Kimi and above all the Allen Institute's Olmo, which also releases its data. Google's Gemini reports confirm the design in outline: sparse MoE, natively multimodal, long context, TPUs, distillation for the small models. OpenAI's GPT-4 report of 2023 set the closed pattern. It "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar."

QuestionOpen-weight reports (DeepSeek, Llama, Olmo)Closed frontier labs
Layer designFull: norms, attention type, sizesGoogle: "sparse MoE", little more; others: nothing
Parameter countExact, total and activeNot published
Training tokensExact or closeNot published
Data sourcesDescribed; Olmo releases the dataCategories only ("web documents, code, images")
HardwareChip type and countGoogle names TPU generations; others: little
CostDeepSeek: GPU hours for the final runNot published

So, when you read a claim about a closed model's size or cost, ask where it comes from. If it is not in a lab's own report or model card, it is a guess or a leak, however often it is repeated.

How the five parts fit together

Put the parts of this series side by side and you get the whole build of a frontier model in 2026.

How a frontier model is built, end to endDatacrawl, filter, dedup,mix, tokenisePre-trainingtrillions of tokens,next-token lossMid-trainingannealing onhigh-quality dataLong contextRoPE scaling,8K → 128K → 1MPost-trainingSFT, preferences,reasoning RL (Part 4)Distillationsmall models learnfrom the big oneServingquantise, KV cache,speculative decodingModel cardwhat the labchooses to publishUnder it all: thousands of GPUs or TPUs, parallelism, bf16 or FP8, checkpoints and restartsParts 1 and 5 cover the first four boxes, Part 4 the post-training box. Labs differ in order and detail.
A composite of the Llama 3, DeepSeek-V3, Gemini 2.5 and Olmo 3 reports. Closed labs confirm only parts of it.

Part 1 gave the parts list: the neuron, backpropagation, attention, the Transformer, the next-token loss. This part showed what they look like at scale. Part 2's classical methods live on inside the pipeline, as the small fast classifiers that filter data. Part 3 and Part 4 take over once the base model exists, and turn it into an assistant that reasons and uses tools.

The lessons repeat across all five parts. A good loss is not the same as a good model. Goodhart's law in Part 4 and the warning about loss in this part both show it. Scale wins, but only when the bottleneck moves: from compute to data, from data to memory, from memory to the cost of every answer. And most progress is a chain of fixes, each solving the limit the step before it left behind.

What is still open

Data is the hardest limit. Good public text is close to used up, so the future depends on synthetic data, other kinds of data and better reuse. Whether a model can learn as much from text written by models as from text written by people is still an open question.

Architecture is less settled than it looks. Many 2025 and 2026 models mix in other layer types, such as linear attention, state-space layers and sparse attention, to make long contexts cheaper. The attention series covers them. DeepSeek-V3.2 added its own sparse attention. The SwiGLU paper's "divine benevolence" is still a fair summary of how many of these choices are made.

And scaling itself is changing shape. Pre-training compute keeps growing, but more of the total now goes to post-training and to thinking at answer time, as Part 4 showed. The next part of the curve may be drawn on that second axis.

StepWhat it fixedWhat it left open
Next-token loss, perplexityOne clear goal for raw textLoss is not ability
Kaplan scaling laws (2020)Predictable gains from small runsWrong split between size and data
Chinchilla (2022)Compute-optimal size and dataIgnores the cost of serving
Over-training (Llama 3, 2024)Small models that are cheap to serveNeeds ever more data
Filtering, dedup, quality classifiersBetter data from the same webHard to test; can tilt the data
Byte-level BPEAny text in a fixed vocabularyUneven cost across languages
Pre-norm, RMSNorm, SwiGLUStable, cheaper deep stacksFound by trial, not theory
RoPE and its scalingRelative positions; longer contextsQuality falls far past the training length
FlashAttentionExact attention without the n2n^2 memoryStill n2n^2 arithmetic
MQA, GQA, MLAA much smaller KV cacheExtra design and tuning
Mixture of expertsMore knowledge per FLOPLoad balance, memory, communication
Parallelism, ZeRO, FP8Training across thousands of chipsFailures, loss spikes, cost
Speculative decoding, quantisation, distillationCheaper, faster servingSmall models trail the big ones

Key papers: Kaplan et al. 2020, Chinchilla 2022, Llama 3 2024. Data and architecture: FineWeb 2024, BPE 2016, RoFormer 2021, FlashAttention 2022. Experts and scale: Switch Transformer 2021, DeepSeek-V3 2024, Gemini 2.5 2025.

Back to Part 4, From RLHF to Reasoning, or to the start of the series, Part 1, From One Neuron to Opus.