BERT, explained · Background · Covers From n-grams (1948) to UltraBERT (2026)

Before and After BERT

How machines learned to read: from counting word pairs to word vectors, recurrent networks, attention and pre-training, then BERT and everything after it, up to 2026. With the maths of each step and small runs you can check.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. NAACL 2019, 2018. arXiv:1810.04805

BERT did not come from nowhere. It joined three ideas that took decades to grow: words as vectors, attention, and pre-training on plain text. After BERT, the field split into new branches. One of them grew into the chat models of today, and BERT's own branch is still growing in 2026.

This page tells that story in order, with the key equation of each step and a small run you can check. Read it before Part 1 if you want the background, or after Part 6 to see where BERT stands.

From counting words to BERT, and after1948n-grams (Shannon)guess the next letter or word from counts1997LSTMa recurrent network with gates that keep a memory2003Neural language modellearn a vector for each word while predicting2013word2veccheap word vectors from 1.6 billion words2014seq2seqread a sentence into one vector, write another2014Attentionlook back at every input word, not one vector2017Transformerattention only, no recurrence2018ULMFiT, ELMo, GPTpre-train a language model, then reuse it2018GLUEone score over nine understanding tasks2018BERTdeep bidirectional pre-training: fill in the blanks2019GPT-2, XLNet, RoBERTa, ALBERTbigger, trained longer, new objectives2019SuperGLUE, T5harder tests; every task as text-to-text2020Scaling laws, GPT-3loss falls smoothly with size; tasks from prompts2021DeBERTafirst single model above humans on SuperGLUE2024ModernBERTan encoder for search: 8192 tokens, 2 trillion tokens2025NeoBERT, mmBERTdeeper encoders; 1800 languages in one model2025LLaDAfill-in-the-blanks becomes a way to write text2025Ettinencoders and decoders trained the same way, compared2026UltraBERTtwo masking ratios in one encoderbefore BERTBERTafter BERT
The road to BERT and beyond. Blue steps came before BERT; orange steps came after it. Each step fixed a weakness of the one before.

The one question every model answers

Every model on this page answers the same question in some form: given some words, which word fits here? A model that answers it well has learned a lot about language. To guess the word after "The capital of France is", it must know some geography. To guess the word after "She dropped the glass and it", it must know some physics.

The question can be asked in two ways, and the difference matters for BERT. A left-to-right model sees only the words before the gap and guesses the next word. A fill-in-the-blank model sees the words on both sides of the gap. BERT is the second kind. Most of the models before it, and most of the famous ones after it, are the first kind.

Before BERT

Step 1: counting words (n-grams)

The oldest method is to count. In 1948, the mathematician Shannon showed how to imitate English text. He picked each letter or word from counts of what followed the previous ones. This became the n-gram model, the standard tool of speech recognition and translation for decades.

The idea is simple. To guess the word after "the", look at every "the" in a large text and count the words that follow it. If "first" follows "the" 1.2% of the time, then the model says P(first∣the)=0.012P(\text{first} \mid \text{the}) = 0.012.

P(wt∣wt−1)=count(wt−1,wt)count(wt−1)P(w_t \mid w_{t-1}) = \frac{\text{count}(w_{t-1}, w_t)}{\text{count}(w_{t-1})}

Read it this way. The left side, P(wt∣wt−1)P(w_t \mid w_{t-1}), is the probability of word wtw_t given the word just before it. The top of the fraction counts how often the pair appears. The bottom counts how often the first word appears at all. This is a bigram model: it looks back one word. A trigram looks back two words, and an n-gram looks back n−1n - 1.

A tiny example. Suppose the text is "the cat sat. the cat ran. the dog sat." The word "the" appears 3 times. It is followed by "cat" twice and by "dog" once. So P(cat∣the)=2/3P(\text{cat} \mid \text{the}) = 2/3 and P(dog∣the)=1/3P(\text{dog} \mid \text{the}) = 1/3.

We measure how good a language model is with perplexity:

perplexity=exp⁡(−1N∑t=1Nlog⁡P(wt∣wt−1))\text{perplexity} = \exp\left(-\frac{1}{N} \sum_{t=1}^{N} \log P(w_t \mid w_{t-1})\right)

The sum adds the log probability that the model gave each real word of a test text, and NN is the number of words. A perplexity of 100 means that the model is, on average, as unsure as if it had to choose among 100 equally likely words. Lower is better.

We built a bigram model of WikiText-2, a collection of Wikipedia articles of about 1.7 million words. We counted on the training part and scored on the test part. A pair never seen in training would get probability 0 and an infinite perplexity, so a model needs smoothing. We tried two kinds:

python
def p_bigram(prev, w):
    """Add-one smoothing: every pair gets one extra imaginary count, so no probability is zero."""
    return (bi[(prev, w)] + 1) / (uni[prev] + V)


def p_interp(prev, w, lam=0.7):
    """Interpolation: trust the pair count with weight lam, fall back on the single-word count otherwise."""
    ml = bi[(prev, w)] / uni[prev] if uni[prev] else 0.0
    return lam * ml + (1 - lam) * p_unigram(w)
plain text
== 1. A bigram model of WikiText-2: P(w_t | w_{t-1}) = count(w_{t-1}, w_t) / count(w_{t-1})
training text: 1,694,385 words, 61,887 different words, 667,984 different word pairs
test text: 198,980 words
perplexity, unigram (no context): 2,079
perplexity, bigram with add-one smoothing: 8,342
perplexity, bigram mixed with unigram (0.7 / 0.3): 1,059
test word pairs never seen in training: 36.3%
check: the probabilities after "the" add up to 1.000000
Perplexity on WikiText-2 test (lower is better), our rununigramunigram: 2,0792,079bigram, add-onebigram, add-one: 8,3428,342bigram, mixedbigram, mixed: 1,0591,05936.3% of the word pairs in the test text never appear in training.
Our run on WikiText-2. One word of context halves the perplexity, but only with good smoothing. Add-one smoothing gives too much probability to pairs that never occur, and does worse than no context at all.

The result shows both the power and the weakness of counting. With one word of context and good smoothing, the perplexity falls from 2,079 to 1,059. With naive smoothing it gets worse, 8,342, because add-one gives too much probability to the 61,887 words that could follow each word.

The deeper problem is the last useful line: 36.3% of the word pairs in the test text never appear in the 1.7 million training words. Counting cannot generalise. The pair "cat sat" teaches it nothing about "dog sat", because to a counter, "cat" and "dog" are just different symbols. Longer contexts make this worse, since most triples and quadruples are never seen at all.

Step 2: words as vectors (word2vec)

The fix is to give each word a list of numbers, a vector, so that similar words get similar vectors. Then "dog sat" can borrow from "cat sat", because "dog" and "cat" are close. Yoshua Bengio and colleagues showed in 2003 that a neural network can learn such vectors while it learns to predict the next word. It worked, but it was slow.

The second word2vec paper gave the training objective that most people use, skip-gram with negative sampling. The skip-gram idea: for each word in the text, predict the words around it.

Here is the objective for one centre word wIw_I and one real neighbour wOw_O, written as a loss to make small:

L=−log⁡σ(uwO⊤vwI)−∑i=1klog⁡σ(−uwi⊤vwI)\mathcal{L} = -\log \sigma\left(u_{w_O}^\top v_{w_I}\right) - \sum_{i=1}^{k} \log \sigma\left(-u_{w_i}^\top v_{w_I}\right)

Read it this way. vwIv_{w_I} is the vector of the centre word, and uwOu_{w_O} is the vector of the real neighbour. Each word has two vectors: one for when it is the centre, one for when it is a neighbour. The dot product u⊤vu^\top v is large when the two vectors point the same way. The sigmoid σ(z)=1/(1+e−z)\sigma(z) = 1 / (1 + e^{-z}) turns it into a number between 0 and 1. The words w1,…,wkw_1, \ldots, w_k are kk random noise words.

A tiny example. If the real pair has dot product 2, then σ(2)=0.88\sigma(2) = 0.88 and −log⁡0.88=0.13-\log 0.88 = 0.13, a small loss. If one noise word also has dot product 2, its term is −log⁡σ(−2)=−log⁡0.12=2.13-\log \sigma(-2) = -\log 0.12 = 2.13, a large loss. Training pushes the real pair together and the noise pair apart.

We trained skip-gram with negative sampling from scratch on WikiText-2 in PyTorch. Method 1 is the loss written as above. Method 2 asks PyTorch's binary cross-entropy the same yes-or-no question. Both must agree:

python
def sgns_loss_by_hand(c, o, k):
    """Method 1, the paper's objective: -log s(u_o . v_c) - sum over k noise words of log s(-u_k . v_c)."""
    vc = v_in(c)                                            # (B, DIM)
    pos = (u_out(o) * vc).sum(-1)                           # (B,)   score of the real context word
    neg = torch.bmm(u_out(k), vc[:, :, None])[..., 0]       # (B, K) scores of the noise words
    return -(F.logsigmoid(pos) + F.logsigmoid(-neg).sum(-1)).mean()


def sgns_loss_library(c, o, k):
    """Method 2: the same as binary classification, real pair = 1, noise pair = 0, with PyTorch's BCE."""
    vc = v_in(c)
    logits = torch.cat([(u_out(o) * vc).sum(-1, keepdim=True), torch.bmm(u_out(k), vc[:, :, None])[..., 0]], 1)
    labels = torch.cat([torch.ones_like(logits[:, :1]), torch.zeros_like(logits[:, 1:])], 1)
    return F.binary_cross_entropy_with_logits(logits, labels, reduction='none').sum(-1).mean()
plain text
== 2. word2vec (skip-gram, negative sampling) trained from scratch on WikiText-2
826,181 training words after subsampling (t = 0.0001), vocabulary of 19,963 words (seen at least 5 times); 100 dimensions, window 2, 5 negatives, 5 epochs
3,304,718 (centre, context) pairs per epoch
check on one batch: loss by hand 2.621636, library 2.621636
epoch 1: mean loss 2.7942
epoch 5: mean loss 2.0632
nearest to 'king': ragnaill 0.75, narasimha 0.75, brienne 0.73, gall 0.72, murchad 0.72, ashurbanipal 0.72
nearest to 'france': italy 0.72, spain 0.71, netherlands 0.70, austria 0.69, ukraine 0.68, russia 0.67
nearest to 'river': missouri 0.80, tributary 0.80, confluence 0.78, tributaries 0.78, pongola 0.77, platte 0.77
nearest to 'bank': waterway 0.70, loyalsock 0.68, plunketts 0.66, susquehanna 0.66, barbours 0.66, logging 0.66
king - man + woman: closest words vira, narasimha, eldest, brienne, regency; "queen" is number 445

The two losses agree to six decimal places. The neighbours show what word vectors learn. "France" sits among other European countries, and "river" among tributaries and confluences. "King" sits among the names of kings from the Wikipedia articles, which is not wrong: the model groups words that occur in the same places.

The famous analogy fails here: "queen" comes 445th. Our text is under 2 million words; the paper's Google News text had 6 billion. Vector arithmetic needs a lot of reading before the directions line up. Keep this in mind, because it is exactly why pre-training on huge text became the rule.

Now look at "bank". Its neighbours are all about rivers: waterway, Susquehanna, Loyalsock Creek. In these Wikipedia articles, banks are mostly river banks. A money bank in a new sentence would get the same vector, because word2vec has one vector per word. That is the limit that ELMo and BERT remove.

Step 3: reading in order (RNNs and LSTMs)

Word vectors give each word a meaning, but a sentence is more than a bag of words. "Dog bites man" and "man bites dog" use the same words. A recurrent neural network, or RNN, reads the words in order and keeps a running summary, its hidden state:

ht=tanh⁡(Wht−1+Uxt)h_t = \tanh\left(W h_{t-1} + U x_t\right)

Read it this way. xtx_t is the vector of word tt. ht−1h_{t-1} is the summary of everything before it. The matrices WW and UU mix the old summary with the new word, and tanh⁡\tanh squeezes each number into the range −1 to 1. The new summary hth_t is passed on to the next word. One network, used again at every step, can read a sentence of any length.

The trouble is training. To learn that the first word matters for the last, the error signal must travel back through every step. At each step it is multiplied by WW and by the slope of tanh⁡\tanh. If those multiply to less than 1, the signal shrinks at every step and soon vanishes. This is the vanishing gradient problem.

A tiny example. If each step multiplies the signal by 0.5, then after 10 steps it is 0.510≈0.0010.5^{10} \approx 0.001 of its size. After 20 steps it is about one millionth.

We measured it. We built a plain RNN with 32 numbers in its hidden state and ran it over 60 random inputs. We then asked how much the last state changes when each input changes. Method 1 walks back step by step and multiplies the slopes by hand. Method 2 asks PyTorch's autograd:

python
# Method 1: walk back through the steps. d loss / d h_t = J_{t+1}^T ... J_T^T target, with J_t = diag(1 - h_t^2) W
g, by_hand = target.clone(), [None] * T
for t in range(T - 1, -1, -1):
    pre = (1 - hs[t] ** 2) * g                              # through the tanh of step t
    by_hand[t] = (Uin.T @ pre).norm().item()               # gradient that reaches input x_t
    g = W.T @ pre                                          # pass on to h_{t-1}
# Method 2: autograd
loss.backward()
auto = xs.grad.norm(dim=1).tolist()
plain text
== 3. A plain RNN, h_t = tanh(W h_{t-1} + U x_t): how much does the last state feel each earlier input?
hidden size 32, 60 steps, largest stretch of W = 0.9
 distance from the end  gradient size (by hand)   autograd
                     0                 3.04e+00   3.04e+00
                     1                 1.18e+00   1.18e+00
                     2                 4.61e-01   4.61e-01
                     5                 1.40e-02   1.40e-02
                    10                 4.59e-05   4.59e-05
                    20                 2.56e-09   2.56e-09
                    40                 7.10e-19   7.10e-19
                    59                 2.43e-28   2.43e-28
largest relative difference between the two methods: 0.0e+00
How much of the gradient reaches each earlier word (plain RNN, our run)11e-51e-101e-151e-201e-250102030405059distance from the last word (steps back)
Our run. The signal that reaches each earlier word falls by a nearly constant factor per step. Ten words back it is 66,000 times weaker than at the last word; fifty-nine words back it is gone.

Both methods agree exactly. Ten words back, the signal is 66,000 times weaker than at the last word. Twenty words back it is a billion times weaker. A plain RNN cannot learn that a word at the start of a paragraph matters for a word at the end.

The LSTM (long short-term memory), published by Hochreiter and Schmidhuber in 1997, fixed much of this. It adds a separate memory cell and gates that decide what to keep, what to forget and what to output. Because the cell is updated by addition rather than repeated multiplication, a signal can pass through many steps without fading. By 2014, LSTMs were the standard tool for text.

Step 4: from one vector to attention (seq2seq)

In 2014 the LSTM was used for translation, and the way it was used set up the next big idea.

ci=∑j=1Txαijhj,αij=exp⁡(eij)∑k=1Txexp⁡(eik)c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j, \qquad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}

A tiny example. Three input words have scores 2, 0 and 0 for the current output word. Then e2=7.39e^2 = 7.39 and e0=1e^0 = 1, so the weights are 7.39/9.39=0.797.39 / 9.39 = 0.79, then 0.110.11 and 0.110.11. The context vector is 79% the first word's state. Change the output word, and the scores and weights change with it.

Step 5: attention only (the Transformer)

By 2017, every good translation model was an RNN with attention on top. The RNN still read the words one at a time, so training could not use the full parallel power of a GPU.

Attention(Q,K,V)=softmax(QK⊤dk)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V

This is Bahdanau's weighted sum again, done for all words at once with matrices. QQ, KK and VV hold the queries, keys and values of all words as rows, and dkd_k is the length of a key. Dividing by dk\sqrt{d_k} keeps the scores in a range where the softmax still learns. The Transformer series works through it in full; here we need only the idea that every word reads every other word in one step.

The Transformer had two halves, an encoder and a decoder, for translation. The next year, two groups took one half each. GPT took the decoder. BERT took the encoder.

Step 6: pre-training arrives (2018)

All the models so far were trained for one task at a time, from scratch or from word vectors. In 2018, three papers showed a better way: train a language model on lots of plain text first, then reuse it.

ELMoktask=γtask∑j=0Lsjtask hk,jLM\text{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^{L} s_j^{task} \, h_{k,j}^{LM}

Read it this way. kk is the position of the token. hk,jLMh_{k,j}^{LM} is the state of token kk in layer jj of the two-way language model; layer 0 is the plain word vector. The weights sjs_j come from a softmax, so they add up to 1. γ\gamma scales the whole vector. With two LSTM layers, L=2L = 2, and a task might learn weights such as 0.2, 0.3 and 0.5 for layers 0, 1 and 2.

GPT's fine-tuning loss adds the language model loss as a helper, with a weight λ\lambda:

L3(C)=L2(C)+λ⋅L1(C)L_3(\mathcal{C}) = L_2(\mathcal{C}) + \lambda \cdot L_1(\mathcal{C})

L2L_2 is the loss of the task, for example the log probability of the right label. L1L_1 is the next-word loss on the same task text. Keeping a little of the old job while learning the new one helps the model not to forget what it learned in pre-training.

To compare all these models, the field needed one test. It arrived in the same year.

BERT: the meeting point

In October 2018, BERT joined the pieces. From word2vec it took the idea that vectors learned from plain text carry meaning. From ELMo it took vectors that depend on the whole sentence. From GPT it took the Transformer and pre-training with fine-tuning. Its own new step was to train a Transformer encoder that sees both sides of every word at once, in every layer.

A left-to-right model cannot simply look right too: each word would then see the very word it must predict. BERT's fix is the masked language model. It hides 15% of the tokens and asks the model to fill them in:

LMLM=−∑i∈Mlog⁡P(xi∣x∖M)\mathcal{L}_{\text{MLM}} = -\sum_{i \in M} \log P\left(x_i \mid x_{\setminus M}\right)

Read it this way. MM is the set of masked positions. x∖Mx_{\setminus M} is the sentence with those positions hidden. For each hidden position ii, the model gives a probability to the real token xix_i, using every visible token on both sides. The loss is the sum of the negative log probabilities. Part 3 works through it on the real model, including the 80/10/10 rule for what replaces a masked token.

The result was a jump. On the GLUE leaderboard, GPT had a score of 72.8. BERT-large reached 80.5, a gain of 7.7 points in four months.

Does BERT really give each use of a word its own vector? We ran the released BERT-base on four sentences with "bank". In the input table, "bank" has one vector, as in word2vec. After the twelve layers, each "bank" has its own:

plain text
== 4. One vector per word (word2vec, BERT input table) against one vector per use (BERT last layer)
word2vec: "bank" has one vector for every sentence; its neighbours are waterway, loyalsock, plunketts, susquehanna
BERT input table: cosine of "bank" in sentence 1 and sentence 2: 1.000
BERT last layer, cosine similarity of "bank" between sentences:
  'he sat on the bank of the river.'           vs 'she put her money in the bank.'             0.485
  'he sat on the bank of the river.'           vs 'the bank approved my loan.'                 0.400
  'he sat on the bank of the river.'           vs 'we walked along the bank of the stream.'    0.899
  'she put her money in the bank.'             vs 'the bank approved my loan.'                 0.787
  'she put her money in the bank.'             vs 'we walked along the bank of the stream.'    0.487
  'the bank approved my loan.'                 vs 'we walked along the bank of the stream.'    0.431
"bank" in BERT's last layer: cosine similaritybank of the riverbank of the river → river: 1.001.00bank of the river → money: 0.490.49bank of the river → loan: 0.400.40bank of the river → stream: 0.900.90money in the bankmoney in the bank → river: 0.490.49money in the bank → money: 1.001.00money in the bank → loan: 0.790.79money in the bank → stream: 0.490.49bank approved my loanbank approved my loan → river: 0.400.40bank approved my loan → money: 0.790.79bank approved my loan → loan: 1.001.00bank approved my loan → stream: 0.430.43bank of the streambank of the stream → river: 0.900.90bank of the stream → money: 0.490.49bank of the stream → loan: 0.430.43bank of the stream → stream: 1.001.00rivermoneyloanstreamIn the input table (and in word2vec) all four cells are 1.000: one vector for every use.
Our run with BERT-base. The two river banks are close (0.899) and the two money banks are close (0.787). Across the two meanings the similarity drops to 0.40 to 0.49.

The river bank and the stream bank have a similarity of 0.899, though the sentences share almost no other words. The two money banks have 0.787. Across the two meanings, the similarity falls to between 0.400 and 0.487. BERT sorted the four uses by meaning without being told that "bank" has two meanings. That is the problem of Step 2, solved.

Parts 1 to 6 read the BERT paper itself, line by line. The rest of this page jumps ahead to what happened next.

After BERT

The curve: scores that rose faster than anyone expected

We collected the scores of the main models from the papers themselves and drew them on one time line:

plain text
== 6. The progress curves (scores copied from the papers named)
GLUE (human baseline 87.1):
  2018-04  BiLSTM + ELMo + Attn    70.0   GLUE paper, Table 4 (multi-task)
  2018-06  OpenAI GPT              72.8   BERT paper, §4.1
  2018-10  BERT-large              80.5   BERT paper, §4.1
  2019-06  XLNet                   88.4   ALBERT paper, Table 9
  2019-07  RoBERTa                 88.5   ALBERT paper, Table 9
  2019-09  ALBERT                  89.4   ALBERT paper, Table 9
  2019-10  T5-11B                  90.3   T5 paper, Table 14
SuperGLUE (human baseline 89.8):
  2019-05  BERT-large              69.0   SuperGLUE paper, Table 3
  2019-10  T5-11B                  89.3   DeBERTa paper, §1
  2021-01  DeBERTa-1.5B            89.9   DeBERTa paper, §1 (leaderboard, January 2021)
Benchmark scores over time (numbers from the papers)GLUE: nine tasks (2018)708090human baseline 87.1BiLSTM + ELMo + Attn: 70.0 (GLUE paper, Table 4 (multi-task))BiLSTM + ELMo + Attn 70.0OpenAI GPT: 72.8 (BERT paper, §4.1)OpenAI GPT 72.8BERT-large: 80.5 (BERT paper, §4.1)BERT-large 80.5XLNet: 88.4 (ALBERT paper, Table 9)RoBERTa: 88.5 (ALBERT paper, Table 9)ALBERT: 89.4 (ALBERT paper, Table 9)T5-11B: 90.3 (T5 paper, Table 14)XLNet 88.4 (Jun 2019)RoBERTa 88.5 (Jul 2019)ALBERT 89.4 (Sep 2019)T5-11B 90.3 (Oct 2019)the four points at the top, left to right:SuperGLUE: eight harder tasks (2019)708090human baseline 89.8BERT-large: 69.0 (SuperGLUE paper, Table 3)BERT-large 69.0T5-11B: 89.3 (DeBERTa paper, §1)T5-11B 89.3DeBERTa-1.5B: 89.9 (DeBERTa paper, §1 (leaderboard, January 2021))DeBERTa-1.5B 89.9201920202021
The learning curve of the field. GLUE (top) climbed from 70.0 to 90.3 in eighteen months and passed its human baseline in 2019. Its harder successor, SuperGLUE (bottom), climbed from BERT's 69.0 to 89.9 in under two years.

The curve has one shape twice. Each benchmark starts with a large gap below the human line, closes it in a year or two, and then has little room left. On GLUE, BERT closed more than half of the gap between GPT and the human baseline in one step. SuperGLUE was designed to be hard for BERT. Even so, models went from 69.0 to above its human level of 89.8 in less than two years. The dates are those of the papers or the leaderboard entries they report.

Two cautions. First, "human performance" here is the score of paid crowd workers with short training, not of experts. Second, passing the human line on a benchmark does not mean that a model understands language as people do. It means that the benchmark has stopped telling good models apart.

Three branches grow from one trunk

After BERT, work split into three branches, following the three ways to use a Transformer.

  1. Encoders, like BERT. They read the whole input at once and are best at understanding: classification, search, finding names in text. RoBERTa, ALBERT, ELECTRA and DeBERTa refine this branch. Part 6 covers each of them.
  2. Encoder-decoders, like the original Transformer. They read an input and write an output. T5 and BART belong here.
  3. Decoders, like GPT. They read left to right and write the next token. GPT-2 and GPT-3 grew this branch, and today's chat models come from it.

Bigger and bigger: the scaling curve

Model size, as the GPT-3 paper lists it (parameters, log scale)100M1B10B100B100MGPT2018300MBERT20181.5BGPT-220198BMegatron201911BT5201917BTuring-NLG2020175BGPT-32020
The size curve. On a log scale, each step up the grid is ten times more parameters. From GPT in 2018 to GPT-3 in 2020, the largest models grew 1,750 times.
L(N)=(NcN)αNL(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}

Read it this way. NN is the number of parameters, not counting the word vector tables. LL is the test loss in nats per token, the average negative log probability of the next token. NcN_c and αN\alpha_N are two constants fitted to the authors' measurements.

A tiny example. Make a model 10 times bigger. The new loss is (Nc/10N)0.076=L(N)×10−0.076=0.839 L(N)(N_c / 10N)^{0.076} = L(N) \times 10^{-0.076} = 0.839 \, L(N). So each tenfold step cuts the loss by about 16%, whatever the starting size. We applied the law at the sizes the GPT-3 paper lists:

plain text
== 5. Kaplan et al. (2020): L(N) = (N_c / N)^alpha_N, alpha_N = 0.076, N_c = 8.8e13 (test loss in nats per token)
GPT (2018)              0.1B parameters -> predicted loss 2.83
BERT-large (2018)       0.3B parameters -> predicted loss 2.60
GPT-2 (2019)            1.5B parameters -> predicted loss 2.30
T5 (2019)              11.0B parameters -> predicted loss 1.98
GPT-3 (2020)          175.0B parameters -> predicted loss 1.60
ten times the parameters multiplies the loss by 10^-0.076 = 0.839
(the law is fitted on non-embedding parameters of models trained on WebText2; the sizes above are total counts)

These are predictions for left-to-right models on the authors' web text, not measured losses of BERT or T5, which were trained differently. The point is the shape: from GPT to GPT-3, the law predicts the loss falls from 2.83 to 1.60. Because the gains were predictable, companies could plan and pay for very large training runs. That is how the decoder branch grew so fast.

Where BERT stands today

If decoders can do everything, why does anyone still train encoders? Because for many jobs, an encoder is far cheaper. Think of deciding whether an email is spam, or finding the ten documents closest to a question. You need a good vector for the text, not a written answer. An encoder gives that vector in one pass.

What ModernBERT changed

ModernBERT is not a new idea. It is BERT rebuilt with six years of lessons from the decoder world. Think of a classic car with a new engine, new brakes and new tyres. The body is the same, and it does the same job, but faster. Here is what changed, read from the two released models.

python
for k, n in NAMES.items():
    c = AutoConfig.from_pretrained(n)
    rows[k] = dict(layers=c.num_hidden_layers, hidden=c.hidden_size, ffn=c.intermediate_size,
                   heads=c.num_attention_heads, vocab=c.vocab_size, max_len=c.max_position_embeddings,
                   params=sum(p.numel() for p in mlm[k].parameters()) / 1e6)
plain text
== 7. BERT (2018) against ModernBERT (2024), read from the released models
                               BERT-base   ModernBERT-base
layers                                12                22
hidden size                          768               768
feed-forward size                  3,072             1,152
attention heads                       12                12
vocabulary                        30,522            50,368
longest input (tokens)               512             8,192
parameters (millions)              109.5             149.7
(ModernBERT uses GeGLU: its first feed-forward layer makes 2 x 1,152 = 2,304 numbers, one half a value, the other a gate)

The width is the same, 768 numbers per token, but ModernBERT is almost twice as deep: 22 layers against 12. The paper calls this "Deep & Narrow" and chose it because deeper models did better on tasks at the same size. The vocabulary grew from 30,522 to 50,368 pieces, because the new tokenizer also has to cut up code. The longest input grew sixteen times, from 512 to 8,192 tokens.

Two of the changes come straight from decoders. RoPE replaces BERT's table of learned position vectors, so the model is not tied to the positions it saw in training. GeGLU replaces the plain feed-forward layer. One half of its numbers is a value, and the other half is a gate that decides how much of each value passes.

plain text
== 8. ModernBERT: every third layer looks at the whole text, the others at a 128-token window
layers: 22; global layers (0-based): [0, 3, 6, 9, 12, 15, 18, 21] -> 8 global, 14 local; window 128 tokens
query-key pairs at 8,192 tokens: all global 1.476e+09, alternating 5.516e+08 -> 37.4% of the work

Eight of the 22 layers are global. At the full length, the attention does 37.4% of the work that 22 global layers would do. That count leaves out the feed-forward layers, so it is a share of the attention cost only. ModernBERT also drops padding tokens before the first layer ("unpadding"), so a batch of short and long texts wastes no work on empty places.

Does all this change what the model knows? Here is the same blank for both models:

plain text
BERT-base        "The capital of France is [MASK]." -> paris (0.42), lille (0.07), lyon (0.06), marseille (0.04), tours (0.03)
ModernBERT-base  "The capital of France is [MASK]." -> Paris (0.92), Lyon (0.04), Nancy (0.02), Nice (0.01), Orleans (0.00)
BERT-base        on a 2,000-token input: fails: RuntimeError
ModernBERT-base  on a 2,000-token input: runs

Both models know the answer, but ModernBERT is far surer: 0.92 against 0.42. It also keeps capital letters (BERT-base lower-cases everything). And it reads a 2,000-token text in one pass, where BERT stops with an error, because it has no position vector after place 512.

2025 and 2026: after ModernBERT

ModernBERT started a small revival. In the next two years, groups built new encoders, tested encoders against decoders fairly, and found a way to make BERT's own objective write text. One question ran through all of it: how much of the text should the model hide?

More encoders: NeoBERT and mmBERT

Encoder or decoder? A fair test

BERT learns to write: masked diffusion

BERT can fill blanks, but it cannot write a paragraph. Or can it? Start with a row of blanks, fill the one you are surest about, and repeat. Image models such as Stable Diffusion work in a similar way: they start from noise and remove a little of it at each step. For text, the "noise" is the [MASK] token.

We computed the loss two ways on BERT's own predictions, once by hand and once with PyTorch's cross-entropy, at three masking levels:

python
    m = inner[torch.rand(len(inner), generator=g) < t]     # forward process: each token masked with probability t
    xt = x0.clone()
    xt[m] = tok[k].mask_token_id
    with torch.no_grad():
        logits = mlm[k](input_ids=xt[None]).logits[0].double()   # float64, so the two methods can agree closely
    # Method 1: by hand. log-softmax of the true token at every masked place, summed, times 1/t
    logp = logits[m] - logits[m].logsumexp(-1, keepdim=True)
    hand = -(1 / t) * logp.gather(1, x0[m][:, None]).sum()
    # Method 2: the library. cross-entropy summed over masked places, then the same 1/t weight
    lib = (1 / t) * F.cross_entropy(logits[m], x0[m], reduction='sum')
plain text
== 9. LLaDA (2025): L = -E_t [ (1/t) * sum over masked tokens of log p(x_i | x_t) ], t ~ U(0, 1)
t = 0.15:  1 of 16 tokens masked; loss by hand 1.3920, library 1.3920; per masked token 0.209 nats
t = 0.50:  7 of 16 tokens masked; loss by hand 38.7060, library 38.7060; per masked token 2.765 nats
t = 0.90: 14 of 16 tokens masked; loss by hand 94.4002, library 94.4002; per masked token 6.069 nats
BERT itself is the t = 0.15 case without the 1/t weight: one fixed masking ratio

The two methods agree. Look at the loss per hidden token: 0.209 when BERT sees most of the sentence, 6.069 when almost all of it is hidden. BERT was trained only at 15%, so it never practised with a nearly empty sentence. A writing model must start from an empty one, and that is why LLaDA trains at every masking level.

So can plain BERT write, with no extra training? We gave bert-base-uncased a prompt and eight blanks. At each step, it fills the one blank it is surest about:

BERT fills 8 blanks after "the best thing about a quiet morning is" (our run)step 1.p = 0.98step 2not.p = 0.54step 3notto.p = 0.41step 4nottoa.p = 0.15step 5nottofora.p = 0.30step 6nottoforawhile.p = 0.50step 7nottobeforawhile.p = 0.53step 8nottobealoneforawhile.p = 0.49
Writing with BERT, one blank per step (our run, bert-base-uncased, no extra training). The teal token is the one filled at that step, orange boxes are still blank, and p is BERT's probability for the new token.
plain text
== 10. Writing with BERT: start from [MASK] x 8, unmask the most confident blank each step
step 1: fills "." (p = 0.98)   the best thing about a quiet morning is _ _ _ _ _ _ _ .
step 2: fills "not" (p = 0.54)   the best thing about a quiet morning is not _ _ _ _ _ _ .
step 3: fills "to" (p = 0.41)   the best thing about a quiet morning is not to _ _ _ _ _ .
step 4: fills "a" (p = 0.15)   the best thing about a quiet morning is not to _ _ _ a _ .
step 5: fills "for" (p = 0.30)   the best thing about a quiet morning is not to _ _ for a _ .
step 6: fills "while" (p = 0.50)   the best thing about a quiet morning is not to _ _ for a while .
step 7: fills "be" (p = 0.53)   the best thing about a quiet morning is not to be _ for a while .
step 8: fills "alone" (p = 0.49)   the best thing about a quiet morning is not to be alone for a while .

BERT writes a correct sentence: "the best thing about a quiet morning is not to be alone for a while".

Look at the order. It fills the full stop first, at 0.98, because it is sure the sentence ends there. Then it writes from both ends: "a" near the end comes before "be" in the middle. A left-to-right model cannot do this. The weak point is also visible: "a" got only 0.15, a guess made with almost nothing around it. Training at every masking level, as LLaDA does, is what makes those early steps reliable.

2026: how much to hide?

The masking ratio started as one number in the BERT paper. Eight years later, it is the main design choice of the encoder branch:

How much of the text is hidden during training0%15%30%50%100%BERT201815%RoBERTa201915%ModernBERT202430%NeoBERT202520%mmBERT202530%, then 15%, then 5% (one stage each)LLaDA2025a new random share for every sequenceUltraBERT20265%50%both in every batch, one output layer each
How much of the text each model hides during training, from the papers named. BERT and RoBERTa used 15%; ModernBERT doubled it; mmBERT lowers it stage by stage; LLaDA draws a new share for every sequence; UltraBERT uses 5% and 50% together.

The story in one table

StepYearMain ideaKey equationWhat it could not do
n-grams1948 oncount what followscount(wt−1,wt)count(wt−1)\frac{\text{count}(w_{t-1}, w_t)}{\text{count}(w_{t-1})}share knowledge between similar words
word2vec2013one vector per word−log⁡σ(uo⊤vc)−∑log⁡σ(−uk⊤vc)-\log\sigma(u_o^\top v_c) - \sum \log\sigma(-u_k^\top v_c)tell two meanings of a word apart
RNN, LSTM1997 to 2014read in order, keep a stateht=tanh⁡(Wht−1+Uxt)h_t = \tanh(W h_{t-1} + U x_t)remember far back (RNN); run in parallel
seq2seq + attention2014look back at every input wordci=∑jαijhjc_i = \sum_j \alpha_{ij} h_jdrop the slow RNN
Transformer2017attention onlysoftmax(QK⊤/dk)V\text{softmax}(QK^\top / \sqrt{d_k}) Vlearn from plain text alone (needs labels)
ELMo, ULMFiT, GPT2018pre-train, then reuse∑ilog⁡P(ui∣u<i)\sum_i \log P(u_i \mid u_{<i})see both sides of a word in every layer
BERT2018fill in the blanks, both sides−∑i∈Mlog⁡P(xi∣x∖M)-\sum_{i \in M} \log P(x_i \mid x_{\setminus M})write text; read more than 512 tokens
RoBERTa to DeBERTa2019 to 2021train BERT bettersame as BERT, refinedwrite text
T52019every task as text to textspan corruptionstay small
GPT-32020scale, then promptL(N)=(Nc/N)αNL(N) = (N_c / N)^{\alpha_N}stay cheap
ModernBERT, NeoBERT, mmBERT2024 to 2025BERT with decoder-era partsBERT's loss, 20 to 30% maskedwrite text
LLaDA2025fill the blanks to write−E[1t∑i∈Mlog⁡p(xi∣xt)]-\mathbb{E}\big[\tfrac{1}{t}\sum_{i \in M} \log p(x_i \mid x_t)\big]write as fast as it reads
UltraBERT2026two masking ratios at onceBERT's loss at 5% and 50%(a preprint; too early to say)

Each row fixed the weakness in the row above. That is the "curve of learning" of the whole field: not one leap, but a chain of fixes, each one possible only because of the last.

What our small runs cannot show

Our runs used small data and small models on a laptop. The bigram model and our word2vec saw 1.7 million words of Wikipedia; the real systems saw billions, and our analogy test failed for that reason. Our RNN had random weights and was never trained; it shows how the signal fades, not how a trained LSTM behaves. The scaling numbers are the law's predictions, not measured losses. The benchmark scores are copied from the papers, with their tables named above, and we did not rerun any of those systems.

Our writing test used plain BERT, which was never trained to start from an empty sentence; it shows the idea of LLaDA, not its quality. The results of UltraBERT come from a preprint that is still under review.

Next: Part 1 reads the BERT paper itself, starting with its title and abstract.

Run it yourself

The script behind this page is code/papers/bert/bert_partb.py. It builds the bigram model and trains word2vec on WikiText-2. It measures the fading gradient of a plain RNN two ways and compares the vectors of "bank" in word2vec and BERT. It also applies the scaling law and prints the benchmark scores with their sources. Every number is saved in results/partb.json. On an Apple M-series laptop it takes about three minutes.

The second script, bert_partb_after.py, covers ModernBERT and after. It compares BERT and ModernBERT, counts the alternating attention, computes the LLaDA loss two ways and lets BERT write. Its numbers are in results/partb_after.json; it runs in about a minute on the CPU.

bash
pip install torch transformers datasets
python bert_partb.py         # downloads Salesforce/wikitext (wikitext-2-raw-v1) and bert-base-uncased
python bert_partb_after.py   # downloads answerdotai/ModernBERT-base (needs transformers 4.48 or later)
Terminal output of bert_partb.py: the bigram perplexities, the word2vec training and neighbours, the RNN gradient table, the bank similarities, the scaling law and the benchmark scores
The full output of bert_partb.py, as it ran for this page.
Terminal output of bert_partb_after.py: the BERT and ModernBERT comparison, the alternating attention count, the LLaDA loss by hand and by library, BERT writing a sentence and the masking ratios
The full output of bert_partb_after.py.

The papers

Before BERT: word2vec · negative sampling · seq2seq · attention · Transformer · ULMFiT · ELMo · GPT · GLUE · BERT

After BERT: SuperGLUE · RoBERTa · T5 · scaling laws · GPT-3 · DeBERTa · ModernBERT · NeoBERT · LLaDA · Ettin · mmBERT · UltraBERT · our scripts