# After BERT: Impact, Limits and Summary

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/bert/part-6-impact-and-summary/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/bert/part-6-impact-and-summary.md


This is the last part. We have read every section of the paper: the idea (Part 1), the model and its input (Part 2), pre-training (Part 3), fine-tuning and results (Part 4) and the ablations (Part 5). Only the short conclusion is left.

After that, this part steps outside the paper. It answers three questions: what did other researchers change after BERT, what can BERT **not** do, and how are BERT-style models used today. Every claim about later work comes from that work's own paper or post, linked at the end. The measurements come from a script you can run.

## The conclusion {§6}

> [!PAPER] · §6
> [![The conclusion of the BERT paper: rich, unsupervised pre-training is an integral part of many language understanding systems; these results enable even low-resource tasks to benefit from deep unidirectional architectures; our major contribution is further generalizing these findings to deep bidirectional architectures, allowing the same pre-trained model to successfully tackle a broad set of NLP tasks](https://ishwarj.com/img/papers/bert/p6-conclusion.pngt/p6-conclusion.png)](/img/papers/bert/p6-conclusion.png)
>
> **What it says:** earlier work showed that "rich, unsupervised pre-training is an integral part of many language understanding systems", and that it lets "even low-resource tasks" benefit from "deep unidirectional architectures". BERT's "major contribution is further generalizing these findings to deep *bidirectional* architectures", so "the same pre-trained model" can handle "a broad set of NLP tasks".
>
> **What it means:**
>
> The authors describe their own contribution modestly. They did not invent pre-training or fine-tuning; GPT and others had already shown that both work. Their claim is narrower: they made the recipe work with a model that looks both ways, in every layer.
>
> Think of a recipe that cooks already knew was good, and one cook who changed a single ingredient. The conclusion names that ingredient: bidirectionality.

Three phrases need unpacking.

> [!DEFINITION] Transfer learning
> Learning something on one task (here, filling in blanks in Wikipedia and books) and reusing that knowledge on a different task (here, sentiment or question answering). Pre-training followed by fine-tuning is transfer learning.

> [!DEFINITION] Low-resource task
> A task with only a few labelled examples. Collecting labels is slow and expensive, so many real tasks only have a few thousand, or even a few hundred. For example, the RTE task in Part 4 has 2.5k training examples.

**"Even low-resource tasks benefit from deep unidirectional architectures."** This sentence is about the work *before* BERT, mainly OpenAI GPT. GPT showed that a deep, left-to-right model, pre-trained on plain text, makes even small tasks work well, because the task no longer has to teach the model language from scratch.

**"Our major contribution is further generalizing these findings to deep bidirectional architectures."** BERT keeps that recipe and changes one thing: the direction. The masked language model (Part 3) is what made it possible to pre-train a model that looks both ways in every layer. The ablations in Part 5 showed that this one change is where most of the gain comes from.

**"The same pre-trained model."** One download, many tasks. This is the practical part of the conclusion, and it is why BERT spread so fast.

<figure class="fig"><svg viewBox="0 0 760 342" role="img" aria-label="A two by two grid. Columns: one direction or both directions. Rows: shallow (directions meet only at the end) or deep (every layer mixes context). ELMo is shallow and two-way, OpenAI GPT is deep and one-way, BERT is deep and two-way."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Where the conclusion puts BERT</text><text class="t-note" x="300.0" y="52.0" text-anchor="middle">one direction (left to right)</text><text class="t-note" x="580.0" y="52.0" text-anchor="middle">both directions</text><text class="t-note" x="118.0" y="132.0" text-anchor="end">shallow:</text><text class="t-muted" x="118.0" y="150.0" text-anchor="end">directions meet</text><text class="t-muted" x="118.0" y="166.0" text-anchor="end">only at the end</text><text class="t-note" x="118.0" y="262.0" text-anchor="end">deep:</text><text class="t-muted" x="118.0" y="280.0" text-anchor="end">every layer</text><text class="t-muted" x="118.0" y="296.0" text-anchor="end">mixes context</text><rect class="box" x="160.0" y="70.0" width="270.0" height="112.0" rx="12"/><text class="t-note" x="295.0" y="110.0" text-anchor="middle">a single one-way LSTM LM</text><text class="t-tick" x="295.0" y="134.0" text-anchor="middle">one direction, nothing to join</text><text class="t-tick" x="295.0" y="152.0" text-anchor="middle"></text><rect class="box-2" x="440.0" y="70.0" width="270.0" height="112.0" rx="12"/><text class="t-note" x="575.0" y="110.0" text-anchor="middle">ELMo (Peters et al., 2018a)</text><text class="t-tick" x="575.0" y="134.0" text-anchor="middle">a left LSTM + a right LSTM,</text><text class="t-tick" x="575.0" y="152.0" text-anchor="middle">outputs concatenated at the top</text><rect class="box-1" x="160.0" y="200.0" width="270.0" height="112.0" rx="12"/><text class="t-note" x="295.0" y="240.0" text-anchor="middle">OpenAI GPT (Radford et al., 2018)</text><text class="t-tick" x="295.0" y="264.0" text-anchor="middle">deep Transformer, left context only:</text><text class="t-tick" x="295.0" y="282.0" text-anchor="middle">"deep unidirectional architectures"</text><rect class="box-on" x="440.0" y="200.0" width="270.0" height="112.0" rx="12"/><text class="t-note" x="575.0" y="240.0" text-anchor="middle">BERT (this paper)</text><text class="t-tick" x="575.0" y="264.0" text-anchor="middle">deep Transformer, both sides in all</text><text class="t-tick" x="575.0" y="282.0" text-anchor="middle">layers: "deep bidirectional"</text><line class="path" x1="432.0" y1="256.0" x2="446.0" y2="256.0" marker-end="url(#ah-on)"/><text class="t-muted" x="440.0" y="330.0" text-anchor="middle">the step the conclusion calls "our major contribution"</text></svg><figcaption>Where the conclusion places BERT. Two questions: does the model read one direction or both, and do the directions meet only at the end (shallow) or in every layer (deep)? ELMo is both-directions but shallow; OpenAI GPT is deep but one-directional; BERT is the first that is deep and bidirectional, which the conclusion calls "our major contribution".</figcaption></figure>

## What happened next

The paper's code and weights were public from the start, so other researchers could build on it right away. The years 2019 and 2020 brought a wave of follow-up models.

<figure class="fig"><svg viewBox="0 0 760 430" role="img" aria-label="A timeline from the Transformer in 2017 to ModernBERT in 2024: BERT in October 2018, then a wave of follow-up models in 2019 and 2020."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="edge-dim" x1="130.0" y1="20.0" x2="130.0" y2="396.0"/><text class="t-tick" x="112.0" y="30.0" text-anchor="end">Jun 2017</text><rect class="box" x="124.0" y="20.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="31.0" text-anchor="start">Transformer</text><text class="t-muted" x="410.0" y="31.0" text-anchor="start">the architecture BERT is built from</text><text class="t-tick" x="112.0" y="60.0" text-anchor="end">Oct 2018</text><rect class="box-on" x="124.0" y="50.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="61.0" text-anchor="start">BERT</text><text class="t-muted" x="410.0" y="61.0" text-anchor="start">this paper: deep bidirectional pre-training</text><text class="t-tick" x="112.0" y="90.0" text-anchor="end">Jun 2019</text><rect class="box-3" x="124.0" y="80.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="91.0" text-anchor="start">BERT wins NAACL best long paper</text><text class="t-muted" x="410.0" y="91.0" text-anchor="start">the conference where it was published</text><text class="t-tick" x="112.0" y="120.0" text-anchor="end">Jun 2019</text><rect class="box-1" x="124.0" y="110.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="121.0" text-anchor="start">XLNet</text><text class="t-muted" x="410.0" y="121.0" text-anchor="start">bidirectional context without [MASK]</text><text class="t-tick" x="112.0" y="150.0" text-anchor="end">Jul 2019</text><rect class="box-1" x="124.0" y="140.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="151.0" text-anchor="start">RoBERTa</text><text class="t-muted" x="410.0" y="151.0" text-anchor="start">same model, trained better; drops NSP</text><text class="t-tick" x="112.0" y="180.0" text-anchor="end">Jul 2019</text><rect class="box-1" x="124.0" y="170.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="181.0" text-anchor="start">SpanBERT</text><text class="t-muted" x="410.0" y="181.0" text-anchor="start">masks whole spans of words</text><text class="t-tick" x="112.0" y="210.0" text-anchor="end">Aug 2019</text><rect class="box-1" x="124.0" y="200.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="211.0" text-anchor="start">Sentence-BERT</text><text class="t-muted" x="410.0" y="211.0" text-anchor="start">turns BERT into a sentence-embedding model</text><text class="t-tick" x="112.0" y="240.0" text-anchor="end">Sep 2019</text><rect class="box-1" x="124.0" y="230.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="241.0" text-anchor="start">ALBERT</text><text class="t-muted" x="410.0" y="241.0" text-anchor="start">far fewer parameters; sentence order instead of NSP</text><text class="t-tick" x="112.0" y="270.0" text-anchor="end">Oct 2019</text><rect class="box-1" x="124.0" y="260.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="271.0" text-anchor="start">DistilBERT</text><text class="t-muted" x="410.0" y="271.0" text-anchor="start">40% smaller, 60% faster, keeps 97%</text><text class="t-tick" x="112.0" y="300.0" text-anchor="end">Oct 2019</text><rect class="box-3" x="124.0" y="290.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="301.0" text-anchor="start">BERT in Google Search</text><text class="t-muted" x="410.0" y="301.0" text-anchor="start">"one in 10 searches in the U.S. in English"</text><text class="t-tick" x="112.0" y="330.0" text-anchor="end">Mar 2020</text><rect class="box-1" x="124.0" y="320.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="331.0" text-anchor="start">ELECTRA</text><text class="t-muted" x="410.0" y="331.0" text-anchor="start">learns from every token, not just the masked 15%</text><text class="t-tick" x="112.0" y="360.0" text-anchor="end">Jun 2020</text><rect class="box-1" x="124.0" y="350.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="361.0" text-anchor="start">DeBERTa</text><text class="t-muted" x="410.0" y="361.0" text-anchor="start">separate vectors for content and position</text><text class="t-tick" x="112.0" y="390.0" text-anchor="end">Dec 2024</text><rect class="box-1" x="124.0" y="380.0" width="12.0" height="12.0" rx="3"/><text class="t-note" x="150.0" y="391.0" text-anchor="start">ModernBERT</text><text class="t-muted" x="410.0" y="391.0" text-anchor="start">8192-token inputs, 2 trillion training tokens</text><text class="t-muted" x="20.0" y="418.0" text-anchor="start">Dates are first arXiv versions, except the award (NAACL, June 2019) and the Search post (25 October 2019).</text></svg><figcaption>From the Transformer to ModernBERT. Most dates are the month of the first arXiv version; the award and the Search post have their own dates. The short notes are explained below.</figcaption></figure>

Two events show how quickly BERT mattered outside research:

- In June 2019 the paper won the **Best Long Paper** award at NAACL 2019, the conference where it was published.
- On 25 October 2019, Google wrote that BERT "will help Search better understand one in 10 searches in the U.S. in English". The same post says Google also applied a BERT model to improve featured snippets "in the two dozen countries where this feature is available".

> [!DEFINITION] Featured snippet
> The short answer box that a search engine sometimes shows above the normal results, quoted from a web page.

The follow-up models changed BERT in four different directions.

### 1. Train the same model better: RoBERTa and SpanBERT

> [!DEFINITION] Hyperparameter
> A setting that people choose before training rather than the model learning it: the batch size, the learning rate, the number of steps, how much data to use.

**RoBERTa** (Liu et al., July 2019) is a careful replication of BERT. Its abstract says plainly: "We find that BERT was significantly undertrained." With the **same architecture**, they changed only the training: "(1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data." They trained on over 160GB of text instead of BERT's 16GB (Wikipedia plus books, in their measurement).

> [!PAPER] Liu et al. (2019), RoBERTa · Abstract
> [![The RoBERTa abstract: a replication study of BERT pretraining that measures the impact of key hyperparameters and training data size; we find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it](https://ishwarj.com/img/papers/bert/p6-roberta.pngbert/p6-roberta.png)](/img/papers/bert/p6-roberta.png)
>
> **What it says:** "We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it."
>
> **What it means:**
>
> "Undertrained" means the model could have learned more, but training stopped too early. It is like judging a runner after only half of the training plan. The architecture was fine; the training recipe had room to grow.
>
> RoBERTa came from another lab (Facebook AI), nine months after BERT. With the same architecture and better training, it matched or beat every model published after BERT. This is why later "BERT-style" models usually mean RoBERTa-style training.

> [!DEFINITION] Dynamic masking
> Choosing new positions to mask every time a sentence is shown to the model. The original BERT pipeline chose the masked positions once, when the training data was prepared (RoBERTa calls this "static" masking). RoBERTa notes that BERT's data was duplicated 10 times "so that each sequence is masked in 10 different ways over the 40 epochs of training", so each fixed mask was still seen several times.

The NSP result is interesting, because it seems to contradict BERT's own Table 5 (Part 5), where removing NSP hurt. RoBERTa found that "removing the NSP loss matches or slightly improves downstream task performance". They offer a possible reason: "the original BERT implementation may only have removed the loss term while still retaining the SEGMENT-PAIR input format". In other words, BERT's "No NSP" model still trained on pairs of shorter text pieces; RoBERTa's trained on long, full blocks of text. Both results can be true, because they removed NSP in different ways.

**SpanBERT** (Joshi et al., July 2019) masks "contiguous random spans, rather than random tokens", and gains most on "span selection tasks such as question answering and coreference resolution". Its abstract reports 94.6 F1 on SQuAD 1.1 and 88.7 on SQuAD 2.0, "with the same training data and model size as BERT-large".

### 2. Make it smaller: ALBERT and DistilBERT

**ALBERT** (Lan et al., September 2019) uses "two parameter-reduction techniques":

> [!PAPER] Lan et al. (2019), ALBERT · Abstract
> [![The ALBERT abstract: two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT, and a self-supervised loss that focuses on modeling inter-sentence coherence](https://ishwarj.com/img/papers/bert/p6-albert.png/bert/p6-albert.png)](/img/papers/bert/p6-albert.png)
>
> **What it says:** "two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT", plus "a self-supervised loss that focuses on modeling inter-sentence coherence".
>
> **What it means:**
>
> ALBERT is a Google follow-up aimed at size. The first half of the sentence is about memory and speed: the two techniques, listed below, make the model much smaller.
>
> The second half is about a pre-training task. The new loss on "inter-sentence coherence" replaces NSP, the part of BERT that later work found weakest.

- **factorized embedding parameterization**: the big vocabulary table is split into two small matrices, so its size no longer grows with the hidden size;
- **cross-layer parameter sharing**: all layers share the same weights, so the size no longer grows with the depth.

ALBERT also replaces NSP with **sentence-order prediction (SOP)**, a task about "inter-sentence coherence": are these two consecutive pieces in the right order or swapped? The paper describes it as designed to address "the ineffectiveness" of NSP. Their example of the size saving: "ALBERT-large has about 18x fewer parameters compared to BERT-large, 18M versus 334M".

> [!DEFINITION] Knowledge distillation
> Training a small "student" model to copy the outputs of a big "teacher" model. The student learns from the teacher's full probability lists, not just from the right answers, which is a richer signal.

**DistilBERT** (Sanh et al., October 2019) uses distillation during pre-training. Its abstract: "it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster."

> [!PAPER] Sanh et al. (2019), DistilBERT · Abstract
> [![The DistilBERT abstract: knowledge distillation during the pre-training phase reduces the size of a BERT model by 40 percent while retaining 97 percent of its language understanding capabilities and being 60 percent faster](https://ishwarj.com/img/papers/bert/p6-distilbert.pngt/p6-distilbert.png)](/img/papers/bert/p6-distilbert.png)
>
> **What it says:** "knowledge distillation during the pre-training phase" makes it possible "to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster".
>
> **What it means:**
>
> DistilBERT is a Hugging Face follow-up aimed at speed. Think of a student who learns from a teacher's full notes, not only from the answer key. The small model learns to copy the big one during pre-training.
>
> The result is a much cheaper model to run, for a small loss in quality: 40% smaller, 60% faster, and 97% of the language understanding. Below we check the size claim and show what distillation trains on.

The distillation loss, for one masked position, compares the student's probabilities $$s$$ with the teacher's $$t$$ over the vocabulary:

$$
\mathcal{L}_{\text{hard}} = -\log s(y), \qquad \mathcal{L}_{\text{soft}} = -\sum_{w} t(w) \log s(w)
$$

where $$y$$ is the true word and the sum runs over every word $$w$$ of the vocabulary. The hard loss only rewards the right answer; the soft loss also rewards giving the teacher's runner-ups some probability. (DistilBERT combines this soft loss with the masked-LM loss and a cosine loss on the hidden vectors; here we show only the idea.) Measured on "i went to the [MASK] to deposit my paycheck." ([`bert_part6_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part6_math.py)):

```text
encoder parameters: teacher 108,891,648 (12 layers), student 66,362,880 (6 layers), ratio 0.609
teacher top 5: bank 0.901, office 0.009, store 0.008, teller 0.008, atm 0.006
student top 5: bank 0.365, store 0.035, mall 0.025, hotel 0.022, supermarket 0.020
hard-label loss  -log s(bank)               = 1.009
soft-target loss -sum_w t(w) log s(w)       = 1.467   (teacher entropy, the smallest possible value: 0.774)
```

The student has 0.609 of the teacher's encoder weights, the "40% smaller" of the abstract. The soft loss can never go below the teacher's own **entropy** (0.774 here), reached only when the student copies the teacher exactly.

<figure class="fig"><svg viewBox="0 0 760 276" role="img" aria-label="The same blank through a teacher (BERT-base) and a student (DistilBERT). The teacher puts 0.901 on bank, the student 0.365. Distillation trains the student to match the teacher's whole list of probabilities, not only the one right word."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Distillation: the student learns from the teacher's whole probability list</text><text class="t-mono" x="20.0" y="44.0" text-anchor="start">i went to the [MASK] to deposit my paycheck.</text><text class="t-title" x="20.0" y="82.0" text-anchor="start">teacher: BERT-base (108.9M encoder weights)</text><text class="t-tick t-strong" x="92.0" y="108.0" text-anchor="end">bank</text><g class="mark"><title>bank: 0.901</title><rect class="s1" x="100.0" y="95.0" width="189.3" height="16.0" rx="3"/></g><text class="t-val" x="295.3" y="108.0" text-anchor="start">0.901</text><text class="t-tick" x="92.0" y="132.0" text-anchor="end">office</text><g class="mark"><title>office: 0.009</title><rect class="s1" x="100.0" y="119.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="108.0" y="132.0" text-anchor="start">0.009</text><text class="t-tick" x="92.0" y="156.0" text-anchor="end">store</text><g class="mark"><title>store: 0.008</title><rect class="s1" x="100.0" y="143.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="108.0" y="156.0" text-anchor="start">0.008</text><text class="t-tick" x="92.0" y="180.0" text-anchor="end">teller</text><g class="mark"><title>teller: 0.008</title><rect class="s1" x="100.0" y="167.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="108.0" y="180.0" text-anchor="start">0.008</text><text class="t-tick" x="92.0" y="204.0" text-anchor="end">atm</text><g class="mark"><title>atm: 0.006</title><rect class="s1" x="100.0" y="191.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="108.0" y="204.0" text-anchor="start">0.006</text><text class="t-title" x="400.0" y="82.0" text-anchor="start">student: DistilBERT (66.4M)</text><text class="t-tick t-strong" x="482.0" y="108.0" text-anchor="end">bank</text><g class="mark"><title>bank: 0.365</title><rect class="s2" x="490.0" y="95.0" width="76.6" height="16.0" rx="3"/></g><text class="t-val" x="572.6" y="108.0" text-anchor="start">0.365</text><text class="t-tick" x="482.0" y="132.0" text-anchor="end">store</text><g class="mark"><title>store: 0.035</title><rect class="s2" x="490.0" y="119.0" width="7.4" height="16.0" rx="3"/></g><text class="t-val" x="503.4" y="132.0" text-anchor="start">0.035</text><text class="t-tick" x="482.0" y="156.0" text-anchor="end">mall</text><g class="mark"><title>mall: 0.025</title><rect class="s2" x="490.0" y="143.0" width="5.2" height="16.0" rx="3"/></g><text class="t-val" x="501.2" y="156.0" text-anchor="start">0.025</text><text class="t-tick" x="482.0" y="180.0" text-anchor="end">hotel</text><g class="mark"><title>hotel: 0.022</title><rect class="s2" x="490.0" y="167.0" width="4.6" height="16.0" rx="3"/></g><text class="t-val" x="500.6" y="180.0" text-anchor="start">0.022</text><text class="t-tick" x="482.0" y="204.0" text-anchor="end">supermarket</text><g class="mark"><title>supermarket: 0.020</title><rect class="s2" x="490.0" y="191.0" width="4.2" height="16.0" rx="3"/></g><text class="t-val" x="500.2" y="204.0" text-anchor="start">0.020</text><text class="t-mono" x="20.0" y="236.0" text-anchor="start">hard label only:  −log s(bank) = 1.009</text><text class="t-mono" x="20.0" y="258.0" text-anchor="start">soft targets:     −Σ t(w) log s(w) = 1.467   (smallest possible: the teacher's entropy, 0.774)</text></svg><figcaption>Distillation on one sentence. The teacher (BERT-base) puts 0.901 on "bank" and small amounts on office, store, teller, atm. The student (DistilBERT) also picks "bank" but with 0.365. Training on the teacher's whole list (soft targets) gives the student more to learn from than the single right answer.</figcaption></figure>

> [!DEFINITION] Entropy
> How spread out a probability list is: $$-\sum_w t(w) \log t(w)$$. 0 for a list that puts everything on one word; larger when the probability is spread over many words.

### 3. Fix the masked language model: XLNet, ELECTRA and DeBERTa

The masked LM has two weak spots, and two papers named them directly. We measure both later in this part.

**XLNet** (Yang et al., June 2019) says that "relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy". XLNet keeps bidirectional context but predicts words in many different random orders instead of using `[MASK]`.

> [!PAPER] Yang et al. (2019), XLNet · Abstract
> [![The XLNet abstract: relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy; XLNet is a generalized autoregressive pretraining method that maximizes the expected likelihood over all permutations of the factorization order](https://ishwarj.com/img/papers/bert/p6-xlnet.pngs/bert/p6-xlnet.png)](/img/papers/bert/p6-xlnet.png)
>
> **What it says:** BERT "neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy". XLNet instead maximises the likelihood "over all permutations of the factorization order".
>
> **What it means:**
>
> XLNet, from CMU and Google, names two of BERT's weaknesses in one sentence. "Neglects dependency between the masked positions" means that BERT fills two blanks without knowing what it put in the other. "Pretrain-finetune discrepancy" means that `[MASK]` appears in training but never in real use.
>
> These are Limits 2 and 4 below, measured on the real model. XLNet's own fix is to predict the words in many random orders, so it never needs `[MASK]`.

**ELECTRA** (Clark et al., March 2020) points at a different cost: BERT only learns from the masked tokens. ELECTRA replaces some tokens with "plausible alternatives sampled from a small generator network", then trains the main model to say, for every token, whether it was replaced. This is "more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out". Its abstract gives an example: "we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark".

> [!PAPER] Clark et al. (2020), ELECTRA · Abstract
> [![The ELECTRA abstract: replaced token detection; instead of masking, corrupt the input by replacing some tokens with plausible alternatives from a small generator network, then train a discriminator that predicts whether each token was replaced; more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out](https://ishwarj.com/img/papers/bert/p6-electra.pngbert/p6-electra.png)](/img/papers/bert/p6-electra.png)
>
> **What it says:** "replaced token detection": a small generator fills some positions with plausible words, and the main model judges every token, original or replaced. This is efficient "because the task is defined over all input tokens rather than just the small subset that was masked out".
>
> **What it means:**
>
> Think of two kinds of reading test. In BERT's test, a few words of a page are blacked out, and the model must guess them. In ELECTRA's test, a few words were quietly swapped, and the main model must check every word.
>
> So every position gives a training signal, not only the 15% that were masked. ELECTRA, from Stanford and Google, attacks Limit 3 head on.

<figure class="fig"><svg viewBox="0 0 760 360" role="img" aria-label="ELECTRA in three steps: mask a few tokens, let a small generator fill them with plausible words, then train the main model to label every token as original or replaced."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">ELECTRA: replaced token detection, in three steps</text><rect class="box" x="10.0" y="36.0" width="740.0" height="76.0" rx="12"/><circle class="node on" cx="28.0" cy="54.0" r="11"/><text class="t-note" x="28.0" y="58.5" text-anchor="middle">1</text><text class="t-title" x="46.0" y="59.0" text-anchor="start">mask a few tokens (like BERT)</text><rect class="box" x="250.0" y="64.0" width="70.0" height="30.0" rx="6"/><text class="t-muted" x="285.0" y="83.5" text-anchor="middle">the</text><rect class="box-mask" x="328.0" y="64.0" width="70.0" height="30.0" rx="6"/><text class="t-muted" x="363.0" y="83.5" text-anchor="middle">[MASK]</text><rect class="box" x="406.0" y="64.0" width="70.0" height="30.0" rx="6"/><text class="t-muted" x="441.0" y="83.5" text-anchor="middle">cooked</text><rect class="box" x="484.0" y="64.0" width="70.0" height="30.0" rx="6"/><text class="t-muted" x="519.0" y="83.5" text-anchor="middle">the</text><rect class="box-mask" x="562.0" y="64.0" width="70.0" height="30.0" rx="6"/><text class="t-muted" x="597.0" y="83.5" text-anchor="middle">[MASK]</text><rect class="box" x="10.0" y="122.0" width="740.0" height="76.0" rx="12"/><circle class="node on" cx="28.0" cy="140.0" r="11"/><text class="t-note" x="28.0" y="144.5" text-anchor="middle">2</text><text class="t-title" x="46.0" y="145.0" text-anchor="start">a small generator (a small masked LM) fills them</text><rect class="box" x="250.0" y="150.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="285.0" y="169.5" text-anchor="middle">the</text><rect class="box-3" x="328.0" y="150.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="363.0" y="169.5" text-anchor="middle">chef</text><rect class="box" x="406.0" y="150.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="441.0" y="169.5" text-anchor="middle">cooked</text><rect class="box" x="484.0" y="150.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="519.0" y="169.5" text-anchor="middle">the</text><rect class="box-2" x="562.0" y="150.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="597.0" y="169.5" text-anchor="middle">soup</text><text class="t-muted" x="363.0" y="196.0" text-anchor="middle">guessed right</text><text class="t-muted" x="597.0" y="196.0" text-anchor="middle">plausible but wrong</text><rect class="box" x="10.0" y="208.0" width="740.0" height="100.0" rx="12"/><circle class="node on" cx="28.0" cy="226.0" r="11"/><text class="t-note" x="28.0" y="230.5" text-anchor="middle">3</text><text class="t-title" x="46.0" y="231.0" text-anchor="start">the main model labels EVERY token: original or replaced?</text><rect class="box" x="250.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="285.0" y="255.5" text-anchor="middle">the</text><rect class="box" x="328.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="363.0" y="255.5" text-anchor="middle">chef</text><rect class="box" x="406.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="441.0" y="255.5" text-anchor="middle">cooked</text><rect class="box" x="484.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="519.0" y="255.5" text-anchor="middle">the</text><rect class="box" x="562.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="597.0" y="255.5" text-anchor="middle">soup</text><line class="edge" x1="285.0" y1="268.0" x2="285.0" y2="280.0" marker-end="url(#ah)"/><text class="t-s3" x="285.0" y="296.0" text-anchor="middle">original</text><line class="edge" x1="363.0" y1="268.0" x2="363.0" y2="280.0" marker-end="url(#ah)"/><text class="t-s3" x="363.0" y="296.0" text-anchor="middle">original</text><line class="edge" x1="441.0" y1="268.0" x2="441.0" y2="280.0" marker-end="url(#ah)"/><text class="t-s3" x="441.0" y="296.0" text-anchor="middle">original</text><line class="edge" x1="519.0" y1="268.0" x2="519.0" y2="280.0" marker-end="url(#ah)"/><text class="t-s3" x="519.0" y="296.0" text-anchor="middle">original</text><line class="edge" x1="597.0" y1="268.0" x2="597.0" y2="280.0" marker-end="url(#ah)"/><text class="t-bad" x="597.0" y="296.0" text-anchor="middle">replaced</text><text class="t-muted" x="20.0" y="330.0" text-anchor="start">BERT learns from 2 of these 5 positions; ELECTRA's main model gets a loss at all 5.</text><text class="t-muted" x="20.0" y="348.0" text-anchor="start">"chef" counts as original, because the generator happened to produce the real word.</text></svg><figcaption>ELECTRA in three steps. Mask a few tokens; a small generator fills them ("chef" guessed right, "soup" plausible but wrong); the main model labels every token as original or replaced. BERT learns from 2 of these 5 positions, ELECTRA's main model from all 5.</figcaption></figure>

**DeBERTa** (He et al., June 2020) changes attention itself: "each word is represented using two vectors that encode its content and position", instead of one vector that adds them together as BERT does (Part 2).

### 4. Use it differently: Sentence-BERT, and a modern BERT

**Sentence-BERT** (Reimers and Gurevych, August 2019) fixed a problem the paper itself warns about in footnote 6: BERT's `[CLS]` vector is not a good sentence vector without fine-tuning. Sentence-BERT says that averaging BERT's outputs or using the `[CLS]` output "yields rather bad sentence embeddings, often worse than averaging GloVe embeddings". It fine-tunes BERT so that sentences with similar meanings get similar vectors. We test this ourselves below.

> [!PAPER] Reimers and Gurevych (2019), Sentence-BERT · Abstract
> [![The Sentence-BERT abstract: finding the most similar pair in 10,000 sentences requires about 50 million inference computations, about 65 hours, with BERT; SBERT uses siamese and triplet networks to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity, reducing this to about 5 seconds](https://ishwarj.com/img/papers/bert/p6-sbert.pngs/bert/p6-sbert.png)](/img/papers/bert/p6-sbert.png)
>
> **What it says:** with BERT reading every pair together, finding the most similar pair among 10,000 sentences "requires about 50 million inference computations (~65 hours)". SBERT produces "semantically meaningful sentence embeddings that can be compared using cosine-similarity", which cuts this "to about 5 seconds".
>
> **What it means:**
>
> Where does 50 million come from? It is the number of pairs: $$10{,}000 \times 9{,}999 / 2 \approx 5 \times 10^7$$. If BERT must read each pair together, that is 50 million BERT runs.
>
> Sentence-BERT gives each sentence its own vector instead. That turns the job into 10,000 BERT runs plus cheap cosine comparisons. This is what made BERT useful for search and similarity.

<figure class="fig"><svg viewBox="0 0 760 270" role="img" aria-label="Two ways to compare a query with a passage. A cross-encoder reads both in one input and outputs one score. A bi-encoder turns each into its own vector with the same BERT, then compares the vectors with cosine similarity."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">Cross-encoder (BERT as in the paper)</text><text class="t-title" x="570.0" y="22.0" text-anchor="middle">Bi-encoder (Sentence-BERT)</text><rect class="box-1" x="40.0" y="200.0" width="140.0" height="30.0" rx="6"/><text class="t-tick" x="110.0" y="219.5" text-anchor="middle">query</text><rect class="box-2" x="200.0" y="200.0" width="140.0" height="30.0" rx="6"/><text class="t-tick" x="270.0" y="219.5" text-anchor="middle">passage</text><line class="edge" x1="190.0" y1="198.0" x2="190.0" y2="172.0" marker-end="url(#ah)"/><rect class="box-on" x="40.0" y="110.0" width="300.0" height="60.0" rx="10"/><text class="t-note" x="190.0" y="136.0" text-anchor="middle">one BERT reads both together</text><text class="t-muted" x="190.0" y="154.0" text-anchor="middle">every query token attends to every passage token</text><line class="edge" x1="190.0" y1="108.0" x2="190.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-3" x="130.0" y="52.0" width="120.0" height="30.0" rx="8"/><text class="t-tick" x="190.0" y="72.0" text-anchor="middle">one score</text><text class="t-muted" x="190.0" y="252.0" text-anchor="middle">accurate; must run once per (query, passage) pair</text><rect class="box-1" x="430.0" y="200.0" width="130.0" height="30.0" rx="6"/><text class="t-tick" x="495.0" y="219.5" text-anchor="middle">query</text><line class="edge" x1="495.0" y1="198.0" x2="495.0" y2="172.0" marker-end="url(#ah)"/><rect class="box-on" x="430.0" y="116.0" width="130.0" height="54.0" rx="10"/><text class="t-note" x="495.0" y="140.0" text-anchor="middle">BERT</text><text class="t-muted" x="495.0" y="158.0" text-anchor="middle">(same weights)</text><line class="edge" x1="495.0" y1="114.0" x2="495.0" y2="98.0" marker-end="url(#ah)"/><rect class="box-ghost" x="447.0" y="81.0" width="88.0" height="18.0" rx="4"/><g class="mark"><rect class="s1" x="450.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s1" x="464.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="478.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="492.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s1" x="506.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s1" x="520.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.86"/></g><rect class="box-2" x="600.0" y="200.0" width="130.0" height="30.0" rx="6"/><text class="t-tick" x="665.0" y="219.5" text-anchor="middle">passage</text><line class="edge" x1="665.0" y1="198.0" x2="665.0" y2="172.0" marker-end="url(#ah)"/><rect class="box-on" x="600.0" y="116.0" width="130.0" height="54.0" rx="10"/><text class="t-note" x="665.0" y="140.0" text-anchor="middle">BERT</text><text class="t-muted" x="665.0" y="158.0" text-anchor="middle">(same weights)</text><line class="edge" x1="665.0" y1="114.0" x2="665.0" y2="98.0" marker-end="url(#ah)"/><rect class="box-ghost" x="617.0" y="81.0" width="88.0" height="18.0" rx="4"/><g class="mark"><rect class="s2" x="620.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s2" x="634.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="648.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s2" x="662.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s2" x="676.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s2" x="690.0" y="84.0" width="12" height="12" rx="2" style="fill-opacity:0.86"/></g><line class="edge-on" x1="544.0" y1="90.0" x2="616.0" y2="90.0"/><text class="t-on" x="580.0" y="74.0" text-anchor="middle">cosine</text><text class="t-muted" x="570.0" y="252.0" text-anchor="middle">fast; passage vectors are computed once, ahead of time</text><line class="edge-dim" x1="380.0" y1="40.0" x2="380.0" y2="262.0"/></svg><figcaption>Two ways to compare texts. Cross-encoder (BERT as in the paper): read the query and the passage together, one score per pair; accurate, but one BERT run for every pair. Bi-encoder (Sentence-BERT): one vector per text, compared by cosine; passage vectors can be computed once, ahead of time.</figcaption></figure>

**ModernBERT** (Warner et al., December 2024) shows the idea is still alive. It calls encoder-only models like BERT "the workhorse of numerous production pipelines", and brings modern training to them: "Trained on 2 trillion tokens with a native 8192 sequence length". Compare that with BERT's 512 tokens.

> [!PAPER] Warner et al. (2024), ModernBERT · Abstract
> [![The ModernBERT abstract: encoder-only transformer models such as BERT are the workhorse of numerous production pipelines; ModernBERT is trained on 2 trillion tokens with a native 8192 sequence length](https://ishwarj.com/img/papers/bert/p6-modernbert.pngt/p6-modernbert.png)](/img/papers/bert/p6-modernbert.png)
>
> **What it says:** encoders like BERT are "the workhorse of numerous production pipelines". ModernBERT is "Trained on 2 trillion tokens with a native 8192 sequence length".
>
> **What it means:**
>
> ModernBERT is a 2024 encoder from Answer.AI, LightOn and others. Six years after BERT, the same design is still in production use, now with modern training.
>
> The context length is the clearest change: 8,192 tokens against BERT's 512, which is 16 times more. Limit 1 below shows why that wall mattered.

<figure class="fig"><svg viewBox="0 0 760 432" role="img" aria-label="Each weakness of BERT on the left, and the follow-up model that attacks it on the right: ModernBERT for length, XLNet and ELECTRA for the mask, ELECTRA for the 15% signal, Sentence-BERT for sentence vectors, ALBERT and DistilBERT for size, RoBERTa and SpanBERT for the training recipe."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Which follow-up attacks which weakness</text><rect class="box-mask" x="20.0" y="40.0" width="250.0" height="34.0" rx="8"/><text class="t-note" x="32.0" y="61.0" text-anchor="start">512-token wall (Limit 1)</text><rect class="box-1" x="330.0" y="44.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="59.0" text-anchor="start">ModernBERT: 8192 tokens</text><line class="edge" x1="272.0" y1="57.0" x2="326.0" y2="57.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="84.0" width="250.0" height="60.0" rx="8"/><text class="t-note" x="32.0" y="118.0" text-anchor="start">[MASK] mismatch (Limit 2)</text><rect class="box-1" x="330.0" y="88.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="103.0" text-anchor="start">XLNet: no [MASK], permuted order</text><rect class="box-1" x="330.0" y="114.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="129.0" text-anchor="start">ELECTRA: real tokens in, real/replaced out</text><line class="edge" x1="272.0" y1="114.0" x2="326.0" y2="114.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="154.0" width="250.0" height="34.0" rx="8"/><text class="t-note" x="32.0" y="175.0" text-anchor="start">only 15% teach (Limit 3)</text><rect class="box-1" x="330.0" y="158.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="173.0" text-anchor="start">ELECTRA: a loss at every token</text><line class="edge" x1="272.0" y1="171.0" x2="326.0" y2="171.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="198.0" width="250.0" height="34.0" rx="8"/><text class="t-note" x="32.0" y="219.0" text-anchor="start">blanks independent (Limit 4)</text><rect class="box-1" x="330.0" y="202.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="217.0" text-anchor="start">XLNet: one at a time, later guesses see earlier</text><line class="edge" x1="272.0" y1="215.0" x2="326.0" y2="215.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="242.0" width="250.0" height="34.0" rx="8"/><text class="t-note" x="32.0" y="263.0" text-anchor="start">no sentence vector (Limit 5)</text><rect class="box-1" x="330.0" y="246.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="261.0" text-anchor="start">Sentence-BERT: fine-tuned for cosine</text><line class="edge" x1="272.0" y1="259.0" x2="326.0" y2="259.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="286.0" width="250.0" height="60.0" rx="8"/><text class="t-note" x="32.0" y="320.0" text-anchor="start">too big, too slow (Limit 7)</text><rect class="box-1" x="330.0" y="290.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="305.0" text-anchor="start">ALBERT: shared layers</text><rect class="box-1" x="330.0" y="316.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="331.0" text-anchor="start">DistilBERT: 6 layers, distilled</text><line class="edge" x1="272.0" y1="316.0" x2="326.0" y2="316.0" marker-end="url(#ah)"/><rect class="box-mask" x="20.0" y="356.0" width="250.0" height="60.0" rx="8"/><text class="t-note" x="32.0" y="390.0" text-anchor="start">undertrained recipe</text><rect class="box-1" x="330.0" y="360.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="375.0" text-anchor="start">RoBERTa: more data, no NSP, dynamic masks</text><rect class="box-1" x="330.0" y="386.0" width="410.0" height="22.0" rx="6"/><text class="t-tick" x="342.0" y="401.0" text-anchor="start">SpanBERT: mask whole spans</text><line class="edge" x1="272.0" y1="386.0" x2="326.0" y2="386.0" marker-end="url(#ah)"/></svg><figcaption>Which follow-up attacks which weakness. 512-token wall: ModernBERT (8,192 tokens). [MASK] mismatch: XLNet, ELECTRA. Only 15% of tokens teach: ELECTRA. Blanks filled independently: XLNet. No sentence vector: Sentence-BERT. Too big or slow: ALBERT, DistilBERT. Undertrained recipe: RoBERTa, SpanBERT.</figcaption></figure>

## BERT's limits, measured

Every model has limits. Here are BERT's main ones. Where the paper itself mentions a limit, its lines are shown; where a limit can be measured, I measured it with [`bert_part6.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part6.py) on the public `bert-base-uncased` model.

### Limit 1: at most 512 tokens

> [!PAPER] · §A.2
> [![Appendix A.2: longer sequences are disproportionately expensive because attention is quadratic to the sequence length. BERT is pre-trained with sequence length 128 for 90 percent of the steps, then 512 for the last 10 percent to learn the positional embeddings](https://ishwarj.com/img/papers/bert/p6-length.png/bert/p6-length.png)](/img/papers/bert/p6-length.png)
>
> **What it says:** "attention is quadratic to the sequence length", so long inputs are expensive. BERT trains on 128-token sequences for 90% of the steps and on 512-token sequences for the last 10%, "to learn the positional embeddings".
>
> **What it means:**
>
> The position table is like a ruler with marks only up to 511. BERT learned one position vector for each mark, and none beyond it. So 512 tokens is a hard wall, not a soft limit.
>
> Why not train on long sequences all the time? Because attention is "quadratic", as the yellow box below explains. The authors saved compute with short sequences for 90% of the steps and long ones only at the end.

> [!DEFINITION] Quadratic
> Growing with the square. If attention compares every token with every token, then 2 times more tokens means 2 × 2 = 4 times more comparisons, and 4 times more tokens means 16 times more.

Recall from Part 2 that BERT adds a **position embedding** to every token, looked up in a table with one row per position. The code reads the size of that table and then tries a long input:

```python
bert = AutoModel.from_pretrained("bert-base-uncased")
print(bert.embeddings.position_embeddings.weight.shape)     # rows = positions it knows

long_text = " ".join(["BERT reads the whole input at once, so every token can look at every other token."] * 40)
print(len(tok(long_text).input_ids))                                   # all tokens
print(len(tok(long_text, truncation=True, max_length=512).input_ids))  # what is kept
bert(**tok(long_text, return_tensors="pt"))                            # try all of them
```

```text
1. The 512-token limit
   position embedding table: 512 rows x 768 numbers
   a long text has 722 tokens; with truncation it keeps 512; the rest is thrown away
   feeding all 722 tokens: RuntimeError: The size of tensor a (722) must match the size of tensor b (512) at non-singleton dimension 1
```

The table has exactly 512 rows. A 722-token text either crashes the model or loses its last 210 tokens. A few long paragraphs are already too much, so long documents have to be cut into pieces. This is the limit ModernBERT's 8192-token context attacks.

> [!DEFINITION] Truncation
> Cutting an input to a maximum length and throwing away the rest.

The usual workaround, used by the released SQuAD code (Part 4), is to read a long text in **overlapping windows**. With window size $$w$$ content tokens (510, leaving room for `[CLS]` and `[SEP]`) and step $$s$$ between window starts, a text of $$N$$ tokens needs

$$
k = 1 + \left\lceil \frac{N - w}{s} \right\rceil \text{ windows}
$$

where $$\lceil \cdot \rceil$$ rounds up. For $$N = 720$$, $$w = 510$$, $$s = 382$$: $$k = 1 + \lceil 210 / 382 \rceil = 2$$. Windows also keep the attention cost linear in $$N$$: each window costs $$512^2$$ per head and layer, so $$k \times 512^2$$ in total, against $$(N + 2)^2$$ for one long input (if the model could take it):

```text
N =    720: windows =   2; one long input (N+2)^2 =     521,284; windows k*512^2 =     524,288; ratio 0.99
N =   2000: windows =   5; one long input (N+2)^2 =   4,008,004; windows k*512^2 =   1,310,720; ratio 3.06
N =  10000: windows =  26; one long input (N+2)^2 = 100,040,004; windows k*512^2 =   6,815,744; ratio 14.68
```

<figure class="fig"><svg viewBox="0 0 760 440" role="img" aria-label="A 720-token text is cut into two 512-token windows that overlap by 128 tokens. Below, the number of attention entries for one long input grows with the square of the length, while windows grow in a straight line."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">A 720-token text, read as 2 overlapping windows of 512</text><rect class="box" x="40.0" y="46.0" width="680.0" height="26.0" rx="5"/><text class="t-tick" x="380.0" y="64.0" text-anchor="middle">the whole text: 720 tokens (BERT has position vectors for only 512)</text><rect class="box-1" x="40.0" y="92.0" width="481.7" height="26.0" rx="5"/><text class="t-tick" x="48.0" y="110.0" text-anchor="start">window 1: tokens 0-509 (510 + [CLS] + [SEP])</text><rect class="box-2" x="400.8" y="132.0" width="319.2" height="26.0" rx="5"/><text class="t-tick" x="408.8" y="150.0" text-anchor="start">window 2: tokens 382-719 (338 + [CLS] + [SEP])</text><rect class="box-on" x="400.8" y="160.0" width="120.9" height="14.0" rx="3"/><text class="t-on" x="461.2" y="192.0" text-anchor="middle">overlap o = 128 tokens, seen by both windows</text><text class="t-muted" x="40.0" y="192.0" text-anchor="start">step s = 382</text><text class="t-title" x="20.0" y="236.0" text-anchor="start">Attention entries per head per layer: one long input vs windows of 512</text><text class="t-note" x="130.0" y="270.0" text-anchor="end">N = 720</text><text class="t-muted" x="130.0" y="288.0" text-anchor="end">2 windows</text><g class="mark"><title>one long input: 521,284</title><rect class="s2" x="140" y="256" width="3.0" height="18" rx="3"/></g><text class="t-tick" x="149.0" y="270.0" text-anchor="start">521,284 (one long input)</text><g class="mark"><title>windows: 524,288</title><rect class="s1" x="140" y="278" width="3.0" height="18" rx="3"/></g><text class="t-tick" x="149.0" y="292.0" text-anchor="start">524,288 (windows)</text><text class="t-note" x="130.0" y="328.0" text-anchor="end">N = 2,000</text><text class="t-muted" x="130.0" y="346.0" text-anchor="end">5 windows</text><g class="mark"><title>one long input: 4,008,004</title><rect class="s2" x="140" y="314" width="17.6" height="18" rx="3"/></g><text class="t-tick" x="163.6" y="328.0" text-anchor="start">4,008,004 (one long input)</text><g class="mark"><title>windows: 1,310,720</title><rect class="s1" x="140" y="336" width="5.8" height="18" rx="3"/></g><text class="t-tick" x="151.8" y="350.0" text-anchor="start">1,310,720 (windows)</text><text class="t-note" x="130.0" y="386.0" text-anchor="end">N = 10,000</text><text class="t-muted" x="130.0" y="404.0" text-anchor="end">26 windows</text><g class="mark"><title>one long input: 100,040,004</title><rect class="s2" x="140" y="372" width="440.0" height="18" rx="3"/></g><text class="t-tick" x="586.0" y="386.0" text-anchor="start">100,040,004 (one long input)</text><g class="mark"><title>windows: 6,815,744</title><rect class="s1" x="140" y="394" width="30.0" height="18" rx="3"/></g><text class="t-tick" x="176.0" y="408.0" text-anchor="start">6,815,744 (windows)</text></svg><figcaption>Reading a 720-token text as two overlapping windows of 512. Window 1 covers tokens 0 to 509, window 2 tokens 382 to 719; the 128 tokens in between are seen by both. Below: attention entries per head and layer, one long input against windows, for 720, 2,000 and 10,000 tokens.</figcaption></figure>

The price: a token in window 1 never sees a token in window 2. Facts that are far apart in a long document cannot be combined. That is the real cost of the 512 wall, and why later models raised it.

### Limit 2: `[MASK]` never appears after pre-training

> [!PAPER] · §3.1
> [![Section 3.1: a downside is that we are creating a mismatch between pre-training and fine-tuning, since the MASK token does not appear during fine-tuning](https://ishwarj.com/img/papers/bert/p6-mismatch.pngert/p6-mismatch.png)](/img/papers/bert/p6-mismatch.png)
>
> **What it says:** "a downside is that we are creating a mismatch between pre-training and fine-tuning, since the [MASK] token does not appear during fine-tuning."
>
> **What it means:**
>
> The authors saw this limit themselves. Think of a pilot who trained in a simulator with a warning light that the real plane does not have. Any habit tied to that light is useless in real flight.
>
> Their fix, the 80/10/10 rule of Part 3, softens the problem but does not remove it.

During pre-training, BERT sees `[MASK]` in about 12% of positions (80% of the 15% chosen). In real use it never sees `[MASK]` at all. Part 3 explained the 80% / 10% / 10% trick that reduces the gap, and Part 5 (Table 8) measured how much it matters. XLNet named this the "pretrain-finetune discrepancy".

### Limit 3: only 15% of tokens teach the model

> [!PAPER] · §A.1
> [![Appendix A.1: compared to standard language model training, the masked LM only makes predictions on 15 percent of tokens in each batch, which suggests that more pre-training steps may be required for the model to converge](https://ishwarj.com/img/papers/bert/p6-fifteen.pngbert/p6-fifteen.png)](/img/papers/bert/p6-fifteen.png)
>
> **What it says:** the masked LM "only make[s] predictions on 15% of tokens in each batch", so "more pre-training steps may be required".
>
> **What it means:**
>
> In a left-to-right model, every token is a prediction target. In BERT, only the 15% chosen tokens are, so 85% of the tokens give no training signal of their own.
>
> Think of a quiz with 100 questions where only 15 are marked. You still learn, but you need more quizzes to learn the same amount. That is the "more pre-training steps" of the sentence.

<figure class="fig"><svg viewBox="0 0 760 252" role="img" aria-label="Where the training signal comes from. In BERT only the masked positions are scored. In ELECTRA every position is scored as real or replaced."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">BERT (masked LM): only the hidden tokens give a training signal</text><text class="t-title" x="20.0" y="134.0" text-anchor="start">ELECTRA (replaced token detection): every token gives a signal</text><rect class="box" x="20.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="41.0" y="59.5" text-anchor="middle">the</text><rect class="box" x="67.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="88.0" y="59.5" text-anchor="middle">cat</text><rect class="box-mask" x="114.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="135.0" y="59.5" text-anchor="middle">[M]</text><rect class="box" x="161.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="182.0" y="59.5" text-anchor="middle">on</text><rect class="box" x="208.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="229.0" y="59.5" text-anchor="middle">the</text><rect class="box" x="255.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="276.0" y="59.5" text-anchor="middle">mat</text><rect class="box" x="302.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="323.0" y="59.5" text-anchor="middle">and</text><rect class="box" x="349.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="370.0" y="59.5" text-anchor="middle">looked</text><rect class="box" x="396.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="417.0" y="59.5" text-anchor="middle">out</text><rect class="box-mask" x="443.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="464.0" y="59.5" text-anchor="middle">[M]</text><rect class="box" x="490.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="511.0" y="59.5" text-anchor="middle">the</text><rect class="box" x="537.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="558.0" y="59.5" text-anchor="middle">window</text><rect class="box" x="584.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="605.0" y="59.5" text-anchor="middle">at</text><rect class="box" x="631.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="652.0" y="59.5" text-anchor="middle">the</text><rect class="box" x="678.0" y="40.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="699.0" y="59.5" text-anchor="middle">birds</text><line class="edge" x1="135.0" y1="72.0" x2="135.0" y2="90.0" marker-end="url(#ah)"/><text class="t-on" x="135.0" y="106.0" text-anchor="middle">loss</text><line class="edge" x1="464.0" y1="72.0" x2="464.0" y2="90.0" marker-end="url(#ah)"/><text class="t-on" x="464.0" y="106.0" text-anchor="middle">loss</text><rect class="box" x="20.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="41.0" y="169.5" text-anchor="middle">the</text><rect class="box" x="67.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="88.0" y="169.5" text-anchor="middle">cat</text><rect class="box" x="114.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="135.0" y="169.5" text-anchor="middle">sat</text><rect class="box" x="161.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="182.0" y="169.5" text-anchor="middle">on</text><rect class="box" x="208.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="229.0" y="169.5" text-anchor="middle">the</text><rect class="box-2" x="255.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="276.0" y="169.5" text-anchor="middle">rug</text><rect class="box" x="302.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="323.0" y="169.5" text-anchor="middle">and</text><rect class="box" x="349.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="370.0" y="169.5" text-anchor="middle">looked</text><rect class="box" x="396.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="417.0" y="169.5" text-anchor="middle">out</text><rect class="box" x="443.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="464.0" y="169.5" text-anchor="middle">of</text><rect class="box" x="490.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="511.0" y="169.5" text-anchor="middle">the</text><rect class="box" x="537.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="558.0" y="169.5" text-anchor="middle">window</text><rect class="box" x="584.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="605.0" y="169.5" text-anchor="middle">at</text><rect class="box" x="631.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="652.0" y="169.5" text-anchor="middle">the</text><rect class="box" x="678.0" y="150.0" width="42.0" height="30.0" rx="6"/><text class="t-muted" x="699.0" y="169.5" text-anchor="middle">birds</text><line class="edge" x1="41.0" y1="182.0" x2="41.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="41.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="88.0" y1="182.0" x2="88.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="88.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="135.0" y1="182.0" x2="135.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="135.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="182.0" y1="182.0" x2="182.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="182.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="229.0" y1="182.0" x2="229.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="229.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="276.0" y1="182.0" x2="276.0" y2="198.0" marker-end="url(#ah)"/><text class="t-bad" x="276.0" y="214.0" text-anchor="middle">fake</text><line class="edge" x1="323.0" y1="182.0" x2="323.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="323.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="370.0" y1="182.0" x2="370.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="370.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="417.0" y1="182.0" x2="417.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="417.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="464.0" y1="182.0" x2="464.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="464.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="511.0" y1="182.0" x2="511.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="511.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="558.0" y1="182.0" x2="558.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="558.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="605.0" y1="182.0" x2="605.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="605.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="652.0" y1="182.0" x2="652.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="652.0" y="214.0" text-anchor="middle">real</text><line class="edge" x1="699.0" y1="182.0" x2="699.0" y2="198.0" marker-end="url(#ah)"/><text class="t-s3" x="699.0" y="214.0" text-anchor="middle">real</text><text class="t-muted" x="20.0" y="240.0" text-anchor="start">The paper masks 15% of tokens (about 2 of these 15). ELECTRA asks "real or replaced?" at every position.</text></svg><figcaption>Where the learning signal comes from. In BERT's masked LM only the masked positions get a loss. In ELECTRA every position is checked as real or replaced (shown here as "fake"), so every token teaches the model something.</figcaption></figure>

Part 5 (Appendix C.1) showed that BERT does learn a little more slowly because of this, but still wins. ELECTRA turned this observation into a new pre-training task.

> [!DEFINITION] Loss
> A number that measures how wrong the model's predictions are. Training changes the weights to make the loss smaller. A position "gets a loss" if its prediction is scored and used for learning.

### Limit 4: two blanks are filled separately

When BERT fills several `[MASK]`s at once, it predicts each one **on its own**. It does not know what it put in the other blank. I masked both words of a two-word city name:

```text
3. Two [MASK]s at once: each one is predicted on its own
   i flew from [MASK] [MASK] to london last week.
     mask 1: new 0.042, the 0.037, chicago 0.037, la 0.028, portland 0.019
     mask 2: town 0.068, york 0.044, beach 0.025, city 0.021, ##town 0.017
   i flew from new [MASK] to london last week.  -> york 0.979, orleans 0.012, jersey 0.003
   i flew from los [MASK] to london last week.  -> angeles 1.000, vegas 0.000, ' 0.000
   i flew from hong [MASK] to london last week. -> kong 0.999, ##kong 0.001, china 0.000
```

Taking the top guess for each blank separately gives "new town". But once the first word is filled in, the second becomes almost certain: "york" (0.979) after "new", "angeles" after "los", "kong" after "hong". The two blanks depend on each other, and BERT's masked LM cannot use that when both are hidden. This is what XLNet meant by "neglects dependency between the masked positions".

In equations, the masked LM scores the two blanks $$a$$ and $$b$$ as if they were independent, while the true probability follows the chain rule of Part 1:

$$
\underbrace{p(a)\, p(b)}_{\text{what BERT's two guesses give}} \qquad \text{versus} \qquad \underbrace{p(a)\, p(b \mid a)}_{\text{the chain rule}}
$$

```text
"new town":  p(a) = 0.0424, p(b) = 0.0678, p(b | a) = 0.0000
   independent p(a) p(b) = 0.002876;  chain rule p(a) p(b | a) = 0.000000
"new york":  p(a) = 0.0424, p(b) = 0.0440, p(b | a) = 0.9794
   independent p(a) p(b) = 0.001868;  chain rule p(a) p(b | a) = 0.041548
best pair by the independent score (over 29 candidate pairs): "new town" 0.002876
best pair by the chain rule:                                  "new york" 0.041548
```

<figure class="fig"><svg viewBox="0 0 760 378" role="img" aria-label="Two blanks in four steps. With both blanks hidden, BERT guesses each one on its own, and multiplying the two guesses favours new town. Filling the first blank with new makes york almost certain, and the chain rule picks new york."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="10.0" y="10.0" width="362.0" height="170.0" rx="12"/><circle class="node on" cx="28.0" cy="28.0" r="11"/><text class="t-note" x="28.0" y="32.5" text-anchor="middle">1</text><text class="t-title" x="46.0" y="33.0" text-anchor="start">both blanks hidden</text><rect class="box" x="388.0" y="10.0" width="362.0" height="170.0" rx="12"/><circle class="node on" cx="406.0" cy="28.0" r="11"/><text class="t-note" x="406.0" y="32.5" text-anchor="middle">2</text><text class="t-title" x="424.0" y="33.0" text-anchor="start">multiply the two guesses</text><rect class="box" x="10.0" y="196.0" width="362.0" height="170.0" rx="12"/><circle class="node on" cx="28.0" cy="214.0" r="11"/><text class="t-note" x="28.0" y="218.5" text-anchor="middle">3</text><text class="t-title" x="46.0" y="219.0" text-anchor="start">fill in the first blank</text><rect class="box" x="388.0" y="196.0" width="362.0" height="170.0" rx="12"/><circle class="node on" cx="406.0" cy="214.0" r="11"/><text class="t-note" x="406.0" y="218.5" text-anchor="middle">4</text><text class="t-title" x="424.0" y="219.0" text-anchor="start">the chain rule</text><rect class="box" x="26.0" y="54.0" width="70.0" height="26.0" rx="6"/><text class="t-muted" x="61.0" y="71.5" text-anchor="middle">from</text><rect class="box-mask" x="102.0" y="54.0" width="70.0" height="26.0" rx="6"/><text class="t-muted" x="137.0" y="71.5" text-anchor="middle">[MASK]</text><rect class="box-mask" x="178.0" y="54.0" width="70.0" height="26.0" rx="6"/><text class="t-muted" x="213.0" y="71.5" text-anchor="middle">[MASK]</text><rect class="box" x="254.0" y="54.0" width="60.0" height="26.0" rx="6"/><text class="t-muted" x="284.0" y="71.5" text-anchor="middle">to</text><text class="t-tick" x="137.0" y="102.0" text-anchor="middle">blank 1</text><text class="t-tick" x="213.0" y="102.0" text-anchor="middle">blank 2</text><text class="t-tick" x="137.0" y="122.0" text-anchor="middle">new 0.042</text><text class="t-tick" x="137.0" y="138.0" text-anchor="middle">the 0.037</text><text class="t-tick" x="137.0" y="154.0" text-anchor="middle">chicago 0.037</text><text class="t-tick" x="213.0" y="122.0" text-anchor="middle">town 0.068</text><text class="t-tick" x="213.0" y="138.0" text-anchor="middle">york 0.044</text><text class="t-tick" x="213.0" y="154.0" text-anchor="middle">beach 0.025</text><text class="t-tick" x="406.0" y="70.0" text-anchor="start">independent score p(a) · p(b):</text><text class="t-mono" x="406.0" y="98.0" text-anchor="start">new town  0.0424 × 0.0678 = 0.0029</text><text class="t-mono" x="406.0" y="122.0" text-anchor="start">new york  0.0424 × 0.0440 = 0.0019</text><text class="t-bad" x="406.0" y="154.0" text-anchor="start">the winner is "new town", a place nobody flew from</text><rect class="box" x="26.0" y="240.0" width="70.0" height="26.0" rx="6"/><text class="t-muted" x="61.0" y="257.5" text-anchor="middle">from</text><rect class="box-on" x="102.0" y="240.0" width="70.0" height="26.0" rx="6"/><text class="t-tick" x="137.0" y="257.5" text-anchor="middle">new</text><rect class="box-mask" x="178.0" y="240.0" width="70.0" height="26.0" rx="6"/><text class="t-muted" x="213.0" y="257.5" text-anchor="middle">[MASK]</text><rect class="box" x="254.0" y="240.0" width="60.0" height="26.0" rx="6"/><text class="t-muted" x="284.0" y="257.5" text-anchor="middle">to</text><text class="t-tick t-strong" x="213.0" y="294.0" text-anchor="middle">york 0.979</text><text class="t-tick" x="213.0" y="310.0" text-anchor="middle">orleans 0.012</text><text class="t-tick" x="213.0" y="326.0" text-anchor="middle">jersey 0.003</text><text class="t-muted" x="28.0" y="346.0" text-anchor="start">knowing blank 1 makes blank 2 almost certain</text><text class="t-tick" x="406.0" y="256.0" text-anchor="start">chain rule p(a) · p(b | a):</text><text class="t-mono" x="406.0" y="284.0" text-anchor="start">new york  0.0424 × 0.9794 = 0.0415</text><text class="t-mono" x="406.0" y="308.0" text-anchor="start">new town  0.0424 × 0.000012 ≈ 0</text><text class="t-s3" x="406.0" y="340.0" text-anchor="start">"new york" wins by far</text></svg><figcaption>The two-blank problem in four panels. Both blanks hidden: BERT guesses "new" for blank 1 and "town" for blank 2. Multiplying the two guesses picks "new town". Filling blank 1 first makes blank 2 "york" with 0.979. The chain rule picks "new york" by far.</figcaption></figure>

> [!NOTE] Why "1.000" and "0.000"?
> The probabilities are rounded to three decimals. "1.000" means more than 0.9995, not exactly 1; "0.000" means less than 0.0005.

### Limit 5: no ready-made sentence vector

> [!PAPER] · §3.1, footnote 6
> [![Footnote 6: the vector C is not a meaningful sentence representation without fine-tuning, since it was trained with NSP](https://ishwarj.com/img/papers/bert/p6-footnote6.pngrt/p6-footnote6.png)](/img/papers/bert/p6-footnote6.png)
>
> **What it says:** "The vector C is not a meaningful sentence representation without fine-tuning, since it was trained with NSP."
>
> **What it means:**
>
> The `[CLS]` vector was trained for one job: to say whether sentence B follows sentence A. Nobody trained it to put similar sentences close together. So, without fine-tuning, it is not a summary of meaning.
>
> Many people still take BERT's `[CLS]` vector as "the meaning of the sentence". The paper warns against exactly that, and the measurement below shows why.

> [!DEFINITION] Cosine similarity
> A score for how much two vectors point the same way: 1 means the same direction, 0 means unrelated (at a right angle), and negative means opposite. It is the standard way to compare sentence vectors.

The formula, $$\cos(a, b) = a \cdot b \,/\, (\lVert a \rVert \lVert b \rVert)$$, on one real pair of `[CLS]` vectors:

```text
a = C("A man is playing a guitar."), b = C("A person is making music."), each with 768 numbers
a . b = 199.086;  |a| = 15.498;  |b| = 15.086
cos = a . b / (|a| |b|) = 199.086 / (15.498 x 15.086) = 0.851   (torch: 0.851)
toy check in 2-D: a = (3, 4), b = (4, 3): a . b = 24, |a| = |b| = 5, cos = 24 / 25 = 0.96
```

<figure class="fig"><svg viewBox="0 0 760 262" role="img" aria-label="Cosine similarity measures the angle between two vectors. In two dimensions, (3, 4) and (4, 3) have cosine 0.96. For two real 768-number [CLS] vectors of BERT, the dot product 199.086 divided by the two lengths gives 0.851."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Cosine similarity: the angle between two vectors</text><line class="edge-dim" x1="70.0" y1="220.0" x2="286.0" y2="220.0"/><line class="edge-dim" x1="70.0" y1="220.0" x2="70.0" y2="22.0"/><line class="edge-1" x1="70.0" y1="220.0" x2="178.0" y2="76.0" marker-end="url(#ah)"/><line class="edge-2" x1="70.0" y1="220.0" x2="214.0" y2="112.0" marker-end="url(#ah)"/><text class="t-s1" x="174.0" y="66.0" text-anchor="middle">a = (3, 4)</text><text class="t-s2" x="226.0" y="112.0" text-anchor="start">b = (4, 3)</text><path class="edge-on" d="M95.9,185.4 A43.2,43.2 0 0 1 104.6,194.1"/><text class="t-on" x="126.0" y="174.0" text-anchor="start">θ</text><text class="t-tick" x="70.0" y="246.0" text-anchor="start">toy, in 2-D: a · b = 3·4 + 4·3 = 24, |a| = |b| = 5, cos θ = 24 / 25 = 0.96</text><text class="t-note" x="380.0" y="60.0" text-anchor="start">Real, in 768-D: [CLS] vectors of bert-base-uncased</text><text class="t-tick" x="380.0" y="84.0" text-anchor="start">a = C("A man is playing a guitar.")</text><text class="t-tick" x="380.0" y="102.0" text-anchor="start">b = C("A person is making music.")</text><rect class="box-ghost" x="377.0" y="113.0" width="116.0" height="18.0" rx="4"/><g class="mark"><rect class="s1" x="380.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s1" x="394.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s1" x="408.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s1" x="422.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.49"/></g><g class="mark"><rect class="s1" x="436.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.49"/></g><g class="mark"><rect class="s1" x="450.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="464.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s1" x="478.0" y="116.0" width="12" height="12" rx="2" style="fill-opacity:0.80"/></g><text class="t-muted" x="500.0" y="126.0" text-anchor="start">... 768 numbers</text><rect class="box-ghost" x="377.0" y="133.0" width="116.0" height="18.0" rx="4"/><g class="mark"><rect class="s2" x="380.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.35"/></g><g class="mark"><rect class="s2" x="394.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.51"/></g><g class="mark"><rect class="s2" x="408.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.58"/></g><g class="mark"><rect class="s2" x="422.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.28"/></g><g class="mark"><rect class="s2" x="436.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s2" x="450.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="464.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s2" x="478.0" y="136.0" width="12" height="12" rx="2" style="fill-opacity:0.37"/></g><text class="t-muted" x="500.0" y="146.0" text-anchor="start">... 768 numbers</text><text class="t-mono" x="380.0" y="180.0" text-anchor="start">a · b = 199.086</text><text class="t-mono" x="380.0" y="200.0" text-anchor="start">|a| = 15.498,  |b| = 15.086</text><text class="t-mono" x="380.0" y="226.0" text-anchor="start">cos = 199.086 / (15.498 × 15.086) = 0.851</text></svg><figcaption>Cosine similarity is the angle between two vectors. Toy, in 2-D: (3, 4) and (4, 3) have cos 0.96. Real, in 768-D: the raw [CLS] vectors of "A man is playing a guitar." and "A person is making music." have cos 0.851.</figcaption></figure>

> [!DEFINITION] Sentence embedding
> One vector for a whole sentence, built so that sentences with similar meanings get vectors that point in similar directions. Search engines and chat-with-your-documents systems compare such vectors to find related text.

I compared six sentence pairs, three related and three unrelated, with three kinds of vector:

- **raw BERT [CLS]**: the vector C of `bert-base-uncased`, as the footnote warns against;
- **raw BERT, mean of tokens**: the average of all of BERT's output vectors;
- **all-MiniLM-L6-v2**: a public model from the Sentence-BERT authors' library. Its config says `model_type=bert`, so it has BERT's architecture, but it is a small distilled model (6 layers, 384 numbers per token, 22.7M parameters, from the script's output), fine-tuned on over 1 billion sentence pairs (its model card) to make similar sentences close.

```text
   pair (R = related, U = unrelated)                                   raw [CLS]  raw mean  MiniLM
   R A man is playing a guitar.   | A person is making music.              0.851     0.746   0.590
   R How do I reset my password?  | I forgot my login details.             0.929     0.672   0.541
   R The weather is lovely today. | It is sunny and warm outside.          0.951     0.857   0.705
   U A man is playing a guitar.   | The stock market fell sharply.         0.776     0.584  -0.064
   U How do I reset my password?  | Penguins live in Antarctica.           0.795     0.517  -0.035
   U The weather is lovely today. | He parked the truck in the garage.     0.873     0.598   0.013
   average related / unrelated:  raw [CLS] 0.910 / 0.815,  raw mean 0.758 / 0.566,  MiniLM 0.612 / -0.029
```

<figure class="fig"><svg viewBox="0 0 760 274" role="img" aria-label="Cosine similarities for three related and three unrelated sentence pairs. With raw BERT the two groups overlap. The model fine-tuned for similarity separates them cleanly."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Cosine similarity of six sentence pairs</text><line class="grid" x1="220.0" y1="50" x2="220.0" y2="218"/><text class="t-tick" x="220.0" y="236.0" text-anchor="middle">-0.2</text><line class="grid" x1="303.3" y1="50" x2="303.3" y2="218"/><text class="t-tick" x="303.3" y="236.0" text-anchor="middle">0.0</text><line class="grid" x1="386.7" y1="50" x2="386.7" y2="218"/><text class="t-tick" x="386.7" y="236.0" text-anchor="middle">0.2</text><line class="grid" x1="470.0" y1="50" x2="470.0" y2="218"/><text class="t-tick" x="470.0" y="236.0" text-anchor="middle">0.4</text><line class="grid" x1="553.3" y1="50" x2="553.3" y2="218"/><text class="t-tick" x="553.3" y="236.0" text-anchor="middle">0.6</text><line class="grid" x1="636.7" y1="50" x2="636.7" y2="218"/><text class="t-tick" x="636.7" y="236.0" text-anchor="middle">0.8</text><line class="grid" x1="720.0" y1="50" x2="720.0" y2="218"/><text class="t-tick" x="720.0" y="236.0" text-anchor="middle">1.0</text><text class="t-tick" x="206.0" y="82.0" text-anchor="end">raw BERT [CLS]</text><g class="mark"><title>A man is playing a guitar. | A person is making music.: 0.851</title><circle class="s3 ring" cx="657.9" cy="71" r="6"/></g><g class="mark"><title>How do I reset my password? | I forgot my login details.: 0.929</title><circle class="s3 ring" cx="690.4" cy="71" r="6"/></g><g class="mark"><title>The weather is lovely today. | It is sunny and warm outside.: 0.951</title><circle class="s3 ring" cx="699.6" cy="71" r="6"/></g><g class="mark"><title>A man is playing a guitar. | The stock market fell sharply.: 0.776</title><circle class="s2 ring" cx="626.7" cy="85" r="6"/></g><g class="mark"><title>How do I reset my password? | Penguins live in Antarctica.: 0.795</title><circle class="s2 ring" cx="634.6" cy="85" r="6"/></g><g class="mark"><title>The weather is lovely today. | He parked the truck in the garage.: 0.873</title><circle class="s2 ring" cx="667.1" cy="85" r="6"/></g><text class="t-tick" x="206.0" y="138.0" text-anchor="end">raw BERT, mean of tokens</text><g class="mark"><title>A man is playing a guitar. | A person is making music.: 0.746</title><circle class="s3 ring" cx="614.2" cy="127" r="6"/></g><g class="mark"><title>How do I reset my password? | I forgot my login details.: 0.672</title><circle class="s3 ring" cx="583.3" cy="127" r="6"/></g><g class="mark"><title>The weather is lovely today. | It is sunny and warm outside.: 0.857</title><circle class="s3 ring" cx="660.4" cy="127" r="6"/></g><g class="mark"><title>A man is playing a guitar. | The stock market fell sharply.: 0.584</title><circle class="s2 ring" cx="546.7" cy="141" r="6"/></g><g class="mark"><title>How do I reset my password? | Penguins live in Antarctica.: 0.517</title><circle class="s2 ring" cx="518.8" cy="141" r="6"/></g><g class="mark"><title>The weather is lovely today. | He parked the truck in the garage.: 0.598</title><circle class="s2 ring" cx="552.5" cy="141" r="6"/></g><text class="t-tick" x="206.0" y="194.0" text-anchor="end">all-MiniLM-L6-v2 (fine-tuned)</text><g class="mark"><title>A man is playing a guitar. | A person is making music.: 0.590</title><circle class="s3 ring" cx="549.2" cy="183" r="6"/></g><g class="mark"><title>How do I reset my password? | I forgot my login details.: 0.541</title><circle class="s3 ring" cx="528.8" cy="183" r="6"/></g><g class="mark"><title>The weather is lovely today. | It is sunny and warm outside.: 0.705</title><circle class="s3 ring" cx="597.1" cy="183" r="6"/></g><g class="mark"><title>A man is playing a guitar. | The stock market fell sharply.: -0.064</title><circle class="s2 ring" cx="276.7" cy="197" r="6"/></g><g class="mark"><title>How do I reset my password? | Penguins live in Antarctica.: -0.035</title><circle class="s2 ring" cx="288.8" cy="197" r="6"/></g><g class="mark"><title>The weather is lovely today. | He parked the truck in the garage.: 0.013</title><circle class="s2 ring" cx="308.8" cy="197" r="6"/></g><circle class="s3" cx="230" cy="258" r="6"/><text class="t-tick" x="242.0" y="262.0" text-anchor="start">related pair</text><circle class="s2" cx="350" cy="258" r="6"/><text class="t-tick" x="362.0" y="262.0" text-anchor="start">unrelated pair</text><text class="t-muted" x="480.0" y="262.0" text-anchor="start">cosine similarity (1 = same direction)</text></svg><figcaption>The same numbers as a picture. With raw BERT's [CLS] vector, every pair looks similar (0.776 to 0.951), and an unrelated pair (0.873) scores higher than a related one (0.851). The model fine-tuned for similarity puts all unrelated pairs near 0 and all related pairs above 0.5.</figcaption></figure>

What the numbers say:

- **Raw [CLS] says everything is similar.** Even "the weather is lovely" and "he parked the truck in the garage" get 0.873, higher than the related guitar pair (0.851). You cannot pick a threshold that separates the two groups. The footnote is right.
- **Averaging BERT's outputs is better but not clean**: related pairs average 0.758 and unrelated 0.566, but the gap is small.
- **A model fine-tuned for similarity separates them clearly**: unrelated pairs score between -0.064 and 0.013, related pairs between 0.541 and 0.705.

This is only six pairs, so treat it as an illustration, not a benchmark. Sentence-BERT's paper measures the same effect on standard similarity datasets.

### Limit 6: it does not write text

BERT is an encoder, and footnote 4 of the paper (Part 2) says what that means for writing text.

> [!PAPER] · §3, footnote 4
> [![Footnote 4 of the BERT paper: the bidirectional Transformer is often referred to as a Transformer encoder while the left-context-only version is referred to as a Transformer decoder since it can be used for text generation](https://ishwarj.com/img/papers/bert/p6-footnote4.pngrt/p6-footnote4.png)](/img/papers/bert/p6-footnote4.png)
>
> **What it says:** the bidirectional version "is often referred to as a 'Transformer encoder'", while the left-context-only version is a "'Transformer decoder' since it can be used for text generation".
>
> **What it means:**
>
> The paper itself draws the line. An encoder reads the whole input at once and gives back one vector per token. A decoder sees only the words to its left, so it can write text one word at a time.
>
> BERT is on the encoder side of that line by design. It has no natural way to write a long answer word by word. Chatbots are built on decoders, in the line of GPT, not on BERT.

### Limit 7: pre-training is expensive

> [!PAPER] · §A.2
> [![Appendix A.2: training of BERT-base was performed on 4 Cloud TPUs in Pod configuration, 16 TPU chips total; BERT-large on 16 Cloud TPUs, 64 TPU chips total; each pre-training took 4 days to complete](https://ishwarj.com/img/papers/bert/p6-compute.pngbert/p6-compute.png)](/img/papers/bert/p6-compute.png)
>
> **What it says:** BERT-base was trained on "4 Cloud TPUs in Pod configuration (16 TPU chips total)", BERT-large on "16 Cloud TPUs (64 TPU chips total)", and "each pre-training took 4 days to complete".
>
> **What it means:**
>
> 16 chips for 4 days is 64 chip-days for BERT-base; 64 chips for 4 days is 256 chip-days for BERT-large. In 2018, few people outside large labs had that much compute.
>
> That is why the public release of the weights mattered so much. Most people could never pre-train BERT, but anyone could download it and fine-tune it.

> [!DEFINITION] TPU
> Tensor Processing Unit: a chip designed by Google for neural network maths, similar in role to a GPU.

The good news is the paper's own: fine-tuning is cheap. Section 3.2 says every result in the paper "can be replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU, starting from the exact same pre-trained model" (Part 4).

> [!PAPER] · §3.2
> [![From Section 3.2: compared to pre-training, fine-tuning is relatively inexpensive; all of the results in the paper can be replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU, starting from the exact same pre-trained model](https://ishwarj.com/img/papers/bert/p6-finetune-cost.png6-finetune-cost.png)](/img/papers/bert/p6-finetune-cost.png)
>
> **What it says:** "All of the results in the paper can be replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU, starting from the exact same pre-trained model."
>
> **What it means:**
>
> Think of a bridge. One company pays to build it, and then everyone crosses it for a small toll. Pre-training is the bridge; fine-tuning is the toll.
>
> The expensive step is done once, by someone else. Everyone downstream pays only the cheap step: at most an hour on one Cloud TPU.

How big was the pre-training budget, in tokens? From the numbers in Appendix A.2:

```text
tokens per batch = 256 x 512 = 131,072  (the paper rounds to 128,000)
tokens seen      = 1,000,000 steps x 131,072 = 131,072,000,000  (an upper bound: 90% of steps use length 128)
with 90% of steps at length 128 and 10% at 512: 1,000,000 x 256 x (0.9 x 128 + 0.1 x 512) = 42,598,400,000
predicted tokens (15%) of that: 6,389,760,000
chip-days: BERT-base 16 chips x 4 days = 64; BERT-large 64 chips x 4 days = 256
```

One subtlety: the paper's "approximately 40 epochs over the 3.3 billion word corpus" matches the upper bound (131 billion tokens ÷ 3.3 billion words ≈ 40). If the batch stayed at 256 sequences during the length-128 phase, the model saw about 42.6 billion tokens, roughly 13 passes. The paper does not say whether the batch size changed between the two phases, so the exact number of epochs cannot be recovered from the text.

## How BERT-style models are used today

Large chatbots get the headlines, but small encoders like BERT do a lot of quiet work, because they are fast and cheap to run on every piece of text. ModernBERT's authors call them "the workhorse of numerous production pipelines". Typical jobs:

- **Classification**: spam or not, which team should handle this support ticket, is this review positive. This is the GLUE recipe from Part 4: `[CLS]` vector, one layer, fine-tune.
- **Finding names and things (NER)**: people, places, products, drug names in text. This is the token-level recipe from Part 4.
- **Search and retrieval**: sentence-embedding models in the Sentence-BERT line turn every document into a vector once, then compare vectors at search time. This is how many "chat with your documents" systems find the right passages.
- **Reranking**: a **cross-encoder** reads the question and one candidate passage together, exactly like BERT's sentence pairs, and scores how well they match. Sentence-BERT describes BERT itself as a cross-encoder: "Two sentences are passed to the transformer network". It is accurate but slow, so it is used to re-order a short list found by the faster vector search.

> [!DEFINITION] Cross-encoder and bi-encoder
> A **cross-encoder** reads two texts together in one input (like BERT's sentence A and sentence B) and outputs one score. A **bi-encoder** turns each text into its own vector separately, so vectors can be computed ahead of time and compared quickly. Sentence-BERT made BERT into a good bi-encoder.

Here is the NER use case with a real public checkpoint, `dslim/bert-base-NER`. Its model card says it is `bert-base-cased` fine-tuned on the CoNLL-2003 dataset (the same NER task as the paper's Section 5.3), with four entity types: person (PER), location (LOC), organisation (ORG) and miscellaneous (MISC).

```python
from transformers import pipeline
ner = pipeline("ner", model="dslim/bert-base-NER", aggregation_strategy="simple")
ner("Ada Lovelace was born in London and worked with Charles Babbage on the Analytical Engine.")
```

```text
4. Named entities with dslim/bert-base-NER (BERT-base-cased fine-tuned on CoNLL-2003)
   Ada Lovelace was born in London and worked with Charles Babbage on the Analytical Engine.
     Ada Lovelace           PER   0.999
     London                 LOC   0.999
     Charles Babbage        PER   0.932
     Analy                  ORG   0.723
     Engine                 ORG   0.781
   token by token, for "Analytical Engine":
     Ana                    B-ORG  0.901
     ##ly                   I-ORG  0.544
     Engine                 I-ORG  0.781
```

The people and the place are found with high confidence. "Analytical Engine" (a machine, not really an organisation) shows a real rough edge: WordPiece split "Analytical" into `Ana`, `##ly` and `##tical`, the last piece got no entity label, and the result came out broken into "Analy" and "Engine". The model card warns about exactly this: the model "occassionally tags subword tokens as entities". This is why Section 5.3 of the paper labels only the **first** sub-token of each word (Part 5).

## The whole paper on one page

<figure class="fig"><svg viewBox="0 0 760 436" role="img" aria-label="BERT on a napkin. Text goes in as WordPiece tokens with [CLS] and [SEP]. A 12-layer bidirectional encoder is pre-trained with masked LM and next sentence prediction. For each task, one small layer is added on top and everything is fine-tuned."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">1. Input (Part 2)</text><text class="t-title" x="20.0" y="150.0" text-anchor="start">2. Pre-training, two tasks at once (Part 3)</text><text class="t-title" x="20.0" y="330.0" text-anchor="start">3. Fine-tuning: same body, one new layer per task (Part 4)</text><rect class="box-on" x="20.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="49.0" y="59.5" text-anchor="middle">[CLS]</text><rect class="box-1" x="84.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="113.0" y="59.5" text-anchor="middle">my</text><rect class="box-1" x="148.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="177.0" y="59.5" text-anchor="middle">dog</text><rect class="box-1" x="212.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="241.0" y="59.5" text-anchor="middle">is</text><rect class="box-mask" x="276.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="305.0" y="59.5" text-anchor="middle">[MASK]</text><rect class="box" x="340.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="369.0" y="59.5" text-anchor="middle">[SEP]</text><rect class="box-2" x="404.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="433.0" y="59.5" text-anchor="middle">he</text><rect class="box-2" x="468.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="497.0" y="59.5" text-anchor="middle">likes</text><rect class="box-2" x="532.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="561.0" y="59.5" text-anchor="middle">play</text><rect class="box-2" x="596.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="625.0" y="59.5" text-anchor="middle">##ing</text><rect class="box" x="660.0" y="40.0" width="58.0" height="30.0" rx="6"/><text class="t-muted" x="689.0" y="59.5" text-anchor="middle">[SEP]</text><text class="t-tick" x="20.0" y="92.0" text-anchor="start">each input vector = token embedding + segment embedding (A or B) + position embedding</text><text class="t-muted" x="20.0" y="110.0" text-anchor="start">blue = sentence A, orange = sentence B; WordPiece splits "playing" into play + ##ing</text><rect class="box-1" x="20.0" y="166.0" width="700.0" height="50.0" rx="10"/><text class="t-note" x="370.0" y="188.0" text-anchor="middle">Transformer encoder: 12 layers, 768 numbers per token, 12 heads (BERT-base)</text><text class="t-tick" x="370.0" y="206.0" text-anchor="middle">every token looks at every token, left and right, in every layer</text><line class="edge" x1="49.0" y1="218.0" x2="49.0" y2="244.0" marker-end="url(#ah)"/><line class="edge" x1="305.0" y1="218.0" x2="305.0" y2="244.0" marker-end="url(#ah)"/><rect class="box-on" x="20.0" y="246.0" width="92.0" height="30.0" rx="6"/><text class="t-tick" x="66.0" y="266.0" text-anchor="middle">C → NSP</text><text class="t-muted" x="119.0" y="266.0" text-anchor="start">IsNext or NotNext?</text><rect class="box-mask" x="259.0" y="246.0" width="92.0" height="30.0" rx="6"/><text class="t-tick" x="305.0" y="266.0" text-anchor="middle">T → MLM</text><text class="t-muted" x="357.0" y="266.0" text-anchor="start">which word was hidden? ("cute")</text><text class="t-muted" x="20.0" y="298.0" text-anchor="start">loss = mean masked-LM loss + mean NSP loss; 15% of tokens are chosen for prediction (80% [MASK], 10% random, 10% kept)</text><rect class="box-3" x="20.0" y="346.0" width="166.0" height="76.0" rx="10"/><text class="t-note" x="103.0" y="368.0" text-anchor="middle">classify</text><text class="t-mono" x="103.0" y="388.0" text-anchor="middle">C · Wᵀ → label</text><text class="t-muted" x="103.0" y="408.0" text-anchor="middle">GLUE, sentiment</text><rect class="box-2" x="198.0" y="346.0" width="166.0" height="76.0" rx="10"/><text class="t-note" x="281.0" y="368.0" text-anchor="middle">find a span</text><text class="t-mono" x="281.0" y="388.0" text-anchor="middle">S · Tᵢ and E · Tⱼ</text><text class="t-muted" x="281.0" y="408.0" text-anchor="middle">SQuAD</text><rect class="box-4" x="376.0" y="346.0" width="166.0" height="76.0" rx="10"/><text class="t-note" x="459.0" y="368.0" text-anchor="middle">tag tokens</text><text class="t-mono" x="459.0" y="388.0" text-anchor="middle">Tᵢ → label</text><text class="t-muted" x="459.0" y="408.0" text-anchor="middle">NER</text><rect class="box-on" x="554.0" y="346.0" width="166.0" height="76.0" rx="10"/><text class="t-note" x="637.0" y="368.0" text-anchor="middle">pick a choice</text><text class="t-mono" x="637.0" y="388.0" text-anchor="middle">v · C, softmax</text><text class="t-muted" x="637.0" y="408.0" text-anchor="middle">SWAG</text></svg><figcaption>BERT on a napkin. Text becomes WordPiece tokens with [CLS] and [SEP], plus segment and position embeddings. A deep bidirectional encoder is pre-trained with the masked LM and next sentence prediction. For each task, one small layer is added on top and everything is fine-tuned.</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 760 486" role="img" aria-label="One sentence pair through BERT-base. WordPiece tokens; three vectors per token (token, segment, position) are added; twelve encoder layers; one output vector per token, C for [CLS] and T for the others; a small head per task reads C or the T vectors."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">One input, all the way through BERT-base</text><text class="t-note" x="20.0" y="50.0" text-anchor="start">1  WordPiece tokens</text><rect class="box-on" x="24.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="51.0" y="79.5" text-anchor="middle">[CLS]</text><rect class="box-1" x="84.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="111.0" y="79.5" text-anchor="middle">my</text><rect class="box-1" x="144.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="171.0" y="79.5" text-anchor="middle">dog</text><rect class="box-1" x="204.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="231.0" y="79.5" text-anchor="middle">is</text><rect class="box-mask" x="264.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="291.0" y="79.5" text-anchor="middle">[MASK]</text><rect class="box" x="324.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="351.0" y="79.5" text-anchor="middle">[SEP]</text><rect class="box-2" x="384.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="411.0" y="79.5" text-anchor="middle">he</text><rect class="box-2" x="444.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="471.0" y="79.5" text-anchor="middle">likes</text><rect class="box-2" x="504.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="531.0" y="79.5" text-anchor="middle">play</text><rect class="box-2" x="564.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="591.0" y="79.5" text-anchor="middle">##ing</text><rect class="box" x="624.0" y="60.0" width="54.0" height="30.0" rx="6"/><text class="t-muted" x="651.0" y="79.5" text-anchor="middle">[SEP]</text><text class="t-note" x="20.0" y="122.0" text-anchor="start">2  three 768-number vectors per token, added: token + segment + position</text><g class="mark"><rect class="s1" x="30.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/></g><g class="mark"><rect class="s1" x="38.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s1" x="47.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="55.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><rect class="s1" x="64.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s3" x="30.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.83"/></g><g class="mark"><rect class="s3" x="38.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.45"/></g><g class="mark"><rect class="s3" x="47.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.87"/></g><g class="mark"><rect class="s3" x="55.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="64.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><rect class="s4" x="30.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="38.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><rect class="s4" x="47.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.48"/></g><g class="mark"><rect class="s4" x="55.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.87"/></g><g class="mark"><rect class="s4" x="64.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><g class="mark"><rect class="s1" x="90.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="98.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s1" x="107.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><rect class="s1" x="115.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s1" x="124.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s3" x="90.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.83"/></g><g class="mark"><rect class="s3" x="98.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="107.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.67"/></g><g class="mark"><rect class="s3" x="115.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><rect class="s3" x="124.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s4" x="90.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="98.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s4" x="107.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="115.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.64"/></g><g class="mark"><rect class="s4" x="124.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.67"/></g><g class="mark"><rect class="s1" x="150.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s1" x="158.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="167.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s1" x="175.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><rect class="s1" x="184.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s3" x="150.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.73"/></g><g class="mark"><rect class="s3" x="158.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><rect class="s3" x="167.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="175.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.90"/></g><g class="mark"><rect class="s3" x="184.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/></g><g class="mark"><rect class="s4" x="150.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="158.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s4" x="167.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s4" x="175.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s4" x="184.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s1" x="210.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s1" x="218.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s1" x="227.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><rect class="s1" x="235.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="244.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.77"/></g><g class="mark"><rect class="s3" x="210.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.90"/></g><g class="mark"><rect class="s3" x="218.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.90"/></g><g class="mark"><rect class="s3" x="227.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><rect class="s3" x="235.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.75"/></g><g class="mark"><rect class="s3" x="244.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="210.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.58"/></g><g class="mark"><rect class="s4" x="218.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="227.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><g class="mark"><rect class="s4" x="235.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s4" x="244.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.81"/></g><g class="mark"><rect class="s1" x="270.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s1" x="278.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="287.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.81"/></g><g class="mark"><rect class="s1" x="295.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.45"/></g><g class="mark"><rect class="s1" x="304.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s3" x="270.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.58"/></g><g class="mark"><rect class="s3" x="278.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.71"/></g><g class="mark"><rect class="s3" x="287.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="295.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><rect class="s3" x="304.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.48"/></g><g class="mark"><rect class="s4" x="270.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s4" x="278.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/></g><g class="mark"><rect class="s4" x="287.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.73"/></g><g class="mark"><rect class="s4" x="295.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="304.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="330.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s1" x="338.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s1" x="347.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.83"/></g><g class="mark"><rect class="s1" x="355.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="364.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s3" x="330.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="338.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.83"/></g><g class="mark"><rect class="s3" x="347.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s3" x="355.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s3" x="364.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s4" x="330.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s4" x="338.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="347.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s4" x="355.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.46"/></g><g class="mark"><rect class="s4" x="364.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.87"/></g><g class="mark"><rect class="s1" x="390.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><rect class="s1" x="398.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="407.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><rect class="s1" x="415.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.59"/></g><g class="mark"><rect class="s1" x="424.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><g class="mark"><rect class="s2" x="390.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.45"/></g><g class="mark"><rect class="s2" x="398.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s2" x="407.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="415.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s2" x="424.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/></g><g class="mark"><rect class="s4" x="390.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s4" x="398.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/></g><g class="mark"><rect class="s4" x="407.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><rect class="s4" x="415.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="424.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><rect class="s1" x="450.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s1" x="458.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s1" x="467.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="475.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s1" x="484.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/></g><g class="mark"><rect class="s2" x="450.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="458.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><rect class="s2" x="467.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s2" x="475.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s2" x="484.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.88"/></g><g class="mark"><rect class="s4" x="450.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s4" x="458.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="467.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><rect class="s4" x="475.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><rect class="s4" x="484.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s1" x="510.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.88"/></g><g class="mark"><rect class="s1" x="518.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><g class="mark"><rect class="s1" x="527.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.58"/></g><g class="mark"><rect class="s1" x="535.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.71"/></g><g class="mark"><rect class="s1" x="544.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="510.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s2" x="518.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s2" x="527.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="535.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s2" x="544.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.77"/></g><g class="mark"><rect class="s4" x="510.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.77"/></g><g class="mark"><rect class="s4" x="518.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s4" x="527.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="535.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s4" x="544.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s1" x="570.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><rect class="s1" x="578.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.67"/></g><g class="mark"><rect class="s1" x="587.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="595.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s1" x="604.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s2" x="570.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s2" x="578.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><rect class="s2" x="587.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><rect class="s2" x="595.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="604.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s4" x="570.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.88"/></g><g class="mark"><rect class="s4" x="578.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s4" x="587.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.57"/></g><g class="mark"><rect class="s4" x="595.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><rect class="s4" x="604.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="630.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s1" x="638.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s1" x="647.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="655.5" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s1" x="664.0" y="134.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="630.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><rect class="s2" x="638.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="647.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><rect class="s2" x="655.5" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/></g><g class="mark"><rect class="s2" x="664.0" y="150.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><rect class="s4" x="630.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/></g><g class="mark"><rect class="s4" x="638.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s4" x="647.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="655.5" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s4" x="664.0" y="166.0" width="7" height="7" rx="2" style="fill-opacity:0.45"/></g><text class="t-s1" x="690.0" y="141.0" text-anchor="start">token</text><text class="t-s3" x="690.0" y="157.0" text-anchor="start">segment A</text><text class="t-s4" x="690.0" y="173.0" text-anchor="start">position</text><text class="t-muted" x="690.0" y="189.0" text-anchor="start">(B: orange)</text><text class="t-note" x="20.0" y="208.0" text-anchor="start">3  twelve identical encoder layers (12 attention heads + a feed-forward network each)</text><rect class="box-1" x="18.0" y="218.0" width="658.0" height="18.0" rx="6"/><rect class="box-1" x="18.0" y="238.0" width="658.0" height="18.0" rx="6"/><rect class="box-1" x="18.0" y="258.0" width="658.0" height="18.0" rx="6"/><text class="t-tick" x="351.0" y="231.0" text-anchor="middle">layer 1   ...   layer 12: every token attends to every token</text><text class="t-note" x="20.0" y="300.0" text-anchor="start">4  one output vector per token (768 numbers each)</text><line class="edge" x1="51.0" y1="306.0" x2="51.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="27.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s2" x="30.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="38.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s2" x="47.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.50"/></g><g class="mark"><rect class="s2" x="55.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s2" x="64.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><text class="t-math" x="51.0" y="356.0" text-anchor="middle">C</text><line class="edge" x1="111.0" y1="306.0" x2="111.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="87.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="90.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s3" x="98.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><rect class="s3" x="107.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><g class="mark"><rect class="s3" x="115.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="124.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.73"/></g><text class="t-math" x="111.0" y="356.0" text-anchor="middle">T₁</text><line class="edge" x1="171.0" y1="306.0" x2="171.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="147.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="150.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.88"/></g><g class="mark"><rect class="s3" x="158.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="167.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.77"/></g><g class="mark"><rect class="s3" x="175.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/></g><g class="mark"><rect class="s3" x="184.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.76"/></g><text class="t-math" x="171.0" y="356.0" text-anchor="middle">T₂</text><line class="edge" x1="231.0" y1="306.0" x2="231.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="207.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="210.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s3" x="218.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.48"/></g><g class="mark"><rect class="s3" x="227.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.73"/></g><g class="mark"><rect class="s3" x="235.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="244.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.92"/></g><text class="t-math" x="231.0" y="356.0" text-anchor="middle">T₃</text><line class="edge" x1="291.0" y1="306.0" x2="291.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="267.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="270.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s3" x="278.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s3" x="287.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="295.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.71"/></g><g class="mark"><rect class="s3" x="304.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.50"/></g><text class="t-math" x="291.0" y="356.0" text-anchor="middle">T₄</text><line class="edge" x1="351.0" y1="306.0" x2="351.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="327.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="330.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s3" x="338.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.74"/></g><g class="mark"><rect class="s3" x="347.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s3" x="355.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s3" x="364.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><text class="t-math" x="351.0" y="356.0" text-anchor="middle">T₅</text><line class="edge" x1="411.0" y1="306.0" x2="411.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="387.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="390.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s3" x="398.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.75"/></g><g class="mark"><rect class="s3" x="407.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="415.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.90"/></g><g class="mark"><rect class="s3" x="424.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.63"/></g><text class="t-math" x="411.0" y="356.0" text-anchor="middle">T₆</text><line class="edge" x1="471.0" y1="306.0" x2="471.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="447.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="450.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="458.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.94"/></g><g class="mark"><rect class="s3" x="467.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s3" x="475.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><rect class="s3" x="484.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><text class="t-math" x="471.0" y="356.0" text-anchor="middle">T₇</text><line class="edge" x1="531.0" y1="306.0" x2="531.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="507.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="510.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.71"/></g><g class="mark"><rect class="s3" x="518.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.47"/></g><g class="mark"><rect class="s3" x="527.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s3" x="535.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="544.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.85"/></g><text class="t-math" x="531.0" y="356.0" text-anchor="middle">T₈</text><line class="edge" x1="591.0" y1="306.0" x2="591.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="567.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="570.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.77"/></g><g class="mark"><rect class="s3" x="578.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="587.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.88"/></g><g class="mark"><rect class="s3" x="595.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.59"/></g><g class="mark"><rect class="s3" x="604.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.61"/></g><text class="t-math" x="591.0" y="356.0" text-anchor="middle">T₉</text><line class="edge" x1="651.0" y1="306.0" x2="651.0" y2="318.0" marker-end="url(#ah)"/><rect class="box-ghost" x="627.0" y="321.0" width="47.0" height="13.0" rx="4"/><g class="mark"><rect class="s3" x="630.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.90"/></g><g class="mark"><rect class="s3" x="638.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.64"/></g><g class="mark"><rect class="s3" x="647.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/></g><g class="mark"><rect class="s3" x="655.5" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s3" x="664.0" y="324.0" width="7" height="7" rx="2" style="fill-opacity:0.95"/></g><text class="t-math" x="651.0" y="356.0" text-anchor="middle">T₁₀</text><text class="t-note" x="20.0" y="392.0" text-anchor="start">5  a small head reads what the task needs</text><rect class="box-3" x="20.0" y="404.0" width="172.0" height="70.0" rx="10"/><text class="t-note" x="106.0" y="424.0" text-anchor="middle">NSP / classification</text><text class="t-tick" x="106.0" y="444.0" text-anchor="middle">softmax(C Wᵀ)</text><text class="t-muted" x="106.0" y="463.0" text-anchor="middle">reads C</text><rect class="box-mask" x="202.0" y="404.0" width="172.0" height="70.0" rx="10"/><text class="t-note" x="288.0" y="424.0" text-anchor="middle">masked LM</text><text class="t-tick" x="288.0" y="444.0" text-anchor="middle">softmax over 30,522 words</text><text class="t-muted" x="288.0" y="463.0" text-anchor="middle">reads T at [MASK]</text><rect class="box-4" x="384.0" y="404.0" width="172.0" height="70.0" rx="10"/><text class="t-note" x="470.0" y="424.0" text-anchor="middle">tagging (NER)</text><text class="t-tick" x="470.0" y="444.0" text-anchor="middle">one label per token</text><text class="t-muted" x="470.0" y="463.0" text-anchor="middle">reads every Tᵢ</text><rect class="box-2" x="566.0" y="404.0" width="172.0" height="70.0" rx="10"/><text class="t-note" x="652.0" y="424.0" text-anchor="middle">answer span</text><text class="t-tick" x="652.0" y="444.0" text-anchor="middle">start and end scores</text><text class="t-muted" x="652.0" y="463.0" text-anchor="middle">reads Tᵢ · S, Tⱼ · E</text></svg><figcaption>One input all the way through BERT-base. 1: WordPiece tokens with [CLS], [SEP] and a [MASK]. 2: three 768-number vectors per token (token, segment, position), added. 3: twelve identical encoder layers, every token attending to every token. 4: one output vector per token, C and T₁ to T₁₀. 5: a small head reads what the task needs: C for classification and NSP, T at [MASK] for the masked LM, every Tᵢ for tagging, Tᵢ·S and Tᵢ·E for answer spans.</figcaption></figure>

Here is the paper, part by part, with its key numbers. Every number is from the paper, with its location.

**[Part 1: The Big Idea](https://ishwarj.com/papers/bert/part-1-the-big-idea/)** (Abstract, §1). Language models read in one direction, which is a real limit for tasks like question answering, where "it is crucial to incorporate context from both directions" (§1). BERT pre-trains a deep bidirectional Transformer by hiding words and predicting them. It set new state-of-the-art results on eleven tasks, including GLUE 80.5 (+7.7 points), MultiNLI 86.7% (+4.6), SQuAD v1.1 Test F1 93.2 (+1.5) and SQuAD v2.0 Test F1 83.1 (+5.1) (Abstract).

**[Part 2: Inside BERT](https://ishwarj.com/papers/bert/part-2-architecture-and-input/)** (§2, §3). BERT is a Transformer encoder. BERT-base has L=12 layers, H=768, A=12 heads and 110M parameters; BERT-large has L=24, H=1024, A=16 and 340M (§3). The feed-forward size is 4H: 3072 and 4096 (footnote 3). Text is cut into WordPiece tokens from a 30,000-token vocabulary; every input starts with `[CLS]`, sentences are separated by `[SEP]`, and each token's input is the sum of a token, a segment and a position embedding (§3, Figure 2).

**[Part 3: Pre-training](https://ishwarj.com/papers/bert/part-3-pre-training/)** (§3.1, A.1, A.2). Masked LM: choose 15% of the WordPiece tokens; replace them with `[MASK]` 80% of the time, a random token 10% and leave them unchanged 10% (§3.1); random replacement is then only 1.5% of all tokens (A.1). Next sentence prediction: half the time sentence B really follows A (IsNext), half the time it is random (NotNext); the final model reaches 97%-98% on it (footnote 5). Data: BooksCorpus (800M words) and English Wikipedia (2,500M words) (§3.1). Training: batches of 256 sequences of 512 tokens ("128,000 tokens/batch") for 1,000,000 steps, "approximately 40 epochs over the 3.3 billion word corpus", Adam with learning rate 1e-4, 10,000 warm-up steps, dropout 0.1, GELU, 90% of the steps at length 128 (A.2).

**[Part 4: Fine-tuning and Results](https://ishwarj.com/papers/bert/part-4-fine-tuning-and-results/)** (§3.2, §4, A.3, A.5, B.1). For classification, the only new weights are W of size K×H, and the loss is $$\log(\text{softmax}(CW^\top))$$ (§4.1). On GLUE, BERT-base and BERT-large average 79.6 and 82.1, against 75.1 for OpenAI GPT (Table 1); on the official leaderboard BERT-large scores 80.5 against GPT's 72.8 (§4.1). For SQuAD, only a start vector S and an end vector E are new; a span from i to j scores $$S \cdot T_i + E \cdot T_j$$ with $$j \ge i$$ (§4.2). SQuAD v1.1 best Test F1 93.2 (ensemble with TriviaQA, Table 2); SQuAD v2.0 Test F1 83.1, +5.1 over the previous best (§4.3, Table 3); SWAG test accuracy 86.3, +27.1% over ESIM+ELMo and +8.3% over GPT (§4.4, Table 4). Good fine-tuning settings: batch size 16 or 32, learning rate 5e-5, 3e-5 or 2e-5, and 2, 3 or 4 epochs (A.3).

**[Part 5: Ablations](https://ishwarj.com/papers/bert/part-5-ablations/)** (§5, A.4, C.1, C.2). Removing NSP hurts (MNLI 84.4 → 83.9, QNLI 88.4 → 84.9); training left-to-right hurts much more (MRPC 77.5, SQuAD F1 77.8 instead of 86.7 and 88.5) (Table 5). Bigger models are better on every task tried, from 3 layers to 24 (Table 6). Using BERT's frozen features (concatenating the last four layers) is "only 0.3 F1 behind fine-tuning the entire model" on named entity recognition (§5.3, Table 7). Training for 1M steps instead of 500k adds "almost 1.0%" on MNLI (C.1), and the 80/10/10 masking mix gives the best named entity recognition results, especially in the feature-based setting, while MNLI barely changes (Table 8).

**This part** (§6). The contribution is to generalise unsupervised pre-training "to deep *bidirectional* architectures" so "the same pre-trained model" can do many tasks (§6).

## The key words, in one list

Each term links to the part where it is first explained.

- **Pre-training**, **fine-tuning**, **encoder**, **feature-based approach**, **language model**, **unidirectional**, **Cloze task**: [Part 1](https://ishwarj.com/papers/bert/part-1-the-big-idea/)
- **Transfer learning**, **L / H / A**, **parameters**, **WordPiece**, **`[CLS]` and `[SEP]`**, **segment and position embeddings**: [Part 2](https://ishwarj.com/papers/bert/part-2-architecture-and-input/)
- **Masked LM**, **the 80 / 10 / 10 rule**, **next sentence prediction**, **cross-entropy loss**, **perplexity**, **warm-up**, **GELU**: [Part 3](https://ishwarj.com/papers/bert/part-3-pre-training/)
- **GLUE**, **SQuAD**, **span**, **F1 and exact match**, **SWAG**, **dev and test sets**: [Part 4](https://ishwarj.com/papers/bert/part-4-fine-tuning-and-results/)
- **Ablation**, **feature extraction from layers**, **masking strategies**: [Part 5](https://ishwarj.com/papers/bert/part-5-ablations/)
- **Dynamic masking**, **distillation**, **sentence embedding**, **cross-encoder**: this part

> [!TAKEAWAYS] Key takeaways
> - The conclusion in one line: unsupervised pre-training already worked for one-directional models; BERT's "major contribution" is making it work for **deep bidirectional** models, so one pre-trained model handles many tasks.
> - BERT's impact was fast: **Best Long Paper** at NAACL 2019, and in October 2019 Google said BERT would help understand **one in 10** US English searches.
> - Later work kept the idea and fixed the training: **RoBERTa** (train longer on more data, drop NSP), **SpanBERT** (mask spans), **ALBERT** and **DistilBERT** (smaller), **XLNet**, **ELECTRA** and **DeBERTa** (better than plain masking), **Sentence-BERT** (good sentence vectors), **ModernBERT** (8192-token inputs).
> - Measured limits: a hard **512-token** wall (a 722-token input crashes or loses 210 tokens); `[MASK]` never appears after pre-training; only **15%** of tokens give a training signal; two blanks are filled independently ("new town", while "new" alone gives "york" at 0.979).
> - The paper's footnote 6 holds: raw `[CLS]` vectors scored unrelated pairs up to **0.873**, above a related pair; a model fine-tuned for similarity put unrelated pairs near 0.
> - BERT-style encoders are still used every day for classification, NER, search and reranking, because they are small, fast and cheap to fine-tune.

<details>
<summary>Run it yourself</summary>

Every measurement in this part comes from [`code/papers/bert/bert_part6.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part6.py). It runs on a CPU in about a minute and downloads `bert-base-uncased`, `sentence-transformers/all-MiniLM-L6-v2` and `dslim/bert-base-NER` the first time.

```bash
pip install torch transformers
python bert_part6.py      # prints everything above, writes results/part6.json
```

<figure><img src="/img/papers/bert/part6-run.png" alt="Terminal output of bert_part6.py: the 512-token limit, cosine similarities for six sentence pairs with three kinds of vectors, two masks predicted separately and then one at a time, and named entities found by a fine-tuned BERT-base model" loading="lazy" /><figcaption>The real output of bert_part6.py.</figcaption></figure>

</details>

## References

**The BERT paper**

1. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805). NAACL 2019 ([ACL Anthology](https://aclanthology.org/N19-1423/)). Best Long Paper at NAACL 2019.
2. Google Research. [BERT code and pre-trained models](https://github.com/google-research/bert).

**Later work discussed in this part**

3. Y. Liu et al. [*RoBERTa: A Robustly Optimized BERT Pretraining Approach*](https://arxiv.org/abs/1907.11692). arXiv 2019.
4. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, O. Levy. [*SpanBERT: Improving Pre-training by Representing and Predicting Spans*](https://arxiv.org/abs/1907.10529). TACL 2020.
5. Z. Lan et al. [*ALBERT: A Lite BERT for Self-supervised Learning of Language Representations*](https://arxiv.org/abs/1909.11942). ICLR 2020.
6. V. Sanh, L. Debut, J. Chaumond, T. Wolf. [*DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter*](https://arxiv.org/abs/1910.01108). NeurIPS EMC² workshop 2019.
7. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, Q. V. Le. [*XLNet: Generalized Autoregressive Pretraining for Language Understanding*](https://arxiv.org/abs/1906.08237). NeurIPS 2019.
8. K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning. [*ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators*](https://arxiv.org/abs/2003.10555). ICLR 2020.
9. P. He, X. Liu, J. Gao, W. Chen. [*DeBERTa: Decoding-enhanced BERT with Disentangled Attention*](https://arxiv.org/abs/2006.03654). ICLR 2021.
10. N. Reimers, I. Gurevych. [*Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks*](https://arxiv.org/abs/1908.10084). EMNLP 2019.
11. B. Warner et al. [*Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference*](https://arxiv.org/abs/2412.13663) (ModernBERT). arXiv 2024.
12. G. Hinton, O. Vinyals, J. Dean. [*Distilling the Knowledge in a Neural Network*](https://arxiv.org/abs/1503.02531). NeurIPS Deep Learning workshop 2014.

**Other sources used in this part**

13. P. Nayak. [*Understanding searches better than ever before*](https://blog.google/products/search/search-language-understanding-bert/). Google blog, 25 October 2019.
14. Public models used in the code: [distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased) and [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2).
15. Code for this part: [`bert_part6.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part6.py) and [`bert_part6_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part6_math.py).

