# The Big Idea: Reading in Both Directions

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/bert/part-1-the-big-idea/
Updated: 2026-10-21
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/bert/part-1-the-big-idea.md


In October 2018, four researchers at Google posted a paper called **BERT**. It reported new best results on eleven language tasks at once. A year later, in October 2019, Google wrote that BERT would help its search engine understand about one in ten English searches in the US. BERT-style models are still used today inside search engines, spam filters, support-ticket routers and retrieval systems.

This series reads the paper slowly, in the paper's own order. Each piece shows the exact lines of the paper as a highlighted screenshot in a teal box, with what they say and what they mean. A picture or real code with its real output follows when it helps.

The small `Paper §1` tag above each heading tells you which section of the paper you are reading. The screenshots come from the paper's second arXiv version (May 2019), the one most people read today.

## The title {§Title}

> [!PAPER] · Title
> [![The title of the BERT paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova, Google AI Language](https://ishwarj.com/img/papers/bert/p1-title.pngs/bert/p1-title.png)](/img/papers/bert/p1-title.png)
>
> **What it says:** "BERT: Pre-training of Deep **Bidirectional Transformers** for Language Understanding."
>
> **What it means:**
>
> A good title works like the label on a tin: it lists what is inside. Here every word of the title is one idea of the paper. If you understand the title, you understand the paper.
>
> The words also follow the order of the work, though the authors do not say so. First comes a long training on ordinary text. The model is a deep stack that reads both ways. The goal is to read and decide, not to write.
>
> The paper was posted to arXiv in October 2018. It was published at NAACL 2019, a major conference on language technology.

Let us take the title apart, one word at a time.

- **Pre-training.** First, train a model on a huge pile of ordinary text, before it ever sees the real task. This is the expensive part, and it is done once.
- **Deep.** The model is a tall stack of layers (12 or 24 of them). Each layer refines what the layer below understood.
- **Bidirectional.** Every word looks at the words on its **left and** on its **right**. This is the key idea of the paper.
- **Transformers.** The kind of neural network used. It was introduced in 2017 in the paper *Attention Is All You Need*.
- **Language Understanding.** The goal: tasks where a model must understand text (classify it, answer questions about it, find names in it), not write new text.

> [!DEFINITION] Neural network and model
> A **neural network** is a program made of many simple maths steps whose numbers (called **weights** or **parameters**) are learned from examples. A **model** is one such network with its learned weights. BERT is a model with about 110 million weights in its smaller version.

> [!DEFINITION] Transformer
> A neural network design built around **attention**: every word computes how much it should "look at" every other word, and mixes in information from the words it looks at. If attention is new to you, the [attention series](https://ishwarj.com/writings/attention-1-self-attention/) explains it from zero. For this paper you only need the idea: words look at other words.

> [!DEFINITION] Pre-training
> Training a model on a large amount of general text first, so it learns how language works. The model is later adapted to a specific job. It is like a student who reads thousands of books before taking any particular exam.

## What BERT is {§Abstract}

> [!PAPER] · Abstract
> [![The first half of the BERT abstract, with three highlights: Bidirectional Encoder Representations from Transformers; jointly conditioning on both left and right context in all layers; one additional output layer](https://ishwarj.com/img/papers/bert/p1-abstract-what.png1-abstract-what.png)](/img/papers/bert/p1-abstract-what.png)
>
> **What it says:** BERT stands for **B**idirectional **E**ncoder **R**epresentations from **T**ransformers. It is pre-trained on "unlabeled text by jointly conditioning on both left and right context in all layers". Afterwards it can be fine-tuned "with just one additional output layer" for many tasks, such as question answering.
>
> **What it means:**
>
> Think of a medical student. First come years of general study, then a short training in one field. BERT works the same way: one long general training on ordinary text, then a short training for each task.
>
> Each phrase carries one part of the plan. "Unlabeled text" means nobody has to write answers, so the supply of training text is almost unlimited. "Both left and right context in all layers" means each word takes its meaning from both sides, at every level. "Just one additional output layer" means the same model serves many tasks, with almost nothing new on top.
>
> So these three sentences are the whole recipe. Learn from plain text, look both ways, then add one small layer per task.

The name has four parts. You already know **bidirectional** and **transformers** from the title. The two new words are **encoder** and **representations**.

> [!DEFINITION] Representation (embedding)
> A list of numbers that stands for a piece of text. A model turns each word into such a list, for example 768 numbers. Words with similar meanings in similar contexts get similar lists. "Representation", "embedding" and "vector" are used almost interchangeably in this paper.

> [!DEFINITION] Encoder
> The part of a Transformer that **reads** text and turns every word into a representation. (The other part, a **decoder**, writes text one word at a time.) BERT is only an encoder: it understands, it does not write.

### Encoder and decoder: the two halves of the Transformer

"Encoder" is a precise word here. It points at one half of the Transformer from *Attention Is All You Need* (Vaswani et al., 2017), the paper BERT is built on.

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · §3, Figure 1
> [![Figure 1 of Attention Is All You Need: the Transformer, an encoder stack on the left with multi-head attention and feed-forward blocks, and a decoder stack on the right with masked multi-head attention, cross-attention and feed-forward blocks, ending in a linear layer and softmax](https://ishwarj.com/img/papers/bert/p1-transformer-fig1.pngransformer-fig1.png)](/img/papers/bert/p1-transformer-fig1.png)
> [![Section 3.1 of Attention Is All You Need: the encoder and decoder are the left and right halves of Figure 1; the decoder's self-attention is modified to prevent positions from attending to subsequent positions, so predictions for position i depend only on outputs at positions before i](https://ishwarj.com/img/papers/bert/p1-transformer-halves.pngnsformer-halves.png)](/img/papers/bert/p1-transformer-halves.png)
>
> **What it says:** both halves are stacks of $$N$$ identical layers. The decoder's self-attention is "modified to prevent positions from attending to subsequent positions", so "the predictions for position $$i$$ can depend only on the known outputs at positions less than $$i$$". The encoder has no such rule.
>
> **What it means:**
>
> The Transformer was built for translation. Think of a human interpreter with two jobs: first read the whole French sentence, then say the English one, word by word. The left half (the **encoder**) does the reading. The right half (the **decoder**) does the writing, one word at a time.
>
> A writer must not see words that it has not written yet. That is why the decoder has a rule against "subsequent positions". A reader has no such need, so the encoder sees the whole sentence at once.
>
> BERT is the left half. OpenAI GPT is (almost) the right half, without the cross-attention block. The single difference between them is that rule about "subsequent positions".

<figure class="fig"><svg viewBox="0 0 760 336" role="img" aria-label="The original Transformer has two halves. The encoder (left) uses self-attention where every word sees every word. The decoder (right) uses masked self-attention where a word sees only the words on its left, plus cross-attention to the encoder. BERT keeps only the encoder half; GPT keeps only the decoder half, without the cross-attention."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="22.0" text-anchor="middle">Left half: the encoder</text><text class="t-title" x="570.0" y="22.0" text-anchor="middle">Right half: the decoder</text><rect class="box-ghost" x="80.0" y="50.0" width="220.0" height="122.0" rx="14"/><rect class="box-1" x="96.0" y="112.0" width="188.0" height="34.0" rx="8"/><text class="t-note" x="190.0" y="127.0" text-anchor="middle">self-attention</text><text class="t-muted" x="190.0" y="141.0" text-anchor="middle">every word sees every word</text><rect class="box" x="96.0" y="66.0" width="188.0" height="34.0" rx="8"/><text class="t-note" x="190.0" y="81.0" text-anchor="middle">feed-forward</text><text class="t-muted" x="190.0" y="95.0" text-anchor="middle">each word on its own</text><text class="t-tick" x="308.0" y="70.0" text-anchor="start">× N</text><rect class="box-ghost" x="450.0" y="50.0" width="240.0" height="168.0" rx="14"/><rect class="box-2" x="466.0" y="158.0" width="208.0" height="34.0" rx="8"/><text class="t-note" x="570.0" y="173.0" text-anchor="middle">masked self-attention</text><text class="t-muted" x="570.0" y="187.0" text-anchor="middle">a word sees only its left</text><rect class="box-ghost" x="466.0" y="112.0" width="208.0" height="34.0" rx="8"/><text class="t-note" x="570.0" y="127.0" text-anchor="middle">cross-attention</text><text class="t-muted" x="570.0" y="141.0" text-anchor="middle">looks at the encoder output</text><rect class="box" x="466.0" y="66.0" width="208.0" height="34.0" rx="8"/><text class="t-note" x="570.0" y="81.0" text-anchor="middle">feed-forward</text><text class="t-muted" x="570.0" y="95.0" text-anchor="middle">each word on its own</text><text class="t-tick" x="698.0" y="70.0" text-anchor="start">× N</text><line class="edge" x1="190.0" y1="192.0" x2="190.0" y2="172.0" marker-end="url(#ah)"/><text class="t-tick" x="190.0" y="208.0" text-anchor="middle">input words</text><line class="edge" x1="570.0" y1="238.0" x2="570.0" y2="218.0" marker-end="url(#ah)"/><text class="t-tick" x="570.0" y="254.0" text-anchor="middle">words written so far</text><line class="edge-dim" x1="300.0" y1="120.0" x2="450.0" y2="120.0" marker-end="url(#ah)"/><rect class="box-1" x="60.0" y="270.0" width="260.0" height="52.0" rx="12"/><text class="t-note" x="190.0" y="292.0" text-anchor="middle">BERT keeps only this half</text><text class="t-muted" x="190.0" y="310.0" text-anchor="middle">no mask: both directions</text><rect class="box-2" x="430.0" y="270.0" width="280.0" height="52.0" rx="12"/><text class="t-note" x="570.0" y="292.0" text-anchor="middle">GPT keeps only this half</text><text class="t-muted" x="570.0" y="310.0" text-anchor="middle">without the cross-attention; mask stays</text></svg><figcaption>The two halves of the original Transformer. BERT keeps only the encoder (every word sees every word). GPT keeps a decoder-style stack without the cross-attention, so a word sees only itself and the words before it.</figcaption></figure>

That rule is implemented by a **mask** inside attention. Here is attention with the mask written in. (The [attention series](https://ishwarj.com/writings/attention-1-self-attention/) derives this equation step by step.)

$$
\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V
$$

where, for a text of $$n$$ tokens:

- $$Q$$, $$K$$, $$V$$ are the **queries**, **keys** and **values**: three $$n \times d_k$$ tables made from the token vectors. Row $$i$$ of $$Q$$ asks "what is token $$i$$ looking for?"; row $$j$$ of $$K$$ says "what does token $$j$$ offer?";
- $$QK^\top$$ is an $$n \times n$$ table of scores: entry $$(i, j)$$ says how well token $$i$$'s question matches token $$j$$;
- $$\sqrt{d_k}$$ (here $$\sqrt{64} = 8$$) keeps the scores from growing too large;
- $$M$$ is the **mask**, also $$n \times n$$: 0 where looking is allowed, $$-\infty$$ where it is blocked;
- softmax turns each row into weights that add up to 1, and $$e^{-\infty} = 0$$, so a blocked position gets weight exactly 0.

The two models differ only in $$M$$:

$$
M^{\text{GPT}}_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i \end{cases}
\qquad\qquad
M^{\text{BERT}}_{ij} = 0 \ \text{ for every } i, j
$$

In words: GPT blocks every key to the right of the query (the upper triangle of the table). BERT blocks nothing. (BERT's real code also masks padding, the empty slots that fill short texts up to a common length, but never real words.)

<figure class="fig"><svg viewBox="0 0 760 336" role="img" aria-label="Attention masks for the sentence "the kid smiles". Rows are queries (the word doing the looking), columns are keys (the word being looked at). In GPT the upper triangle is blocked, so each word sees only itself and the words before it. In BERT all nine squares are allowed."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="169.2" y="36.0" text-anchor="middle">GPT: causal (masked)</text><text class="t-muted" x="169.2" y="54.0" text-anchor="middle">key (column)</text><text class="t-tick" x="117.2" y="72.0" text-anchor="middle">the</text><text class="t-tick" x="169.2" y="72.0" text-anchor="middle">kid</text><text class="t-tick" x="221.2" y="72.0" text-anchor="middle">smiles</text><text class="t-tick" x="83.2" y="110.0" text-anchor="end">the</text><g class="mark"><title>the attends to the</title><rect class="s2" x="92.7" y="81.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="144.7" y="81.5" width="49.0" height="49.0" rx="3"/><line class="edge" x1="159.8" y1="96.6" x2="178.6" y2="115.4"/><line class="edge" x1="178.6" y1="96.6" x2="159.8" y2="115.4"/><rect class="cell-masked" x="196.7" y="81.5" width="49.0" height="49.0" rx="3"/><line class="edge" x1="211.8" y1="96.6" x2="230.6" y2="115.4"/><line class="edge" x1="230.6" y1="96.6" x2="211.8" y2="115.4"/><text class="t-tick" x="83.2" y="162.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to the</title><rect class="s2" x="92.7" y="133.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>kid attends to kid</title><rect class="s2" x="144.7" y="133.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="196.7" y="133.5" width="49.0" height="49.0" rx="3"/><line class="edge" x1="211.8" y1="148.6" x2="230.6" y2="167.4"/><line class="edge" x1="230.6" y1="148.6" x2="211.8" y2="167.4"/><text class="t-tick" x="83.2" y="214.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to the</title><rect class="s2" x="92.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>smiles attends to kid</title><rect class="s2" x="144.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>smiles attends to smiles</title><rect class="s2" x="196.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="40.0" y="256.0" text-anchor="start">query (row)</text><text class="t-title" x="549.2" y="36.0" text-anchor="middle">BERT: bidirectional</text><text class="t-muted" x="549.2" y="54.0" text-anchor="middle">key (column)</text><text class="t-tick" x="497.2" y="72.0" text-anchor="middle">the</text><text class="t-tick" x="549.2" y="72.0" text-anchor="middle">kid</text><text class="t-tick" x="601.2" y="72.0" text-anchor="middle">smiles</text><text class="t-tick" x="463.2" y="110.0" text-anchor="end">the</text><g class="mark"><title>the attends to the</title><rect class="s1" x="472.7" y="81.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>the attends to kid</title><rect class="s1" x="524.7" y="81.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>the attends to smiles</title><rect class="s1" x="576.7" y="81.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="463.2" y="162.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to the</title><rect class="s1" x="472.7" y="133.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>kid attends to kid</title><rect class="s1" x="524.7" y="133.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>kid attends to smiles</title><rect class="s1" x="576.7" y="133.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="463.2" y="214.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to the</title><rect class="s1" x="472.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>smiles attends to kid</title><rect class="s1" x="524.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>smiles attends to smiles</title><rect class="s1" x="576.7" y="185.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="420.0" y="256.0" text-anchor="start">query (row)</text><text class="t-tick" x="40.0" y="280.0" text-anchor="start">Row "kid": GPT lets it read "the" and itself; "smiles" is blocked.</text><text class="t-tick" x="40.0" y="300.0" text-anchor="start">Row "kid" in BERT: it reads "the", itself and "smiles".</text><text class="t-muted" x="40.0" y="320.0" text-anchor="start">Blocked squares (×) are the upper triangle: keys to the right of the query.</text></svg><figcaption>The mask for "the kid smiles". Rows are queries (the word that looks), columns are keys (the word looked at). GPT blocks the upper triangle: "kid" may read "the" and itself, but not "smiles". BERT allows every square.</figcaption></figure>

**A worked example with real numbers.** I took the first attention head of GPT-2's first layer, fed it "the kid smiles", and recomputed the attention by hand ([`bert_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1_math.py)):

```python
q, k, v = blk.attn.c_attn(blk.ln_1(x)).split(768, dim=2)      # GPT-2 layer 1: queries, keys, values
qh, kh = q[0, :, 0:64], k[0, :, 0:64]                          # head 1 uses 64 of the 768 numbers
S = (qh @ kh.T) / math.sqrt(64)                                # 3 x 3 table of scores
M = torch.triu(torch.full((3, 3), float('-inf')), diagonal=1)  # -inf above the diagonal: the future
A = torch.softmax(S + M, -1)                                   # each row adds up to 1
```

```text
scores s = q.k / sqrt(64):
  the     -0.618   -1.104   -0.860
  kid      0.770   -0.374   -0.562
  smiles   0.345   -0.316   -0.987
mask M (0 = allowed, -inf = blocked: a key to the right of the query):
  the      0.000     -inf     -inf
  kid      0.000    0.000     -inf
  smiles   0.000    0.000    0.000
weights a = softmax(s + M), row by row:
  the      1.000    0.000    0.000   (row sum 1.000)
  kid      0.758    0.242    0.000   (row sum 1.000)
  smiles   0.562    0.290    0.148   (row sum 1.000)
max difference from the library's own attention weights: 0.0e+00
```

Follow the row for "kid" by hand. Its score for "smiles" (-0.562) is replaced by $$-\infty$$, so only two scores are left:

$$
a_{\text{kid},\text{the}} = \frac{e^{0.770}}{e^{0.770} + e^{-0.374}} = \frac{2.160}{2.160 + 0.688} = 0.758,
\qquad
a_{\text{kid},\text{kid}} = \frac{0.688}{2.848} = 0.242
$$

The first row is even simpler: "the" can see only itself, so its weight on itself is 1.000 whatever its score is. Our hand computation matches the library exactly (difference 0.0).

<figure class="fig"><svg viewBox="0 0 760 400" role="img" aria-label="Real attention weights for "the kid smiles". Left: one head of GPT-2 layer 1, recomputed by hand; the upper triangle is exactly zero. Right: BERT layer 1, averaged over its 12 heads, including the [CLS] and [SEP] tokens; every square has some weight."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="155.2" y="40.0" text-anchor="middle">GPT-2, layer 1, head 1</text><text class="t-muted" x="155.2" y="58.0" text-anchor="middle">key (column)</text><text class="t-tick" x="99.2" y="76.0" text-anchor="middle">the</text><text class="t-tick" x="155.2" y="76.0" text-anchor="middle">kid</text><text class="t-tick" x="211.2" y="76.0" text-anchor="middle">smiles</text><text class="t-tick" x="63.2" y="116.0" text-anchor="end">the</text><g class="mark"><title>the attends to the: 1.00</title><rect class="s2" x="72.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:1.00"/></g><text class="t-cell on" x="99.2" y="116.0" text-anchor="middle">1.00</text><rect class="cell-masked" x="128.7" y="85.5" width="53.0" height="53.0" rx="3"/><line class="edge" x1="145.1" y1="101.9" x2="165.3" y2="122.1"/><line class="edge" x1="165.3" y1="101.9" x2="145.1" y2="122.1"/><rect class="cell-masked" x="184.7" y="85.5" width="53.0" height="53.0" rx="3"/><line class="edge" x1="201.1" y1="101.9" x2="221.3" y2="122.1"/><line class="edge" x1="221.3" y1="101.9" x2="201.1" y2="122.1"/><text class="t-tick" x="63.2" y="172.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to the: 0.76</title><rect class="s2" x="72.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.76"/></g><text class="t-cell on" x="99.2" y="172.0" text-anchor="middle">0.76</text><g class="mark"><title>kid attends to kid: 0.24</title><rect class="s2" x="128.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.24"/></g><text class="t-cell" x="155.2" y="172.0" text-anchor="middle">0.24</text><rect class="cell-masked" x="184.7" y="141.5" width="53.0" height="53.0" rx="3"/><line class="edge" x1="201.1" y1="157.9" x2="221.3" y2="178.1"/><line class="edge" x1="221.3" y1="157.9" x2="201.1" y2="178.1"/><text class="t-tick" x="63.2" y="228.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to the: 0.56</title><rect class="s2" x="72.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.56"/></g><text class="t-cell on" x="99.2" y="228.0" text-anchor="middle">0.56</text><g class="mark"><title>smiles attends to kid: 0.29</title><rect class="s2" x="128.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.29"/></g><text class="t-cell" x="155.2" y="228.0" text-anchor="middle">0.29</text><g class="mark"><title>smiles attends to smiles: 0.15</title><rect class="s2" x="184.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="211.2" y="228.0" text-anchor="middle">0.15</text><text class="t-muted" x="20.0" y="272.0" text-anchor="start">query (row)</text><text class="t-title" x="521.2" y="40.0" text-anchor="middle">BERT, layer 1, average of 12 heads</text><text class="t-muted" x="521.2" y="58.0" text-anchor="middle">key (column)</text><text class="t-tick" x="409.2" y="76.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="465.2" y="76.0" text-anchor="middle">the</text><text class="t-tick" x="521.2" y="76.0" text-anchor="middle">kid</text><text class="t-tick" x="577.2" y="76.0" text-anchor="middle">smiles</text><text class="t-tick" x="633.2" y="76.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="373.2" y="116.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]: 0.62</title><rect class="cell" x="382.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-cell on" x="409.2" y="116.0" text-anchor="middle">0.62</text><g class="mark"><title>[CLS] attends to the: 0.17</title><rect class="cell" x="438.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.17"/></g><text class="t-cell" x="465.2" y="116.0" text-anchor="middle">0.17</text><g class="mark"><title>[CLS] attends to kid: 0.05</title><rect class="cell" x="494.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="521.2" y="116.0" text-anchor="middle">0.05</text><g class="mark"><title>[CLS] attends to smiles: 0.05</title><rect class="cell" x="550.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="577.2" y="116.0" text-anchor="middle">0.05</text><g class="mark"><title>[CLS] attends to [SEP]: 0.11</title><rect class="cell" x="606.7" y="85.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="633.2" y="116.0" text-anchor="middle">0.11</text><text class="t-tick" x="373.2" y="172.0" text-anchor="end">the</text><g class="mark"><title>the attends to [CLS]: 0.32</title><rect class="cell" x="382.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.32"/></g><text class="t-cell" x="409.2" y="172.0" text-anchor="middle">0.32</text><g class="mark"><title>the attends to the: 0.13</title><rect class="cell" x="438.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.13"/></g><text class="t-cell" x="465.2" y="172.0" text-anchor="middle">0.13</text><g class="mark"><title>the attends to kid: 0.15</title><rect class="cell" x="494.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="521.2" y="172.0" text-anchor="middle">0.15</text><g class="mark"><title>the attends to smiles: 0.22</title><rect class="cell" x="550.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.22"/></g><text class="t-cell" x="577.2" y="172.0" text-anchor="middle">0.22</text><g class="mark"><title>the attends to [SEP]: 0.18</title><rect class="cell" x="606.7" y="141.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.18"/></g><text class="t-cell" x="633.2" y="172.0" text-anchor="middle">0.18</text><text class="t-tick" x="373.2" y="228.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to [CLS]: 0.19</title><rect class="cell" x="382.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.19"/></g><text class="t-cell" x="409.2" y="228.0" text-anchor="middle">0.19</text><g class="mark"><title>kid attends to the: 0.15</title><rect class="cell" x="438.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="465.2" y="228.0" text-anchor="middle">0.15</text><g class="mark"><title>kid attends to kid: 0.15</title><rect class="cell" x="494.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="521.2" y="228.0" text-anchor="middle">0.15</text><g class="mark"><title>kid attends to smiles: 0.32</title><rect class="cell" x="550.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.32"/></g><text class="t-cell" x="577.2" y="228.0" text-anchor="middle">0.32</text><g class="mark"><title>kid attends to [SEP]: 0.18</title><rect class="cell" x="606.7" y="197.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.18"/></g><text class="t-cell" x="633.2" y="228.0" text-anchor="middle">0.18</text><text class="t-tick" x="373.2" y="284.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to [CLS]: 0.21</title><rect class="cell" x="382.7" y="253.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.21"/></g><text class="t-cell" x="409.2" y="284.0" text-anchor="middle">0.21</text><g class="mark"><title>smiles attends to the: 0.10</title><rect class="cell" x="438.7" y="253.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="465.2" y="284.0" text-anchor="middle">0.10</text><g class="mark"><title>smiles attends to kid: 0.21</title><rect class="cell" x="494.7" y="253.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.21"/></g><text class="t-cell" x="521.2" y="284.0" text-anchor="middle">0.21</text><g class="mark"><title>smiles attends to smiles: 0.24</title><rect class="cell" x="550.7" y="253.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.24"/></g><text class="t-cell" x="577.2" y="284.0" text-anchor="middle">0.24</text><g class="mark"><title>smiles attends to [SEP]: 0.25</title><rect class="cell" x="606.7" y="253.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.25"/></g><text class="t-cell" x="633.2" y="284.0" text-anchor="middle">0.25</text><text class="t-tick" x="373.2" y="340.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]: 0.31</title><rect class="cell" x="382.7" y="309.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.31"/></g><text class="t-cell" x="409.2" y="340.0" text-anchor="middle">0.31</text><g class="mark"><title>[SEP] attends to the: 0.15</title><rect class="cell" x="438.7" y="309.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="465.2" y="340.0" text-anchor="middle">0.15</text><g class="mark"><title>[SEP] attends to kid: 0.07</title><rect class="cell" x="494.7" y="309.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="521.2" y="340.0" text-anchor="middle">0.07</text><g class="mark"><title>[SEP] attends to smiles: 0.22</title><rect class="cell" x="550.7" y="309.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.22"/></g><text class="t-cell" x="577.2" y="340.0" text-anchor="middle">0.22</text><g class="mark"><title>[SEP] attends to [SEP]: 0.24</title><rect class="cell" x="606.7" y="309.5" width="53.0" height="53.0" rx="3" style="fill-opacity:0.24"/></g><text class="t-cell" x="633.2" y="340.0" text-anchor="middle">0.24</text><text class="t-muted" x="330.0" y="384.0" text-anchor="start">query (row)</text><text class="t-muted" x="20.0" y="300.0" text-anchor="start">every row adds up to 1</text><text class="t-muted" x="20.0" y="318.0" text-anchor="start">grey × = weight forced to 0 by the mask</text></svg><figcaption>Real attention weights on the same words. Left: GPT-2, layer 1, head 1, with the blocked squares forced to 0. Right: BERT, layer 1, averaged over its 12 heads, on "[CLS] the kid smiles [SEP]" (BERT adds those two special tokens, explained in Part 2). Every square is filled, and every row still adds up to 1.</figcaption></figure>

So when the paper says BERT is "bidirectional", this is the concrete meaning: an all-zero mask. The hard part, which the rest of the paper solves, is how to *train* such a model, since the usual training game (predict the next word) breaks when the model can see the next word.

The abstract makes three claims. Let us translate each one.

**1. "Pre-train deep bidirectional representations from unlabeled text."** BERT learns from text that nobody has labelled: Wikipedia and a collection of books. Nobody had to mark which sentences are happy or sad. Labelled data is expensive. Plain text is almost free, and there is a lot of it.

> [!DEFINITION] Labeled and unlabeled data
> **Labeled** data has the right answer attached by a person, like a movie review marked "positive". **Unlabeled** data is just text, with no answers attached.

How can a model learn anything from text that has no answers attached? The trick is to make the answers out of the text itself. Hide a piece of the text, and the hidden piece *is* the answer.

<figure class="fig"><svg viewBox="0 0 760 342" role="img" aria-label="Self-supervised learning in three frames. Start from plain text. A language model turns it into (left side, next word) pairs. A masked language model hides a word and asks for it back, using both sides. In both cases the label was already in the text."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="10.0" y="10.0" width="740.0" height="64.0" rx="12"/><circle class="node on" cx="28.0" cy="28.0" r="11"/><text class="t-note" x="28.0" y="32.5" text-anchor="middle">1</text><text class="t-title" x="46.0" y="33.0" text-anchor="start">Start with plain text. Nobody labels anything.</text><rect class="box" x="372.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="399.0" y="53.5" text-anchor="middle">the</text><rect class="box" x="432.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="459.0" y="53.5" text-anchor="middle">kid</text><rect class="box" x="492.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="519.0" y="53.5" text-anchor="middle">smiles</text><rect class="box" x="552.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="579.0" y="53.5" text-anchor="middle">at</text><rect class="box" x="612.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="639.0" y="53.5" text-anchor="middle">the</text><rect class="box" x="672.0" y="34.0" width="54.0" height="30.0" rx="6"/><text class="t-tick" x="699.0" y="53.5" text-anchor="middle">dog</text><rect class="box" x="10.0" y="86.0" width="740.0" height="106.0" rx="12"/><circle class="node on" cx="28.0" cy="104.0" r="11"/><text class="t-note" x="28.0" y="108.5" text-anchor="middle">2</text><text class="t-title" x="46.0" y="109.0" text-anchor="start">Language model (GPT): input = the left side, label = the next word</text><rect class="box-2" x="40.0" y="118.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="137.5" text-anchor="middle">the</text><rect class="box-2" x="102.0" y="118.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="130.0" y="137.5" text-anchor="middle">kid</text><line class="edge" x1="164.0" y1="133.0" x2="194.0" y2="133.0" marker-end="url(#ah)"/><rect class="box-on" x="198.0" y="118.0" width="70.0" height="30.0" rx="6"/><text class="t-note" x="233.0" y="137.5" text-anchor="middle">smiles</text><text class="t-tick" x="372.0" y="138.0" text-anchor="start">P(smiles | the kid)</text><rect class="box-2" x="40.0" y="154.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="173.5" text-anchor="middle">the</text><rect class="box-2" x="102.0" y="154.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="130.0" y="173.5" text-anchor="middle">kid</text><rect class="box-2" x="164.0" y="154.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="192.0" y="173.5" text-anchor="middle">smiles</text><line class="edge" x1="226.0" y1="169.0" x2="256.0" y2="169.0" marker-end="url(#ah)"/><rect class="box-on" x="260.0" y="154.0" width="70.0" height="30.0" rx="6"/><text class="t-note" x="295.0" y="173.5" text-anchor="middle">at</text><text class="t-tick" x="372.0" y="174.0" text-anchor="start">P(at | the kid smiles)</text><rect class="box" x="10.0" y="204.0" width="740.0" height="106.0" rx="12"/><circle class="node on" cx="28.0" cy="222.0" r="11"/><text class="t-note" x="28.0" y="226.5" text-anchor="middle">3</text><text class="t-title" x="46.0" y="227.0" text-anchor="start">Masked LM (BERT): input = text with a hole, label = the hidden word</text><rect class="box-1" x="40.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="255.5" text-anchor="middle">the</text><rect class="box-1" x="102.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="130.0" y="255.5" text-anchor="middle">kid</text><rect class="box-mask" x="164.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="192.0" y="255.5" text-anchor="middle">[MASK]</text><rect class="box-1" x="226.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="254.0" y="255.5" text-anchor="middle">at</text><rect class="box-1" x="288.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="316.0" y="255.5" text-anchor="middle">the</text><rect class="box-1" x="350.0" y="236.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="378.0" y="255.5" text-anchor="middle">dog</text><line class="edge" x1="412.0" y1="251.0" x2="442.0" y2="251.0" marker-end="url(#ah)"/><rect class="box-on" x="446.0" y="236.0" width="70.0" height="30.0" rx="6"/><text class="t-note" x="481.0" y="255.5" text-anchor="middle">smiles</text><text class="t-muted" x="530.0" y="256.0" text-anchor="start">both sides are input</text><rect class="box-1" x="40.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="291.5" text-anchor="middle">the</text><rect class="box-1" x="102.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="130.0" y="291.5" text-anchor="middle">kid</text><rect class="box-1" x="164.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="192.0" y="291.5" text-anchor="middle">smiles</text><rect class="box-1" x="226.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="254.0" y="291.5" text-anchor="middle">at</text><rect class="box-1" x="288.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="316.0" y="291.5" text-anchor="middle">the</text><rect class="box-mask" x="350.0" y="272.0" width="56.0" height="30.0" rx="6"/><text class="t-tick" x="378.0" y="291.5" text-anchor="middle">[MASK]</text><line class="edge" x1="412.0" y1="287.0" x2="442.0" y2="287.0" marker-end="url(#ah)"/><rect class="box-on" x="446.0" y="272.0" width="70.0" height="30.0" rx="6"/><text class="t-note" x="481.0" y="291.5" text-anchor="middle">dog</text><text class="t-muted" x="530.0" y="292.0" text-anchor="start">both sides are input</text><text class="t-tick" x="380.0" y="330.0" text-anchor="middle">The labels come from the text itself, so every sentence ever written is training data.</text></svg><figcaption>Where the "labels" come from when nobody labels anything. A language model like GPT uses each next word as the answer for the words before it. BERT's masked language model hides a word in the middle and uses it as the answer, with both sides as input. Either way, every sentence ever written becomes training data.</figcaption></figure>

> [!DEFINITION] Self-supervised learning
> Learning from data whose labels are made automatically from the data itself, for example by hiding a word and asking for it back. The BERT paper calls this "unsupervised" (no person supervises it). Today it is usually called **self-supervised**.

**2. "Jointly conditioning on both left and right context in all layers."** When BERT builds the representation of a word, it uses the words before it *and* the words after it, and it does this at every layer, not just at the end. "Conditioning on" simply means "using as input".

**3. "Fine-tuned with just one additional output layer."** After pre-training, you take BERT, put one small new layer on top, and keep training the whole thing a little on your task. The same pre-trained BERT can become a sentiment classifier, a question-answering system or a name finder.

> [!DEFINITION] Fine-tuning
> Taking a pre-trained model and training it a bit more on a specific task with labeled examples. All the weights move a little; nothing starts from zero except the small new layer. It is much cheaper than pre-training: minutes or hours instead of days.

<figure class="fig"><svg viewBox="0 0 760 292" role="img" aria-label="BERT is pre-trained once on unlabeled text, then a copy is fine-tuned for each task with one small extra output layer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Step 1: pre-training (once, very expensive)</text><text class="t-title" x="430.0" y="26.0" text-anchor="start">Step 2: fine-tuning (per task, cheap)</text><rect class="box" x="30.0" y="52.0" width="130.0" height="90.0" rx="6"/><rect class="box" x="36.0" y="58.0" width="130.0" height="90.0" rx="6"/><rect class="box" x="42.0" y="64.0" width="130.0" height="90.0" rx="6"/><rect class="box" x="48.0" y="70.0" width="130.0" height="90.0" rx="6"/><text class="t-note" x="101.0" y="96.0" text-anchor="middle">Wikipedia</text><text class="t-note" x="101.0" y="116.0" text-anchor="middle">+ books</text><text class="t-muted" x="101.0" y="136.0" text-anchor="middle">no labels</text><text class="t-tick" x="101.0" y="178.0" text-anchor="middle">3.3 billion words</text><line class="edge" x1="178.0" y1="100.0" x2="228.0" y2="100.0" marker-end="url(#ah)"/><rect class="box-1" x="232.0" y="70.0" width="130.0" height="64.0" rx="12"/><text class="t-big" x="297.0" y="98.0" text-anchor="middle">BERT</text><text class="t-tick" x="297.0" y="118.0" text-anchor="middle">learns language</text><text class="t-tick" x="297.0" y="160.0" text-anchor="middle">all the weights</text><text class="t-tick" x="297.0" y="178.0" text-anchor="middle">are learned here</text><line class="edge" x1="364.0" y1="102.0" x2="444.0" y2="74.0" marker-end="url(#ah)"/><rect class="box-1" x="448.0" y="48.0" width="104.0" height="52.0" rx="10"/><text class="t-note" x="500.0" y="71.0" text-anchor="middle">BERT copy</text><text class="t-muted" x="500.0" y="89.0" text-anchor="middle">starts pre-trained</text><rect class="box-3" x="552.0" y="56.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="580.0" y1="74.0" x2="606.0" y2="74.0" marker-end="url(#ah)"/><text class="t-note" x="612.0" y="69.0" text-anchor="start">sentiment</text><text class="t-tick" x="612.0" y="87.0" text-anchor="start">positive / negative</text><line class="edge" x1="364.0" y1="102.0" x2="444.0" y2="148.0" marker-end="url(#ah)"/><rect class="box-1" x="448.0" y="122.0" width="104.0" height="52.0" rx="10"/><text class="t-note" x="500.0" y="145.0" text-anchor="middle">BERT copy</text><text class="t-muted" x="500.0" y="163.0" text-anchor="middle">starts pre-trained</text><rect class="box-2" x="552.0" y="130.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="580.0" y1="148.0" x2="606.0" y2="148.0" marker-end="url(#ah)"/><text class="t-note" x="612.0" y="143.0" text-anchor="start">question answering</text><text class="t-tick" x="612.0" y="161.0" text-anchor="start">find the answer span</text><line class="edge" x1="364.0" y1="102.0" x2="444.0" y2="222.0" marker-end="url(#ah)"/><rect class="box-1" x="448.0" y="196.0" width="104.0" height="52.0" rx="10"/><text class="t-note" x="500.0" y="219.0" text-anchor="middle">BERT copy</text><text class="t-muted" x="500.0" y="237.0" text-anchor="middle">starts pre-trained</text><rect class="box-4" x="552.0" y="204.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="580.0" y1="222.0" x2="606.0" y2="222.0" marker-end="url(#ah)"/><text class="t-note" x="612.0" y="217.0" text-anchor="start">named entities</text><text class="t-tick" x="612.0" y="235.0" text-anchor="start">person / place / ...</text><text class="t-tick" x="565.0" y="278.0" text-anchor="middle">+ one small output layer per task</text></svg><figcaption>The whole BERT recipe. Pre-train once on unlabeled text (the expensive step). Then, for every task, start from a copy of the pre-trained model, add one small output layer, and fine-tune.</figcaption></figure>

**Use case.** A company that wants to sort support emails into "billing", "bug report" and "feature request" does not need to teach a model English. It starts from pre-trained BERT and fine-tunes it on a few thousand labelled emails.

### Why an "understanding" model, when GPT-style models can write?

A fair question today: chat models write fluent text, so why read a paper about a model that does not write? Because writing is only one job. Many everyday language jobs are about **reading and deciding**: is this review positive, which department should get this email, where are the names in this contract, which passage answers this question. Images have the same split: generating a new picture is one job, recognising what is in a photo is another, and most practical vision systems do the second.

<figure class="fig"><svg viewBox="0 0 760 306" role="img" aria-label="Understanding tasks read a text and output a decision about it, such as a label or a span; BERT is built for these. Generation tasks write new text one word at a time; GPT is built for these. Images have the same split: recognising a cat in a photo versus drawing a new picture of a cat."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">Understanding: read, then decide</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">Generation: write something new</text><line class="edge-dim" x1="380.0" y1="40.0" x2="380.0" y2="300.0"/><text class="t-muted" x="20.0" y="58.0" text-anchor="start">text</text><rect class="box" x="20.0" y="66.0" width="340.0" height="32.0" rx="6"/><text class="t-tick" x="190.0" y="86.5" text-anchor="middle">The battery died after two days.</text><line class="edge" x1="190.0" y1="100.0" x2="190.0" y2="122.0" marker-end="url(#ah)"/><rect class="box-1" x="110.0" y="124.0" width="160.0" height="36.0" rx="10"/><text class="t-note" x="190.0" y="147.0" text-anchor="middle">BERT (encoder)</text><line class="edge" x1="190.0" y1="162.0" x2="190.0" y2="184.0" marker-end="url(#ah)"/><rect class="box-2" x="60.0" y="186.0" width="120.0" height="28.0" rx="6"/><text class="t-note" x="120.0" y="204.5" text-anchor="middle">negative</text><rect class="box-3" x="200.0" y="186.0" width="120.0" height="28.0" rx="6"/><text class="t-tick" x="260.0" y="204.5" text-anchor="middle">product: battery</text><text class="t-muted" x="190.0" y="232.0" text-anchor="middle">a label, a span, a tag per word</text><text class="t-muted" x="400.0" y="58.0" text-anchor="start">prompt</text><rect class="box" x="400.0" y="66.0" width="340.0" height="32.0" rx="6"/><text class="t-tick" x="570.0" y="86.5" text-anchor="middle">Write a review of this phone:</text><line class="edge" x1="570.0" y1="100.0" x2="570.0" y2="122.0" marker-end="url(#ah)"/><rect class="box-2" x="490.0" y="124.0" width="160.0" height="36.0" rx="10"/><text class="t-note" x="570.0" y="147.0" text-anchor="middle">GPT (decoder)</text><line class="edge" x1="570.0" y1="162.0" x2="570.0" y2="184.0" marker-end="url(#ah)"/><rect class="box-4" x="418.0" y="186.0" width="42.0" height="28.0" rx="6"/><text class="t-tick" x="439.0" y="204.5" text-anchor="middle">The</text><rect class="box-4" x="466.0" y="186.0" width="74.0" height="28.0" rx="6"/><text class="t-tick" x="503.0" y="204.5" text-anchor="middle">battery</text><rect class="box-4" x="546.0" y="186.0" width="34.0" height="28.0" rx="6"/><text class="t-tick" x="563.0" y="204.5" text-anchor="middle">is</text><rect class="box-4" x="586.0" y="186.0" width="50.0" height="28.0" rx="6"/><text class="t-tick" x="611.0" y="204.5" text-anchor="middle">weak</text><rect class="box-ghost" x="642.0" y="186.0" width="42.0" height="28.0" rx="6"/><text class="t-tick" x="663.0" y="204.5" text-anchor="middle">...</text><text class="t-muted" x="570.0" y="232.0" text-anchor="middle">new text, one word at a time</text><text class="t-note" x="20.0" y="270.0" text-anchor="start">Same split for images:</text><text class="t-tick" x="190.0" y="292.0" text-anchor="middle">recognise it: photo → "cat"</text><text class="t-tick" x="570.0" y="292.0" text-anchor="middle">draw it: "a cat" → new photo</text></svg><figcaption>Two families of jobs. Understanding: read the whole input, then output a label, a span or a tag per word (BERT's home ground). Generation: write new text one word at a time (GPT's home ground). The same split exists for images.</figcaption></figure>

For understanding jobs, a model that sees the whole input at once is a natural fit, and it can be small and fast. That is why BERT-style models still run inside many search, classification and retrieval systems (Part 6 shows where).

### What exactly gets trained when you fine-tune?

This is a common point of confusion, so it is worth stating carefully. The paper is explicit (we will see it in Section 3): "all of the parameters are fine-tuned". During fine-tuning, **every one of BERT's roughly 110 million weights keeps learning**, together with the small new output layer, which is the only part that starts from random numbers.

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="Which weights change. Feature-based: the pre-trained model is frozen and a separate task model is trained. Updating only the top layer is a common misreading and is not what BERT does. BERT fine-tuning updates every pre-trained weight, from the embeddings to layer 12, together with the new output layer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="130.0" y="22.0" text-anchor="middle">Feature-based</text><text class="t-title" x="380.0" y="22.0" text-anchor="middle">Not BERT: only the top</text><text class="t-title" x="630.0" y="22.0" text-anchor="middle">BERT fine-tuning</text><rect class="box-ghost" x="30.0" y="262.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="130.0" y="281.5" text-anchor="middle">embeddings</text><text class="t-muted" x="226.0" y="282.0" text-anchor="end">frozen</text><rect class="box-ghost" x="30.0" y="224.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="130.0" y="243.5" text-anchor="middle">layer 1</text><text class="t-muted" x="226.0" y="244.0" text-anchor="end">frozen</text><rect class="box-ghost" x="30.0" y="186.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="130.0" y="205.5" text-anchor="middle">layer 2</text><text class="t-muted" x="226.0" y="206.0" text-anchor="end">frozen</text><rect class="box-ghost" x="30.0" y="148.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="130.0" y="167.5" text-anchor="middle">...</text><text class="t-muted" x="226.0" y="168.0" text-anchor="end">frozen</text><rect class="box-ghost" x="30.0" y="110.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="130.0" y="129.5" text-anchor="middle">layer 12</text><text class="t-muted" x="226.0" y="130.0" text-anchor="end">frozen</text><rect class="box-3" x="30.0" y="66.0" width="200.0" height="34.0" rx="6"/><text class="t-note" x="130.0" y="87.5" text-anchor="middle">task model (BiLSTM ...)</text><text class="t-s3" x="130.0" y="58.0" text-anchor="middle">trained from zero</text><rect class="box-ghost" x="280.0" y="262.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="380.0" y="281.5" text-anchor="middle">embeddings</text><text class="t-muted" x="476.0" y="282.0" text-anchor="end">frozen</text><rect class="box-ghost" x="280.0" y="224.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="380.0" y="243.5" text-anchor="middle">layer 1</text><text class="t-muted" x="476.0" y="244.0" text-anchor="end">frozen</text><rect class="box-ghost" x="280.0" y="186.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="380.0" y="205.5" text-anchor="middle">layer 2</text><text class="t-muted" x="476.0" y="206.0" text-anchor="end">frozen</text><rect class="box-ghost" x="280.0" y="148.0" width="200.0" height="30.0" rx="6"/><text class="t-muted" x="380.0" y="167.5" text-anchor="middle">...</text><text class="t-muted" x="476.0" y="168.0" text-anchor="end">frozen</text><rect class="box-1" x="280.0" y="110.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="380.0" y="129.5" text-anchor="middle">layer 12</text><text class="t-s3" x="476.0" y="130.0" text-anchor="end">updated</text><rect class="box-3" x="330.0" y="66.0" width="100.0" height="34.0" rx="6"/><text class="t-note" x="380.0" y="87.5" text-anchor="middle">new layer</text><text class="t-s3" x="380.0" y="58.0" text-anchor="middle">trained from zero</text><rect class="box-1" x="530.0" y="262.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="630.0" y="281.5" text-anchor="middle">embeddings</text><text class="t-s3" x="726.0" y="282.0" text-anchor="end">updated</text><rect class="box-1" x="530.0" y="224.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="630.0" y="243.5" text-anchor="middle">layer 1</text><text class="t-s3" x="726.0" y="244.0" text-anchor="end">updated</text><rect class="box-1" x="530.0" y="186.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="630.0" y="205.5" text-anchor="middle">layer 2</text><text class="t-s3" x="726.0" y="206.0" text-anchor="end">updated</text><rect class="box-1" x="530.0" y="148.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="630.0" y="167.5" text-anchor="middle">...</text><text class="t-s3" x="726.0" y="168.0" text-anchor="end">updated</text><rect class="box-1" x="530.0" y="110.0" width="200.0" height="30.0" rx="6"/><text class="t-tick" x="630.0" y="129.5" text-anchor="middle">layer 12</text><text class="t-s3" x="726.0" y="130.0" text-anchor="end">updated</text><rect class="box-3" x="580.0" y="66.0" width="100.0" height="34.0" rx="6"/><text class="t-note" x="630.0" y="87.5" text-anchor="middle">new layer</text><text class="t-s3" x="630.0" y="58.0" text-anchor="middle">trained from zero</text><line class="bad-line" x1="260.0" y1="86.0" x2="500.0" y2="286.0"/><line class="bad-line" x1="500.0" y1="86.0" x2="260.0" y2="286.0"/><text class="t-muted" x="130.0" y="316.0" text-anchor="middle">ELMo-style</text><text class="t-bad" x="380.0" y="316.0" text-anchor="middle">a common misreading</text><text class="t-s3" x="630.0" y="316.0" text-anchor="middle">all 110M weights + the new layer</text></svg><figcaption>Three ways to reuse a pre-trained model. Left, feature-based (ELMo-style): the pre-trained layers are frozen and a separate task model is trained from zero. Middle, a common misreading of BERT: freeze everything and train only a new top layer. Right, what BERT actually does: all layers keep training, a little, together with the new layer.</figcaption></figure>

> [!WARNING] A common misreading
> "Fine-tuning BERT means freezing it and training only the last layer" is **not** what the paper does. Freezing BERT and training something on top is the *feature-based* approach, which the paper tests separately in Section 5.3 (Part 5). In fine-tuning, all weights move, but only a little, because training is short and the learning rate is small.

The idea itself came from computer vision, which the paper mentions in its related work (Section 2.3, Part 2): train a big network once on ImageNet, a large labelled photo collection, then reuse it for many smaller tasks. BERT brings that recipe to language, with one difference: its first stage needs no labels at all.

<figure class="fig"><svg viewBox="0 0 760 264" role="img" aria-label="Transfer learning in vision and in language. An image network pre-trained on ImageNet photos is reused, with a new last layer, for a small task like reading chest X-rays. BERT does the same for text: pre-train on Wikipedia and books, then fine-tune with a new layer on a small labelled task."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Transfer learning: learn once on a big general dataset, reuse for a small specific one</text><text class="t-note" x="20.0" y="64.0" text-anchor="start">Vision</text><rect class="box" x="20.0" y="76.0" width="220.0" height="44.0" rx="6"/><text class="t-tick" x="130.0" y="102.5" text-anchor="middle">ImageNet, 1.2M labelled photos</text><line class="edge" x1="244.0" y1="98.0" x2="276.0" y2="98.0" marker-end="url(#ah)"/><rect class="box-1" x="280.0" y="76.0" width="130.0" height="44.0" rx="6"/><text class="t-note" x="345.0" y="102.5" text-anchor="middle">image network</text><text class="t-muted" x="345.0" y="136.0" text-anchor="middle">pre-train</text><line class="edge" x1="414.0" y1="98.0" x2="446.0" y2="98.0" marker-end="url(#ah)"/><rect class="box-1" x="450.0" y="76.0" width="80.0" height="44.0" rx="6"/><text class="t-tick" x="490.0" y="102.5" text-anchor="middle">network</text><rect class="box-3" x="532.0" y="76.0" width="30.0" height="44.0" rx="6"/><text class="t-note" x="547.0" y="102.5" text-anchor="middle">+</text><text class="t-muted" x="505.0" y="136.0" text-anchor="middle">fine-tune on a few</text><text class="t-muted" x="505.0" y="150.0" text-anchor="middle">thousand examples</text><line class="edge" x1="566.0" y1="98.0" x2="590.0" y2="98.0" marker-end="url(#ah)"/><text class="t-tick" x="596.0" y="94.0" text-anchor="start">chest X-ray:</text><text class="t-tick" x="596.0" y="112.0" text-anchor="start">normal or not?</text><text class="t-note" x="20.0" y="168.0" text-anchor="start">Language (BERT, 2018)</text><rect class="box" x="20.0" y="180.0" width="220.0" height="44.0" rx="6"/><text class="t-tick" x="130.0" y="206.5" text-anchor="middle">Wikipedia + books, 3.3B words</text><line class="edge" x1="244.0" y1="202.0" x2="276.0" y2="202.0" marker-end="url(#ah)"/><rect class="box-1" x="280.0" y="180.0" width="130.0" height="44.0" rx="6"/><text class="t-note" x="345.0" y="206.5" text-anchor="middle">BERT</text><text class="t-muted" x="345.0" y="240.0" text-anchor="middle">pre-train</text><line class="edge" x1="414.0" y1="202.0" x2="446.0" y2="202.0" marker-end="url(#ah)"/><rect class="box-1" x="450.0" y="180.0" width="80.0" height="44.0" rx="6"/><text class="t-tick" x="490.0" y="206.5" text-anchor="middle">BERT</text><rect class="box-3" x="532.0" y="180.0" width="30.0" height="44.0" rx="6"/><text class="t-note" x="547.0" y="206.5" text-anchor="middle">+</text><text class="t-muted" x="505.0" y="240.0" text-anchor="middle">fine-tune on a few</text><text class="t-muted" x="505.0" y="254.0" text-anchor="middle">thousand examples</text><line class="edge" x1="566.0" y1="202.0" x2="590.0" y2="202.0" marker-end="url(#ah)"/><text class="t-tick" x="596.0" y="198.0" text-anchor="start">support email:</text><text class="t-tick" x="596.0" y="216.0" text-anchor="start">billing or bug?</text></svg><figcaption>Transfer learning in two fields. Vision: pre-train on ImageNet's labelled photos, then fine-tune for a narrow task such as reading chest X-rays. Language: pre-train BERT on unlabeled Wikipedia and books, then fine-tune for a narrow task such as routing support emails.</figcaption></figure>

> [!DEFINITION] Transfer learning
> Reusing what a model learned on one task (usually a big, general one) to do better on another task (usually a small, specific one). Pre-training followed by fine-tuning is transfer learning.

## The results in the abstract {§Abstract}

> [!PAPER] · Abstract
> [![The second paragraph of the BERT abstract: new state-of-the-art results on eleven tasks, GLUE score 80.5 percent, MultiNLI accuracy 86.7 percent, SQuAD v1.1 Test F1 93.2 and SQuAD v2.0 Test F1 83.1](https://ishwarj.com/img/papers/bert/p1-abstract-results.pngbstract-results.png)](/img/papers/bert/p1-abstract-results.png)
>
> **What it says:** BERT is "conceptually simple and empirically powerful". It sets new best results on eleven tasks, and the abstract lists four of them with how much it improved.
>
> **What it means:**
>
> Think of the world record in the 100 metres. It usually falls by a few hundredths of a second. A runner who takes off half a second at once changes the sport, not only the record.
>
> These were jumps of that kind. Benchmarks usually move by fractions of a point at a time, and here the GLUE score moved by 7.7 points.
>
> The two adjectives make two separate claims. "Conceptually simple" means no new kind of layer: the gains come from how the model is trained. "Empirically powerful" means the measured results, and eleven tasks show that one recipe works on very different jobs.

> [!DEFINITION] State of the art (SOTA)
> The best result anyone has published so far on a task. "New state of the art" means "better than every earlier system".

> [!DEFINITION] Benchmark
> A fixed, public test that everyone uses, so results can be compared fairly. A model is given the test questions, and its answers are scored.

Here are the four results from the abstract:

| Benchmark | What it tests | BERT's score | Improvement |
|---|---|---|---|
| GLUE | nine sentence-understanding tasks, averaged | 80.5 | +7.7 points |
| MultiNLI | does sentence B follow from sentence A? | 86.7% accuracy | +4.6 points |
| SQuAD v1.1 | find the answer to a question in a paragraph | 93.2 Test F1 | +1.5 points |
| SQuAD v2.0 | the same, but some questions have no answer | 83.1 Test F1 | +5.1 points |

> [!DEFINITION] Accuracy and F1
> **Accuracy** is the share of answers that are exactly right. **F1** is a score between 0 and 100 that gives partial credit: for question answering, it measures how many words of the predicted answer overlap with the true answer. Part 4 explains both properly.

A note on the wording "7.7% point absolute improvement": it means the score went up by 7.7 **points** (for example from 72.8 to 80.5), not by 7.7 percent of the old score. Part 4 goes through every one of these tasks and every table of results.

## Pre-training already worked, for two kinds of tasks {§1}

Now the introduction. Its first paragraph sets the scene.

> [!PAPER] · §1
> [![The first paragraph of the introduction: language model pre-training has been shown to be effective for sentence-level tasks such as natural language inference and paraphrasing, and token-level tasks such as named entity recognition and question answering](https://ishwarj.com/img/papers/bert/p1-intro-tasks.png/p1-intro-tasks.png)](/img/papers/bert/p1-intro-tasks.png)
>
> **What it says:** "Language model pre-training" already helps many tasks. These come in two kinds: **sentence-level tasks**, which look at whole sentences ("analyzing them holistically"), and **token-level tasks**, which need an answer for every single word ("fine-grained output at the token level").
>
> **What it means:**
>
> Think of a teacher who marks an essay. Sometimes the teacher gives one grade for the whole essay. Sometimes the teacher marks each wrong word, one by one. Sentence-level tasks are the first kind, and token-level tasks are the second kind.
>
> The two kinds need different things from a model. One grade needs a good summary of the whole text. A mark per word needs a good vector at every position, and each of those vectors must know its neighbours.
>
> BERT wants to be good at **both** kinds with the same model. Keep the two kinds in mind; the argument of the next paragraphs depends on them.

> [!DEFINITION] Token
> A small piece of text: usually a word, sometimes part of a word or a punctuation mark. Models read tokens, not letters. Part 2 shows exactly how BERT cuts text into tokens.

The paper names four example tasks:

> [!DEFINITION] Natural language inference (NLI)
> Given two sentences, decide if the second one **follows from** the first (entailment), **contradicts** it, or is **neutral**. "A man is playing a guitar" → "A person is making music": entailment.

> [!DEFINITION] Paraphrasing
> Deciding whether two sentences mean the same thing, even with different words.

> [!DEFINITION] Named entity recognition (NER)
> Marking every word that is part of a name, and what kind of name it is: a person, a place, an organisation, and so on.

> [!DEFINITION] Question answering (QA)
> Given a question and a paragraph, find the words in the paragraph that answer the question.

<figure class="fig"><svg viewBox="0 0 760 280" role="img" aria-label="Two kinds of tasks. Sentence-level tasks give one answer per input, like whether one sentence follows from another. Token-level tasks give an answer for every token, like marking which words are names of people or places."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Sentence-level task: one answer for the whole input</text><text class="t-title" x="20.0" y="168.0" text-anchor="start">Token-level task: one answer for every token</text><rect class="box" x="20.0" y="40.0" width="330.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="60.0" text-anchor="middle">A man is playing a guitar.</text><rect class="box" x="20.0" y="78.0" width="330.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="98.0" text-anchor="middle">A person is making music.</text><line class="edge" x1="354.0" y1="74.0" x2="420.0" y2="74.0" marker-end="url(#ah)"/><rect class="box-3" x="424.0" y="56.0" width="150.0" height="36.0" rx="8"/><text class="t-note" x="499.0" y="79.0" text-anchor="middle">entailment</text><text class="t-muted" x="590.0" y="70.0" text-anchor="start">the second sentence</text><text class="t-muted" x="590.0" y="86.0" text-anchor="start">follows from the first</text><rect class="box" x="20.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="68.0" y="205.5" text-anchor="middle">Ada</text><rect class="box" x="126.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="174.0" y="205.5" text-anchor="middle">Lovelace</text><rect class="box" x="232.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="280.0" y="205.5" text-anchor="middle">was</text><rect class="box" x="338.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="386.0" y="205.5" text-anchor="middle">born</text><rect class="box" x="444.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="492.0" y="205.5" text-anchor="middle">in</text><rect class="box" x="550.0" y="186.0" width="96.0" height="30.0" rx="6"/><text class="t-tick" x="598.0" y="205.5" text-anchor="middle">London</text><line class="edge" x1="68.0" y1="218.0" x2="68.0" y2="236.0" marker-end="url(#ah)"/><rect class="box-2" x="24.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-note" x="68.0" y="256.5" text-anchor="middle">PERSON</text><line class="edge" x1="174.0" y1="218.0" x2="174.0" y2="236.0" marker-end="url(#ah)"/><rect class="box-2" x="130.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-note" x="174.0" y="256.5" text-anchor="middle">PERSON</text><line class="edge" x1="280.0" y1="218.0" x2="280.0" y2="236.0" marker-end="url(#ah)"/><rect class="box" x="236.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-muted" x="280.0" y="256.5" text-anchor="middle">none</text><line class="edge" x1="386.0" y1="218.0" x2="386.0" y2="236.0" marker-end="url(#ah)"/><rect class="box" x="342.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-muted" x="386.0" y="256.5" text-anchor="middle">none</text><line class="edge" x1="492.0" y1="218.0" x2="492.0" y2="236.0" marker-end="url(#ah)"/><rect class="box" x="448.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-muted" x="492.0" y="256.5" text-anchor="middle">none</text><line class="edge" x1="598.0" y1="218.0" x2="598.0" y2="236.0" marker-end="url(#ah)"/><rect class="box-4" x="554.0" y="238.0" width="88.0" height="28.0" rx="6"/><text class="t-note" x="598.0" y="256.5" text-anchor="middle">PLACE</text></svg><figcaption>Two kinds of tasks. A sentence-level task (here, natural language inference) gives one answer for the whole input. A token-level task (here, named entity recognition) gives an answer for every token.</figcaption></figure>

The two token-level examples in that paragraph deserve a closer look, because both come back in Part 4.

<figure class="fig"><svg viewBox="0 0 760 304" role="img" aria-label="Two token-level tasks. Named entity recognition gives every token a tag: B-PER and I-PER mark the beginning and inside of a person name, B-LOC and I-LOC a place, O means outside any name. Question answering marks a start token and an end token in the passage; the answer is the span between them."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Named entity recognition: one tag per token</text><rect class="box-2" x="20.0" y="36.0" width="72.0" height="30.0" rx="6"/><text class="t-tick" x="56.0" y="55.5" text-anchor="middle">Barack</text><line class="edge" x1="56.0" y1="68.0" x2="56.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-2" x="20.0" y="86.0" width="72.0" height="24.0" rx="6"/><text class="t-tick" x="56.0" y="102.5" text-anchor="middle">B-PER</text><rect class="box-2" x="98.0" y="36.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="133.0" y="55.5" text-anchor="middle">Obama</text><line class="edge" x1="133.0" y1="68.0" x2="133.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-2" x="98.0" y="86.0" width="70.0" height="24.0" rx="6"/><text class="t-tick" x="133.0" y="102.5" text-anchor="middle">I-PER</text><rect class="box" x="174.0" y="36.0" width="46.0" height="30.0" rx="6"/><text class="t-tick" x="197.0" y="55.5" text-anchor="middle">was</text><line class="edge" x1="197.0" y1="68.0" x2="197.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-ghost" x="174.0" y="86.0" width="46.0" height="24.0" rx="6"/><text class="t-muted" x="197.0" y="102.5" text-anchor="middle">O</text><rect class="box" x="226.0" y="36.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="252.0" y="55.5" text-anchor="middle">born</text><line class="edge" x1="252.0" y1="68.0" x2="252.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-ghost" x="226.0" y="86.0" width="52.0" height="24.0" rx="6"/><text class="t-muted" x="252.0" y="102.5" text-anchor="middle">O</text><rect class="box" x="284.0" y="36.0" width="36.0" height="30.0" rx="6"/><text class="t-tick" x="302.0" y="55.5" text-anchor="middle">in</text><line class="edge" x1="302.0" y1="68.0" x2="302.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-ghost" x="284.0" y="86.0" width="36.0" height="24.0" rx="6"/><text class="t-muted" x="302.0" y="102.5" text-anchor="middle">O</text><rect class="box-3" x="326.0" y="36.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="361.0" y="55.5" text-anchor="middle">Hawaii</text><line class="edge" x1="361.0" y1="68.0" x2="361.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-3" x="326.0" y="86.0" width="70.0" height="24.0" rx="6"/><text class="t-tick" x="361.0" y="102.5" text-anchor="middle">B-LOC</text><rect class="box" x="402.0" y="36.0" width="26.0" height="30.0" rx="6"/><text class="t-tick" x="415.0" y="55.5" text-anchor="middle">,</text><line class="edge" x1="415.0" y1="68.0" x2="415.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-ghost" x="402.0" y="86.0" width="26.0" height="24.0" rx="6"/><text class="t-muted" x="415.0" y="102.5" text-anchor="middle">O</text><rect class="box-3" x="434.0" y="36.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="469.0" y="55.5" text-anchor="middle">United</text><line class="edge" x1="469.0" y1="68.0" x2="469.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-3" x="434.0" y="86.0" width="70.0" height="24.0" rx="6"/><text class="t-tick" x="469.0" y="102.5" text-anchor="middle">B-LOC</text><rect class="box-3" x="510.0" y="36.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="545.0" y="55.5" text-anchor="middle">States</text><line class="edge" x1="545.0" y1="68.0" x2="545.0" y2="84.0" marker-end="url(#ah)"/><rect class="box-3" x="510.0" y="86.0" width="70.0" height="24.0" rx="6"/><text class="t-tick" x="545.0" y="102.5" text-anchor="middle">I-LOC</text><path class="edge-2" d="M20.0,116.0 L20.0,124.0 L162.0,124.0 L162.0,116.0"/><text class="t-tick" x="91.0" y="142.0" text-anchor="middle">a person</text><path class="edge-3" d="M368.0,116.0 L368.0,124.0 L440.0,124.0 L440.0,116.0"/><text class="t-tick" x="404.0" y="142.0" text-anchor="middle">a place</text><path class="edge-3" d="M476.0,116.0 L476.0,124.0 L622.0,124.0 L622.0,116.0"/><text class="t-tick" x="549.0" y="142.0" text-anchor="middle">a place</text><text class="t-title" x="20.0" y="182.0" text-anchor="start">Question answering: point at the answer inside the passage</text><rect class="box-1" x="20.0" y="196.0" width="330.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="215.5" text-anchor="middle">Q: Where was Barack Obama born?</text><rect class="box" x="20.0" y="240.0" width="64.0" height="30.0" rx="6"/><text class="t-tick" x="52.0" y="259.5" text-anchor="middle">Barack</text><rect class="box" x="90.0" y="240.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="121.0" y="259.5" text-anchor="middle">Obama</text><rect class="box" x="158.0" y="240.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="178.0" y="259.5" text-anchor="middle">was</text><rect class="box" x="204.0" y="240.0" width="46.0" height="30.0" rx="6"/><text class="t-tick" x="227.0" y="259.5" text-anchor="middle">born</text><rect class="box" x="256.0" y="240.0" width="30.0" height="30.0" rx="6"/><text class="t-tick" x="271.0" y="259.5" text-anchor="middle">in</text><rect class="box-on" x="292.0" y="240.0" width="80.0" height="30.0" rx="6"/><text class="t-tick" x="332.0" y="259.5" text-anchor="middle">Honolulu</text><rect class="box-on" x="378.0" y="240.0" width="22.0" height="30.0" rx="6"/><text class="t-tick" x="389.0" y="259.5" text-anchor="middle">,</text><rect class="box-on" x="406.0" y="240.0" width="64.0" height="30.0" rx="6"/><text class="t-tick" x="438.0" y="259.5" text-anchor="middle">Hawaii</text><rect class="box" x="476.0" y="240.0" width="22.0" height="30.0" rx="6"/><text class="t-tick" x="487.0" y="259.5" text-anchor="middle">.</text><text class="t-on" x="294.0" y="290.0" text-anchor="start">▲ start</text><text class="t-on" x="468.0" y="290.0" text-anchor="end">end ▲</text><text class="t-tick" x="500.0" y="260.0" text-anchor="start">answer = the span</text><text class="t-note" x="500.0" y="276.0" text-anchor="start">"Honolulu, Hawaii"</text></svg><figcaption>The two token-level tasks the paragraph names. Named entity recognition gives every word a tag: B-PER starts a person's name, I-PER continues it, B-LOC starts a place, O means "not a name". Question answering points at two positions in the passage: where the answer starts and where it ends.</figcaption></figure>

> [!DEFINITION] Named entity recognition (NER)
> Finding the names of people, places, organisations and similar things in a text, and labelling each word. The usual tags are **B-** (beginning of a name), **I-** (inside a name) and **O** (outside any name), so "Barack Obama" becomes B-PER I-PER.

## Two ways to reuse a pre-trained model {§1}

> [!PAPER] · §1
> [![The second paragraph of the introduction: there are two existing strategies, feature-based and fine-tuning. ELMo uses task-specific architectures that include the pre-trained representations as additional features. OpenAI GPT is trained by simply fine-tuning all pre-trained parameters. Both use unidirectional language models](https://ishwarj.com/img/papers/bert/p1-intro-strategies.pngntro-strategies.png)](/img/papers/bert/p1-intro-strategies.png)
>
> **What it says:** in 2018 there were two ways to use a pre-trained model. The **feature-based** approach (ELMo) feeds the pre-trained representations as extra inputs into a model built specially for each task. The **fine-tuning** approach (OpenAI GPT) adds very few new weights and fine-tunes "all pre-trained parameters". Both learn during pre-training with "unidirectional language models".
>
> **What it means:**
>
> Think of a hired expert. In the feature-based approach, the expert writes a report and leaves. Your own team, built for the task, uses that report as one more input. In the fine-tuning approach, the expert joins your team and keeps learning on the job.
>
> BERT picks the fine-tuning side. The reason is in the word "all": every pre-trained weight can adapt to the task, and almost nothing must be built new for each task.
>
> The last sentence names the weakness BERT attacks: both approaches pre-train in only one direction. The next paragraphs explain why that is a real limit.

> [!DEFINITION] Downstream task
> The real task you care about in the end (sentiment, question answering, and so on), as opposed to the pre-training task. It sits "downstream" of pre-training.

> [!DEFINITION] ELMo
> A 2018 model by Peters et al. (the "a" in "2018a" just tells two papers by the same authors apart). It reads text with two separate networks, one left to right and one right to left, and gives every word a vector that depends on its sentence. Its vectors are used as **features**: extra inputs for another model.

> [!DEFINITION] OpenAI GPT
> A 2018 model by Radford et al. at OpenAI (the first "Generative Pre-trained Transformer"). A Transformer trained to predict the next word, left to right. To use it for a task, you fine-tune the whole model with a small extra layer. BERT copies this recipe, and changes the direction.

<figure class="fig"><svg viewBox="0 0 760 236" role="img" aria-label="Two ways to reuse a pre-trained model. Feature-based: the model is frozen and only produces input features for a separate task model. Fine-tuning: the whole model keeps training on the task, with one tiny new layer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">Feature-based (ELMo)</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">Fine-tuning (OpenAI GPT, BERT)</text><rect class="box" x="60.0" y="190.0" width="260.0" height="30.0" rx="6"/><text class="t-tick" x="190.0" y="210.0" text-anchor="middle">your sentence</text><line class="edge" x1="190.0" y1="188.0" x2="190.0" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="60.0" y="116.0" width="260.0" height="46.0" rx="10"/><rect class="box-1" x="60.0" y="116.0" width="260.0" height="46.0" rx="10"/><text class="t-note" x="190.0" y="137.0" text-anchor="middle">pre-trained model</text><text class="t-muted" x="190.0" y="154.0" text-anchor="middle">frozen: its weights never change</text><line class="edge" x1="190.0" y1="114.0" x2="190.0" y2="92.0" marker-end="url(#ah)"/><text class="t-muted" x="200.0" y="104.0" text-anchor="start">features (vectors)</text><rect class="box-2" x="60.0" y="44.0" width="260.0" height="46.0" rx="10"/><text class="t-note" x="190.0" y="65.0" text-anchor="middle">task-specific model</text><text class="t-muted" x="190.0" y="82.0" text-anchor="middle">designed by hand, trained from zero</text><rect class="box" x="440.0" y="190.0" width="260.0" height="30.0" rx="6"/><text class="t-tick" x="570.0" y="210.0" text-anchor="middle">your sentence</text><line class="edge" x1="570.0" y1="188.0" x2="570.0" y2="164.0" marker-end="url(#ah)"/><rect class="box-1" x="440.0" y="98.0" width="260.0" height="64.0" rx="10"/><text class="t-note" x="570.0" y="124.0" text-anchor="middle">pre-trained model</text><text class="t-muted" x="570.0" y="143.0" text-anchor="middle">every weight is updated a little</text><line class="edge" x1="570.0" y1="96.0" x2="570.0" y2="80.0" marker-end="url(#ah)"/><rect class="box-3" x="520.0" y="50.0" width="100.0" height="28.0" rx="8"/><text class="t-tick" x="570.0" y="69.0" text-anchor="middle">tiny layer</text><text class="t-muted" x="570.0" y="40.0" text-anchor="middle">the only new part</text><line class="edge-dim" x1="380.0" y1="40.0" x2="380.0" y2="222.0"/></svg><figcaption>The two strategies. Feature-based: the pre-trained model is frozen and only supplies features to a separate task model. Fine-tuning: the whole pre-trained model keeps training on the task, and only a tiny layer is new.</figcaption></figure>

Both approaches, the paper says, pre-train with a **language model**. That word is the key to the whole argument, so let us define it carefully.

> [!DEFINITION] Language model (LM)
> A model that gives a probability to the next word, given the words before it. Shown "I went to the", a good language model gives high probability to "store", "park" or "bank", and low probability to "elephant".

A language model reads left to right and predicts each word from the words before it. Written as an equation, the probability of a whole sentence is built up one word at a time:

$$
P(w_1, w_2, \dots, w_n) = \prod_{i=1}^{n} P(w_i \mid w_1, \dots, w_{i-1})
$$

where:

- $$w_1, \dots, w_n$$ are the words of the sentence, in order, and $$n$$ is how many there are;
- $$P(w_i \mid w_1, \dots, w_{i-1})$$ is the probability of word $$i$$ **given** (that is what the bar $$\mid$$ means) all the words before it;
- $$\prod$$ means "multiply all of these together", for $$i$$ from 1 to $$n$$.

Look at what is on the right of the bar: only earlier words. That is what **unidirectional** means. The training signal is free (the next word is always in the text), which is why language models are such a good way to pre-train. But each word only ever learns from its left.

**The chain rule with real numbers.** Here is the equation applied to "The kid smiles at the dog." by GPT-2 ([`bert_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1_math.py)). Each row is one factor of the product: the probability GPT-2 gave the real next word, seeing only the words on its left.

```text
 i  word     given (left side only)     P(word | left)    log P
 1  The      (start)                          0.037700   -3.278
 2  kid      The                              0.000040  -10.114
 3  smiles   The kid                          0.000257   -8.268
 4  at       The kid smiles                   0.087426   -2.437
 5  the      The kid smiles at                0.150418   -1.894
 6  dog      The kid smiles at the            0.002395   -6.034
 7  .        The kid smiles at the dog        0.177917   -1.726
sum of log P = -33.752
P(sentence) = exp(-33.752) = 2.196e-15
average negative log-likelihood = 4.822  (perplexity exp of that = 124.2)
```

<figure class="fig"><svg viewBox="0 0 760 346" role="img" aria-label="The chain rule on a real model. GPT-2 gives each token a probability given only the tokens on its left. Multiplying the seven probabilities gives the probability of the whole sentence, about 2.2 in a million billion; adding their logarithms gives the same information as -33.752."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">P("The kid smiles at the dog.") built one word at a time, left side only (GPT-2)</text><text class="t-muted" x="20.0" y="64.0" text-anchor="start">(start)</text><line class="edge" x1="196.0" y1="59.0" x2="216.0" y2="59.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="44.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="62.5" text-anchor="middle">The</text><g class="mark"><title>P(The | (start)) = 0.0377</title><rect class="s2" x="310" y="50" width="210.6" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="526.6" y="63.0" text-anchor="start">0.0377</text><text class="t-muted" x="20.0" y="98.0" text-anchor="start">The</text><line class="edge" x1="196.0" y1="93.0" x2="216.0" y2="93.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="78.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="96.5" text-anchor="middle">kid</text><g class="mark"><title>P(kid | The) = 4e-05</title><rect class="s2" x="310" y="84" width="24.2" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="340.2" y="97.0" text-anchor="start">0.00004</text><text class="t-muted" x="20.0" y="132.0" text-anchor="start">The kid</text><line class="edge" x1="196.0" y1="127.0" x2="216.0" y2="127.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="112.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="130.5" text-anchor="middle">smiles</text><g class="mark"><title>P(smiles | The kid) = 0.000257</title><rect class="s2" x="310" y="118" width="74.5" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="390.5" y="131.0" text-anchor="start">0.000257</text><text class="t-muted" x="20.0" y="166.0" text-anchor="start">The kid smiles</text><line class="edge" x1="196.0" y1="161.0" x2="216.0" y2="161.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="146.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="164.5" text-anchor="middle">at</text><g class="mark"><title>P(at | The kid smiles) = 0.087426</title><rect class="s2" x="310" y="152" width="233.5" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="549.5" y="165.0" text-anchor="start">0.0874</text><text class="t-muted" x="20.0" y="200.0" text-anchor="start">The kid smiles at</text><line class="edge" x1="196.0" y1="195.0" x2="216.0" y2="195.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="180.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="198.5" text-anchor="middle">the</text><g class="mark"><title>P(the | The kid smiles at) = 0.150418</title><rect class="s2" x="310" y="186" width="248.3" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="564.3" y="199.0" text-anchor="start">0.1504</text><text class="t-muted" x="20.0" y="234.0" text-anchor="start">The kid smiles at the</text><line class="edge" x1="196.0" y1="229.0" x2="216.0" y2="229.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="214.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="232.5" text-anchor="middle">dog</text><g class="mark"><title>P(dog | The kid smiles at the) = 0.002395</title><rect class="s2" x="310" y="220" width="135.4" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="451.4" y="233.0" text-anchor="start">0.0024</text><text class="t-muted" x="20.0" y="268.0" text-anchor="start">The kid smiles at the dog</text><line class="edge" x1="196.0" y1="263.0" x2="216.0" y2="263.0" marker-end="url(#ah)"/><rect class="box-2" x="220.0" y="248.0" width="74.0" height="28.0" rx="6"/><text class="t-note" x="257.0" y="266.5" text-anchor="middle">.</text><g class="mark"><title>P(. | The kid smiles at the dog) = 0.177917</title><rect class="s2" x="310" y="254" width="252.9" height="16" rx="3" style="fill-opacity:0.8"/></g><text class="t-val" x="568.9" y="267.0" text-anchor="start">0.1779</text><line class="edge-dim" x1="20.0" y1="290.0" x2="740.0" y2="290.0"/><text class="t-note" x="20.0" y="312.0" text-anchor="start">multiply the seven:  P(sentence) = 2.20e-15</text><text class="t-tick" x="20.0" y="332.0" text-anchor="start">add the logs:  sum of log P = -33.752   (the same number, safe from underflow)</text><text class="t-muted" x="310.0" y="36.0" text-anchor="start">bar length = log P (longer = more likely)</text></svg><figcaption>The chain rule, one factor per word. Each bar is the probability GPT-2 gave the real next word from its left side only. Multiplying all seven gives the probability of the whole sentence, 2.2 × 10⁻¹⁵.</figcaption></figure>

Three things to notice:

- **Multiplying tiny numbers underflows**, so in practice everyone adds logarithms instead: $$\log \prod_i p_i = \sum_i \log p_i$$. Here the seven logs add up to $$-33.752$$, and $$e^{-33.752} = 2.2 \times 10^{-15}$$, the same probability.
- **Training a language model** means making this sum as large as possible over billions of sentences. Written as a loss to make small, it is the **negative log-likelihood**:

$$
\mathcal{L}_{\text{LM}} = -\sum_{i=1}^{n} \log P(w_i \mid w_1, \dots, w_{i-1})
$$

  where $$\mathcal{L}$$ is the loss (lower is better) and the other symbols are as above. For our sentence, $$\mathcal{L} = 33.752$$, or $$4.822$$ per word.
- **"kid" got only 0.00004.** After "The", thousands of words are possible, so no single one gets much probability. Every word is predicted from its left only; the model never got to use "smiles at the dog" to help with "kid".

> [!DEFINITION] Logarithm and log-likelihood
> The natural logarithm $$\log x$$ answers "$$e$$ to what power gives $$x$$?", with $$e \approx 2.718$$. It turns multiplication into addition, and small probabilities into manageable negative numbers ($$\log 0.00004 = -10.1$$). The **log-likelihood** of a text is the sum of the logs of the probabilities a model gave its words.

> [!DEFINITION] Perplexity
> $$e$$ raised to the average negative log-likelihood per word. It reads as "on average, the model was as unsure as if it were choosing among this many equally likely words". Here 124.2. Lower is better. Part 5 uses perplexity to compare BERT sizes.

> [!DEFINITION] Objective function (training objective)
> The score a model is trained to improve. For a language model, it is "give high probability to the real next word". Training means nudging the weights, millions of times, so this score gets better.

## The problem: reading in only one direction {§1}

> [!PAPER] · §1
> [![The third paragraph of the introduction: the major limitation is that standard language models are unidirectional. In OpenAI GPT every token can only attend to previous tokens. This is sub-optimal for sentence-level tasks and could be very harmful for token-level tasks such as question answering, where it is crucial to incorporate context from both directions](https://ishwarj.com/img/papers/bert/p1-intro-limitation.pngntro-limitation.png)](/img/papers/bert/p1-intro-limitation.png)
>
> **What it says:** "The major limitation is that standard language models are unidirectional." In OpenAI GPT, "every token can only attend to previous tokens". This is "sub-optimal" for sentence-level tasks and "could be very harmful" for token-level tasks like question answering, "where it is crucial to incorporate context from both directions".
>
> **What it means:**
>
> Read a mystery novel and stop halfway through. You can guess the killer, but the real clue often comes later in the book. A left-to-right model is in that place at every word: it builds each word before it sees the words after it.
>
> The authors grade the damage with care. For a whole-sentence task it is only "sub-optimal", because the last token has seen the whole sentence. For a per-word task it "could be very harmful", because the answer at each position depends on both sides.
>
> This paragraph is the problem statement of the whole paper. Everything else in the paper is the solution and the evidence.

> [!DEFINITION] Attend to
> In a Transformer, "token A attends to token B" means A looks at B and takes information from it. In GPT, a token may attend only to itself and the tokens before it. A **mask** blocks every token to the right.

Why is one direction a problem? Because the meaning of a word often depends on what comes **after** it. Read this sentence and stop at the blank:

> I went to the ____

You cannot know the missing word. Now read the whole sentence:

> I went to the ____ to deposit my paycheck.

Now it is obviously "bank". The clue was on the right. A left-to-right model building the representation of that position has not seen "deposit my paycheck" yet, so it cannot use it.

<figure class="fig"><svg viewBox="0 0 760 470" role="img" aria-label="What the blank can see. A left-to-right model sees only the earlier words. ELMo runs a left reader and a right reader separately and joins their outputs at the end. BERT lets every layer look at both sides."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Left-to-right (OpenAI GPT)</text><text class="t-tick" x="20.0" y="44.0" text-anchor="start">the blank sees only the words before it</text><rect class="box-on" x="440.0" y="66.0" width="62.0" height="24.0" rx="6"/><text class="t-muted" x="471.0" y="82.0" text-anchor="middle">layer</text><line class="edge-1" x1="261.0" y1="114.0" x2="439.5" y2="90.0"/><line class="edge-1" x1="331.0" y1="114.0" x2="450.0" y2="90.0"/><line class="edge-1" x1="401.0" y1="114.0" x2="460.5" y2="90.0"/><line class="edge-1" x1="471.0" y1="114.0" x2="471.0" y2="90.0"/><text class="t-muted" x="541.0" y="162.0" text-anchor="middle">hidden</text><text class="t-muted" x="611.0" y="162.0" text-anchor="middle">hidden</text><text class="t-muted" x="681.0" y="162.0" text-anchor="middle">hidden</text><rect class="box" x="230.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="261.0" y="135.5" text-anchor="middle">went</text><rect class="box" x="300.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="331.0" y="135.5" text-anchor="middle">to</text><rect class="box" x="370.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="401.0" y="135.5" text-anchor="middle">the</text><rect class="box-mask" x="440.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="471.0" y="135.5" text-anchor="middle">?</text><rect class="box" x="510.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="541.0" y="135.5" text-anchor="middle">to</text><rect class="box" x="580.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="611.0" y="135.5" text-anchor="middle">deposit</text><rect class="box" x="650.0" y="116.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="681.0" y="135.5" text-anchor="middle">my</text><text class="t-title" x="20.0" y="176.0" text-anchor="start">Two one-way readers, joined (ELMo)</text><text class="t-tick" x="20.0" y="194.0" text-anchor="start">each reader sees one side; they meet only at the end</text><rect class="box-1" x="306.0" y="220.0" width="120.0" height="24.0" rx="6"/><text class="t-tick" x="366.0" y="236.0" text-anchor="middle">left reader</text><rect class="box-2" x="516.0" y="220.0" width="120.0" height="24.0" rx="6"/><text class="t-tick" x="576.0" y="236.0" text-anchor="middle">right reader</text><rect class="box-on" x="440.0" y="182.0" width="62.0" height="24.0" rx="6"/><text class="t-muted" x="471.0" y="198.0" text-anchor="middle">join</text><line class="edge-1" x1="406.0" y1="220.0" x2="459.0" y2="206.0"/><line class="edge-2" x1="536.0" y1="220.0" x2="483.0" y2="206.0"/><line class="edge-1" x1="261.0" y1="264.0" x2="313.5" y2="244.0"/><line class="edge-1" x1="331.0" y1="264.0" x2="348.5" y2="244.0"/><line class="edge-1" x1="401.0" y1="264.0" x2="383.5" y2="244.0"/><line class="edge-1" x1="471.0" y1="264.0" x2="418.5" y2="244.0"/><line class="edge-2" x1="471.0" y1="264.0" x2="523.5" y2="244.0"/><line class="edge-2" x1="541.0" y1="264.0" x2="558.5" y2="244.0"/><line class="edge-2" x1="611.0" y1="264.0" x2="593.5" y2="244.0"/><line class="edge-2" x1="681.0" y1="264.0" x2="628.5" y2="244.0"/><rect class="box" x="230.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="261.0" y="285.5" text-anchor="middle">went</text><rect class="box" x="300.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="331.0" y="285.5" text-anchor="middle">to</text><rect class="box" x="370.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="401.0" y="285.5" text-anchor="middle">the</text><rect class="box-mask" x="440.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="471.0" y="285.5" text-anchor="middle">?</text><rect class="box" x="510.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="541.0" y="285.5" text-anchor="middle">to</text><rect class="box" x="580.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="611.0" y="285.5" text-anchor="middle">deposit</text><rect class="box" x="650.0" y="266.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="681.0" y="285.5" text-anchor="middle">my</text><text class="t-title" x="20.0" y="326.0" text-anchor="start">Deeply bidirectional (BERT)</text><text class="t-tick" x="20.0" y="344.0" text-anchor="start">the blank sees both sides, in every layer</text><rect class="box-on" x="440.0" y="366.0" width="62.0" height="24.0" rx="6"/><text class="t-muted" x="471.0" y="382.0" text-anchor="middle">layer</text><text class="t-muted" x="515.0" y="382.0" text-anchor="start">the same in all 12 layers</text><line class="edge-on" x1="261.0" y1="414.0" x2="439.5" y2="390.0"/><line class="edge-on" x1="331.0" y1="414.0" x2="450.0" y2="390.0"/><line class="edge-on" x1="401.0" y1="414.0" x2="460.5" y2="390.0"/><line class="edge-on" x1="471.0" y1="414.0" x2="471.0" y2="390.0"/><line class="edge-on" x1="541.0" y1="414.0" x2="481.5" y2="390.0"/><line class="edge-on" x1="611.0" y1="414.0" x2="492.0" y2="390.0"/><line class="edge-on" x1="681.0" y1="414.0" x2="502.5" y2="390.0"/><rect class="box" x="230.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="261.0" y="435.5" text-anchor="middle">went</text><rect class="box" x="300.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="331.0" y="435.5" text-anchor="middle">to</text><rect class="box" x="370.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="401.0" y="435.5" text-anchor="middle">the</text><rect class="box-mask" x="440.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="471.0" y="435.5" text-anchor="middle">?</text><rect class="box" x="510.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="541.0" y="435.5" text-anchor="middle">to</text><rect class="box" x="580.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="611.0" y="435.5" text-anchor="middle">deposit</text><rect class="box" x="650.0" y="416.0" width="62.0" height="30.0" rx="6"/><text class="t-tick" x="681.0" y="435.5" text-anchor="middle">my</text></svg><figcaption>What the blank can see in three kinds of model. Left to right (GPT): only the earlier words. ELMo: a left reader and a right reader that work separately and are joined only at the very end. BERT: both sides, in every layer.</figcaption></figure>

### Let us test it on real models

I gave the same three sentences to two real, public models:

- **GPT-2**, a left-to-right model from 2019 (the small version). It sees only the words before the blank and predicts the next word.
- **BERT** (`bert-base-uncased`, the model released with this paper). It sees the whole sentence with the blank replaced by a special `[MASK]` token, and predicts the missing word.

```python
from transformers import AutoTokenizer, BertForMaskedLM, GPT2LMHeadModel
import torch

btok = AutoTokenizer.from_pretrained("bert-base-uncased")
bert = BertForMaskedLM.from_pretrained("bert-base-uncased").eval()
gtok = AutoTokenizer.from_pretrained("openai-community/gpt2")
gpt2 = GPT2LMHeadModel.from_pretrained("openai-community/gpt2").eval()

# GPT-2: only the left side, predict the next token
ids = gtok("I went to the", return_tensors="pt").input_ids
p_gpt2 = torch.softmax(gpt2(ids).logits[0, -1], -1)

# BERT: the whole sentence, predict the [MASK]
enc = btok("i went to the [MASK] to deposit my paycheck.", return_tensors="pt")
pos = (enc.input_ids[0] == btok.mask_token_id).nonzero().item()
p_bert = torch.softmax(bert(**enc).logits[0, pos], -1)
```

These are the five most likely words each model gave, with their probabilities (the real output of [`bert_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1.py)):

```text
sentence: i went to the ____ to deposit my paycheck.   (hidden word: bank)
  GPT-2, left side only: hospital 0.033, doctor 0.024, store 0.022, gym 0.018, office 0.015
  BERT, both sides:      bank 0.901, office 0.009, store 0.008, teller 0.008, atm 0.006

sentence: my neighbor's ____ barked at the mailman all morning.   (hidden word: dog)
  GPT-2, left side only: house 0.071, dog 0.037, son 0.026, car 0.024, daughter 0.023
  BERT, both sides:      dog 0.651, dogs 0.142, voice 0.020, phone 0.018, had 0.014

sentence: she picked up her ____ and started to play a song.   (hidden word: guitar)
  GPT-2, left side only: phone 0.100, bag 0.033, gun 0.018, daughter 0.014, own 0.013
  BERT, both sides:      guitar 0.677, phone 0.081, ipod 0.050, fiddle 0.017, instrument 0.013
```

<figure class="fig"><svg viewBox="0 0 760 200" role="img" aria-label="Top five guesses for the blank. GPT-2, reading left to right, has no way to know about the paycheck and spreads its guesses. BERT sees "to deposit my paycheck" and puts 0.901 on "bank"."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">I went to the ____ to deposit my paycheck.</text><text class="t-title" x="20.0" y="56.0" text-anchor="start">GPT-2 (sees "I went to the")</text><text class="t-tick" x="92.0" y="82.0" text-anchor="end">hospital</text><g class="mark"><title>hospital: 0.033</title><rect class="s2" x="100.0" y="69.0" width="7.6" height="16.0" rx="3"/></g><text class="t-val" x="113.6" y="82.0" text-anchor="start">0.033</text><text class="t-tick" x="92.0" y="106.0" text-anchor="end">doctor</text><g class="mark"><title>doctor: 0.024</title><rect class="s2" x="100.0" y="93.0" width="5.5" height="16.0" rx="3"/></g><text class="t-val" x="111.5" y="106.0" text-anchor="start">0.024</text><text class="t-tick" x="92.0" y="130.0" text-anchor="end">store</text><g class="mark"><title>store: 0.022</title><rect class="s2" x="100.0" y="117.0" width="5.1" height="16.0" rx="3"/></g><text class="t-val" x="111.1" y="130.0" text-anchor="start">0.022</text><text class="t-tick" x="92.0" y="154.0" text-anchor="end">gym</text><g class="mark"><title>gym: 0.018</title><rect class="s2" x="100.0" y="141.0" width="4.1" height="16.0" rx="3"/></g><text class="t-val" x="110.1" y="154.0" text-anchor="start">0.018</text><text class="t-tick" x="92.0" y="178.0" text-anchor="end">office</text><g class="mark"><title>office: 0.015</title><rect class="s2" x="100.0" y="165.0" width="3.4" height="16.0" rx="3"/></g><text class="t-val" x="109.5" y="178.0" text-anchor="start">0.015</text><text class="t-title" x="400.0" y="56.0" text-anchor="start">BERT (sees both sides)</text><text class="t-tick t-strong" x="462.0" y="82.0" text-anchor="end">bank</text><g class="mark"><title>bank: 0.901</title><rect class="s1" x="470.0" y="69.0" width="207.2" height="16.0" rx="3"/></g><text class="t-val" x="683.2" y="82.0" text-anchor="start">0.901</text><text class="t-tick" x="462.0" y="106.0" text-anchor="end">office</text><g class="mark"><title>office: 0.009</title><rect class="s1" x="470.0" y="93.0" width="2.1" height="16.0" rx="3"/></g><text class="t-val" x="478.1" y="106.0" text-anchor="start">0.009</text><text class="t-tick" x="462.0" y="130.0" text-anchor="end">store</text><g class="mark"><title>store: 0.008</title><rect class="s1" x="470.0" y="117.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="478.0" y="130.0" text-anchor="start">0.008</text><text class="t-tick" x="462.0" y="154.0" text-anchor="end">teller</text><g class="mark"><title>teller: 0.008</title><rect class="s1" x="470.0" y="141.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="478.0" y="154.0" text-anchor="start">0.008</text><text class="t-tick" x="462.0" y="178.0" text-anchor="end">atm</text><g class="mark"><title>atm: 0.006</title><rect class="s1" x="470.0" y="165.0" width="2.0" height="16.0" rx="3"/></g><text class="t-val" x="478.0" y="178.0" text-anchor="start">0.006</text></svg><figcaption>The first sentence as a picture. GPT-2 cannot see "to deposit my paycheck" and spreads its guesses over places you might go. BERT sees both sides and puts 0.901 on "bank".</figcaption></figure>

Both sides do not make every word easy, though. Take "smiles" from our chain-rule sentence:

```text
GPT-2  P(smiles | "The kid")                    = 0.0003   top 5: who 0.1939, is 0.0687, in 0.0596, was 0.0550, 's 0.0507
BERT   P(smiles | "the kid [MASK] at the dog.") = 0.0008   top 5: looked 0.3233, stared 0.1029, glanced 0.0795, pointed 0.0620, glared 0.0595
```

<figure class="fig"><svg viewBox="0 0 760 276" role="img" aria-label="The same position predicted two ways. GPT-2 sees only "The kid" and guesses words like who, is and was. BERT sees "at the dog" on the right and guesses verbs that take "at": looked, stared, glanced, pointed, glared."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Predicting "smiles" with one side versus with both sides</text><rect class="box-2" x="20.0" y="40.0" width="340.0" height="30.0" rx="6"/><text class="t-tick" x="190.0" y="59.5" text-anchor="middle">GPT-2 sees:  The kid ____</text><rect class="box-1" x="400.0" y="40.0" width="340.0" height="30.0" rx="6"/><text class="t-tick" x="570.0" y="59.5" text-anchor="middle">BERT sees:  the kid [MASK] at the dog.</text><text class="t-tick" x="82.0" y="116.0" text-anchor="end">who</text><g class="mark"><title>who: 0.194</title><rect class="s2" x="90.0" y="103.0" width="121.9" height="16.0" rx="3"/></g><text class="t-val" x="217.9" y="116.0" text-anchor="start">0.194</text><text class="t-tick" x="82.0" y="140.0" text-anchor="end">is</text><g class="mark"><title>is: 0.069</title><rect class="s2" x="90.0" y="127.0" width="43.2" height="16.0" rx="3"/></g><text class="t-val" x="139.2" y="140.0" text-anchor="start">0.069</text><text class="t-tick" x="82.0" y="164.0" text-anchor="end">in</text><g class="mark"><title>in: 0.060</title><rect class="s2" x="90.0" y="151.0" width="37.5" height="16.0" rx="3"/></g><text class="t-val" x="133.5" y="164.0" text-anchor="start">0.060</text><text class="t-tick" x="82.0" y="188.0" text-anchor="end">was</text><g class="mark"><title>was: 0.055</title><rect class="s2" x="90.0" y="175.0" width="34.6" height="16.0" rx="3"/></g><text class="t-val" x="130.6" y="188.0" text-anchor="start">0.055</text><text class="t-tick" x="82.0" y="212.0" text-anchor="end">'s</text><g class="mark"><title>'s: 0.051</title><rect class="s2" x="90.0" y="199.0" width="31.9" height="16.0" rx="3"/></g><text class="t-val" x="127.9" y="212.0" text-anchor="start">0.051</text><text class="t-tick" x="462.0" y="116.0" text-anchor="end">looked</text><g class="mark"><title>looked: 0.323</title><rect class="s1" x="470.0" y="103.0" width="203.2" height="16.0" rx="3"/></g><text class="t-val" x="679.2" y="116.0" text-anchor="start">0.323</text><text class="t-tick" x="462.0" y="140.0" text-anchor="end">stared</text><g class="mark"><title>stared: 0.103</title><rect class="s1" x="470.0" y="127.0" width="64.7" height="16.0" rx="3"/></g><text class="t-val" x="540.7" y="140.0" text-anchor="start">0.103</text><text class="t-tick" x="462.0" y="164.0" text-anchor="end">glanced</text><g class="mark"><title>glanced: 0.080</title><rect class="s1" x="470.0" y="151.0" width="50.0" height="16.0" rx="3"/></g><text class="t-val" x="526.0" y="164.0" text-anchor="start">0.080</text><text class="t-tick" x="462.0" y="188.0" text-anchor="end">pointed</text><g class="mark"><title>pointed: 0.062</title><rect class="s1" x="470.0" y="175.0" width="39.0" height="16.0" rx="3"/></g><text class="t-val" x="515.0" y="188.0" text-anchor="start">0.062</text><text class="t-tick" x="462.0" y="212.0" text-anchor="end">glared</text><g class="mark"><title>glared: 0.059</title><rect class="s1" x="470.0" y="199.0" width="37.4" height="16.0" rx="3"/></g><text class="t-val" x="513.4" y="212.0" text-anchor="start">0.059</text><text class="t-muted" x="20.0" y="92.0" text-anchor="start">top 5 guesses</text><text class="t-muted" x="400.0" y="92.0" text-anchor="start">top 5 guesses</text><text class="t-tick" x="20.0" y="238.0" text-anchor="start">P(smiles) = 0.0003</text><text class="t-tick" x="400.0" y="238.0" text-anchor="start">P(smiles) = 0.0008</text><text class="t-muted" x="20.0" y="260.0" text-anchor="start">guesses fit "The kid ..." (who, is, was)</text><text class="t-muted" x="400.0" y="260.0" text-anchor="start">every guess fits "___ at the dog" (a verb + at)</text></svg><figcaption>One hidden word, predicted with one side and with both. GPT-2, seeing "The kid", guesses words that can follow any noun (who, is, in). BERT, seeing "the kid ___ at the dog.", guesses only verbs that take "at": looked, stared, glanced. Neither gets "smiles" right, because many verbs fit.</figcaption></figure>

BERT's probability for "smiles" is still small, because "looked at the dog" fits just as well. But look at the *kind* of guesses. With only the left side, GPT-2's guesses are almost generic. With both sides, every one of BERT's top five is a verb that can be followed by "at". The right side did not reveal the answer; it narrowed the space of sensible answers. That narrowing is what makes BERT's vectors useful for understanding tasks.

> [!DEFINITION] Probability
> A number from 0 to 1 that says how likely something is. 0.901 means "about 90% sure". The probabilities over all possible words add up to 1.

What this shows, and what it does not:

- With the right side hidden, the guess is a guess. GPT-2's top answer for the first sentence has a probability of only 0.033.
- With both sides visible, the answer is nearly certain: BERT gives "bank" 0.901 and "guitar" 0.677.
- This is **not** a contest between two models. GPT-2 was simply not given the words on the right, because a left-to-right model never gets them at this position. That missing information is exactly the paper's point.

**Use case.** Question answering is the paper's own example. In "Who founded the company that makes the iPhone?", the word "company" needs the words after it to know which company. A model that builds every word's meaning from both sides can answer such questions far better.

## BERT's fix: fill in the blanks {§1}

If language models are one-directional by nature, how do you pre-train a two-directional model? Here is the paper's answer.

> [!PAPER] · §1
> [![The fourth paragraph of the introduction: BERT alleviates the unidirectionality constraint by using a masked language model pre-training objective, inspired by the Cloze task. The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word](https://ishwarj.com/img/papers/bert/p1-intro-mlm.pngrt/p1-intro-mlm.png)](/img/papers/bert/p1-intro-mlm.png)
> [![The paragraph continues on page 2: based only on its context. The MLM objective enables the representation to fuse the left and the right context. In addition, a next sentence prediction task jointly pre-trains text-pair representations](https://ishwarj.com/img/papers/bert/p1-intro-mlm2.pngt/p1-intro-mlm2.png)](/img/papers/bert/p1-intro-mlm2.png)
>
> **What it says:** BERT uses a **"masked language model" (MLM)**, "inspired by the Cloze task". It "randomly masks some of the tokens from the input", and the model must "predict the original vocabulary id of the masked word based only on its context". This lets the representation "fuse the left and the right context". A second task, **"next sentence prediction"**, teaches the model about pairs of sentences.
>
> **What it means:**
>
> Think of a school worksheet with blanks: "The cat sat on the ____." To fill a blank, you read the words before it and after it. You cannot cheat, because the answer is not on the page.
>
> That is the reason the trick works, though the paragraph does not spell it out. A normal language model must hide the future, or the next word would give away its own answer. With masking, the answer is hidden in place, so the model may look both ways safely.
>
> So the task changes from predicting the *next* word to predicting a *hidden* word, using everything around it. This one change makes deep bidirectional pre-training possible.

> [!DEFINITION] Cloze task
> A fill-in-the-blank test. Wilson Taylor described it in 1953 as a way to measure how readable a text is. "The cat sat on the ____." Readers fill in the missing word from the context.

> [!DEFINITION] Masking
> Hiding a token from the model by replacing it with the special token `[MASK]`. The model must work out what was there.

> [!DEFINITION] Vocabulary id
> Every token the model knows has a number, its **id**, in a fixed list called the **vocabulary**. BERT's vocabulary has about 30,000 tokens. "Predict the original vocabulary id" just means "say which word was hidden".

You already saw masked language modelling in action: the `[MASK]` sentences above are exactly this pre-training task. The trick works because a hidden word cannot leak its own answer, so the model is free to look in both directions. In Part 3 we will see *why* an ordinary language model cannot simply look both ways (each word would "see itself"), and we will run the exact masking recipe, which has a few surprising details.

The paragraph ends with a second task, **next sentence prediction**: given two pieces of text, guess whether the second really came right after the first. Many tasks are about *pairs* of sentences (does this answer this question? does this sentence follow from that one?), and ordinary language modelling never practises that. Part 3 covers it too.

## The three contributions {§1}

> [!PAPER] · §1
> [![The contributions of the paper as three bullet points: the importance of bidirectional pre-training; pre-trained representations reduce the need for heavily-engineered task-specific architectures; BERT advances the state of the art for eleven NLP tasks. The code and models are released on GitHub](https://ishwarj.com/img/papers/bert/p1-contributions.png1-contributions.png)](/img/papers/bert/p1-contributions.png)
>
> **What it says:** the authors list what is new in their paper as three contributions. (1) Bidirectional pre-training matters, and BERT is deeply bidirectional, unlike GPT (one direction) and unlike ELMo's "shallow concatenation of independently trained left-to-right and right-to-left LMs". (2) Pre-trained representations "reduce the need for many heavily-engineered task-specific architectures". (3) BERT "advances the state of the art for eleven NLP tasks", and the code and models are public.
>
> **What it means:**
>
> A list of contributions works like a contract. The authors state their promises here, and the rest of the paper must pay each one with evidence.
>
> The first claim attacks both rivals at once: GPT for its single direction, ELMo for joining its two directions only at the end. The second claim is about cost, because before BERT each task had its own network, designed by hand. The third claim is the proof that the recipe works.
>
> Each claim is tested later. Claim 1 by the ablation studies in Part 5, claims 2 and 3 by the experiments in Part 4.

Let us unpack each contribution.

**1. Deep, not shallow, bidirectionality.** ELMo does look both ways, but in a shallow way: one network reads left to right, another reads right to left, and their outputs are glued together ("concatenated") at the very end. Inside each network, information still flows one way only. In BERT, every layer of one single network mixes both sides. The paper argues this is "strictly more powerful".

> [!DEFINITION] Concatenation
> Joining two lists of numbers end to end. A list of 512 numbers concatenated with another list of 512 numbers gives one list of 1,024 numbers. Nothing is mixed; the two halves just sit side by side.

To see what "shallow" means exactly, look at how ELMo is trained, in its own paper:

> [!PAPER] Peters et al. (2018a), Deep contextualized word representations (ELMo) · §3.1
> [![Section 3.1 of the ELMo paper: a forward language model predicts each token from the tokens before it, a backward language model predicts each token from the tokens after it, and the biLM maximizes the sum of both log likelihoods, with separate parameters for the LSTMs in each direction](https://ishwarj.com/img/papers/bert/p1-elmo-bilm.pngrt/p1-elmo-bilm.png)](/img/papers/bert/p1-elmo-bilm.png)
>
> **What it says:** ELMo's training objective has two parts. A **forward** language model predicts each token from the tokens before it; a **backward** one predicts "the previous token given the future context". ELMo "jointly maximizes the log likelihood of the forward and backward directions", sharing only the input layer and the final softmax, while keeping "separate parameters for the LSTMs in each direction".
>
> **What it means:**
>
> Picture two people who read the same sentence. One reads from the start to the end. The other reads from the end back to the start. They never talk while they read, and at the end their notes are stapled together.
>
> That is what "separate parameters for the LSTMs in each direction" means, and it is the shallow part. Each reader is still one-directional. They meet only when their outputs are glued together at the end, so no layer ever mixes both sides.

ELMo's objective, in the notation we used for GPT:

$$
\mathcal{L}_{\text{ELMo}} = -\sum_{k=1}^{N} \Big( \log p(t_k \mid t_1, \dots, t_{k-1}) + \log p(t_k \mid t_{k+1}, \dots, t_N) \Big)
$$

where $$t_1, \dots, t_N$$ are the tokens, the first term is the forward model (left side only) and the second the backward model (right side only). Each term on its own is an ordinary one-directional language model. No single term ever conditions on both sides at once, which is the thing BERT's masked language model does (Part 3).

<figure class="fig"><svg viewBox="0 0 760 336" role="img" aria-label="Three ways to build the top vector of the word "kid" with two layers. BERT: each layer reads every position of the layer below, so information from "smiles" reaches "kid" inside every layer. GPT: each layer reads only positions on the left. ELMo: a left-to-right tower and a right-to-left tower run separately and their top vectors are only concatenated at the end."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="125.0" y="24.0" text-anchor="middle">BERT</text><text class="t-muted" x="125.0" y="42.0" text-anchor="middle">every layer mixes both sides</text><line class="edge-on" x1="55.0" y1="258.0" x2="55.0" y2="210.0"/><line class="edge-on" x1="125.0" y1="258.0" x2="55.0" y2="210.0"/><line class="edge-on" x1="195.0" y1="258.0" x2="55.0" y2="210.0"/><line class="edge-on" x1="55.0" y1="258.0" x2="125.0" y2="210.0"/><line class="edge-on" x1="125.0" y1="258.0" x2="125.0" y2="210.0"/><line class="edge-on" x1="195.0" y1="258.0" x2="125.0" y2="210.0"/><line class="edge-on" x1="55.0" y1="258.0" x2="195.0" y2="210.0"/><line class="edge-on" x1="125.0" y1="258.0" x2="195.0" y2="210.0"/><line class="edge-on" x1="195.0" y1="258.0" x2="195.0" y2="210.0"/><line class="edge" x1="55.0" y1="182.0" x2="55.0" y2="134.0"/><line class="edge" x1="125.0" y1="182.0" x2="55.0" y2="134.0"/><line class="edge" x1="195.0" y1="182.0" x2="55.0" y2="134.0"/><line class="edge-on" x1="55.0" y1="182.0" x2="125.0" y2="134.0"/><line class="edge-on" x1="125.0" y1="182.0" x2="125.0" y2="134.0"/><line class="edge-on" x1="195.0" y1="182.0" x2="125.0" y2="134.0"/><line class="edge" x1="55.0" y1="182.0" x2="195.0" y2="134.0"/><line class="edge" x1="125.0" y1="182.0" x2="195.0" y2="134.0"/><line class="edge" x1="195.0" y1="182.0" x2="195.0" y2="134.0"/><rect class="box-1" x="29.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="55.0" y="200.5" text-anchor="middle">L1</text><rect class="box-1" x="99.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="125.0" y="200.5" text-anchor="middle">L1</text><rect class="box-1" x="169.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="195.0" y="200.5" text-anchor="middle">L1</text><rect class="box-1" x="29.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="55.0" y="124.5" text-anchor="middle">L2</text><rect class="box-1" x="99.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="125.0" y="124.5" text-anchor="middle">L2</text><rect class="box-1" x="169.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="195.0" y="124.5" text-anchor="middle">L2</text><rect class="box" x="25.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="55.0" y="276.5" text-anchor="middle">the</text><rect class="box-mask" x="95.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="125.0" y="276.5" text-anchor="middle">kid</text><rect class="box" x="165.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="195.0" y="276.5" text-anchor="middle">smiles</text><text class="t-title" x="380.0" y="24.0" text-anchor="middle">OpenAI GPT</text><text class="t-muted" x="380.0" y="42.0" text-anchor="middle">every layer sees only the left</text><line class="edge-on" x1="310.0" y1="258.0" x2="310.0" y2="210.0"/><line class="edge-on" x1="310.0" y1="258.0" x2="380.0" y2="210.0"/><line class="edge-on" x1="380.0" y1="258.0" x2="380.0" y2="210.0"/><line class="edge" x1="310.0" y1="258.0" x2="450.0" y2="210.0"/><line class="edge" x1="380.0" y1="258.0" x2="450.0" y2="210.0"/><line class="edge" x1="450.0" y1="258.0" x2="450.0" y2="210.0"/><line class="edge" x1="310.0" y1="182.0" x2="310.0" y2="134.0"/><line class="edge-on" x1="310.0" y1="182.0" x2="380.0" y2="134.0"/><line class="edge-on" x1="380.0" y1="182.0" x2="380.0" y2="134.0"/><line class="edge" x1="310.0" y1="182.0" x2="450.0" y2="134.0"/><line class="edge" x1="380.0" y1="182.0" x2="450.0" y2="134.0"/><line class="edge" x1="450.0" y1="182.0" x2="450.0" y2="134.0"/><rect class="box-2" x="284.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="310.0" y="200.5" text-anchor="middle">L1</text><rect class="box-2" x="354.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="380.0" y="200.5" text-anchor="middle">L1</text><rect class="box-2" x="424.0" y="182.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="450.0" y="200.5" text-anchor="middle">L1</text><rect class="box-2" x="284.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="310.0" y="124.5" text-anchor="middle">L2</text><rect class="box-2" x="354.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="380.0" y="124.5" text-anchor="middle">L2</text><rect class="box-2" x="424.0" y="106.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="450.0" y="124.5" text-anchor="middle">L2</text><rect class="box" x="280.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="310.0" y="276.5" text-anchor="middle">the</text><rect class="box-mask" x="350.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="380.0" y="276.5" text-anchor="middle">kid</text><rect class="box" x="420.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="450.0" y="276.5" text-anchor="middle">smiles</text><text class="t-title" x="635.0" y="24.0" text-anchor="middle">ELMo</text><text class="t-muted" x="635.0" y="42.0" text-anchor="middle">two separate towers, joined at the top</text><rect class="box-1" x="539.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="551.0" y="200.5" text-anchor="middle">→</text><rect class="box-2" x="567.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="579.0" y="200.5" text-anchor="middle">←</text><rect class="box-1" x="539.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="551.0" y="124.5" text-anchor="middle">→</text><rect class="box-2" x="567.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="579.0" y="124.5" text-anchor="middle">←</text><line class="edge" x1="565.0" y1="258.0" x2="551.0" y2="208.0"/><line class="edge" x1="565.0" y1="258.0" x2="579.0" y2="208.0"/><line class="edge-1" x1="551.0" y1="184.0" x2="551.0" y2="132.0"/><line class="edge-2" x1="579.0" y1="184.0" x2="579.0" y2="132.0"/><path class="edge-1" d="M551.0,184.0 Q586.0,156.0 621.0,183.0" marker-end="url(#ah)"/><path class="edge-1" d="M551.0,108.0 Q586.0,80.0 621.0,107.0" marker-end="url(#ah)"/><rect class="box-1" x="609.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="621.0" y="200.5" text-anchor="middle">→</text><rect class="box-2" x="637.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="649.0" y="200.5" text-anchor="middle">←</text><rect class="box-1" x="609.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="621.0" y="124.5" text-anchor="middle">→</text><rect class="box-2" x="637.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="649.0" y="124.5" text-anchor="middle">←</text><line class="edge" x1="635.0" y1="258.0" x2="621.0" y2="208.0"/><line class="edge" x1="635.0" y1="258.0" x2="649.0" y2="208.0"/><line class="edge-1" x1="621.0" y1="184.0" x2="621.0" y2="132.0"/><line class="edge-2" x1="649.0" y1="184.0" x2="649.0" y2="132.0"/><path class="edge-1" d="M621.0,184.0 Q656.0,156.0 691.0,183.0" marker-end="url(#ah)"/><path class="edge-2" d="M649.0,208.0 Q614.0,236.0 579.0,209.0" marker-end="url(#ah)"/><path class="edge-1" d="M621.0,108.0 Q656.0,80.0 691.0,107.0" marker-end="url(#ah)"/><path class="edge-2" d="M649.0,132.0 Q614.0,160.0 579.0,133.0" marker-end="url(#ah)"/><rect class="box-1" x="679.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="691.0" y="200.5" text-anchor="middle">→</text><rect class="box-2" x="707.0" y="184.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="719.0" y="200.5" text-anchor="middle">←</text><rect class="box-1" x="679.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="691.0" y="124.5" text-anchor="middle">→</text><rect class="box-2" x="707.0" y="108.0" width="24.0" height="24.0" rx="6"/><text class="t-tick" x="719.0" y="124.5" text-anchor="middle">←</text><line class="edge" x1="705.0" y1="258.0" x2="691.0" y2="208.0"/><line class="edge" x1="705.0" y1="258.0" x2="719.0" y2="208.0"/><line class="edge-1" x1="691.0" y1="184.0" x2="691.0" y2="132.0"/><line class="edge-2" x1="719.0" y1="184.0" x2="719.0" y2="132.0"/><path class="edge-2" d="M719.0,208.0 Q684.0,236.0 649.0,209.0" marker-end="url(#ah)"/><path class="edge-2" d="M719.0,132.0 Q684.0,160.0 649.0,133.0" marker-end="url(#ah)"/><rect class="box-on" x="595.0" y="54.0" width="80.0" height="26.0" rx="6"/><text class="t-tick" x="635.0" y="71.5" text-anchor="middle">concat</text><line class="edge-1" x1="621.0" y1="108.0" x2="621.0" y2="78.0"/><line class="edge-2" x1="649.0" y1="108.0" x2="649.0" y2="78.0"/><rect class="box" x="535.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="565.0" y="276.5" text-anchor="middle">the</text><rect class="box-mask" x="605.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="635.0" y="276.5" text-anchor="middle">kid</text><rect class="box" x="675.0" y="258.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="705.0" y="276.5" text-anchor="middle">smiles</text><text class="t-tick" x="380.0" y="322.0" text-anchor="middle">Highlighted lines: what feeds "kid" at the top. Only in BERT does "smiles" reach it inside the layers.</text></svg><figcaption>"In all layers", drawn. Highlighted lines show what feeds the word "kid" at the top. BERT: both neighbours, at every layer, so after two layers "kid" has mixed in everything. GPT: only "the" and itself. ELMo: the left tower sees "the", the right tower sees "smiles", and the two halves meet only in the final concatenation.</figcaption></figure>

> [!DEFINITION] Architecture
> The design of a model: which layers it has, in what order, and how they connect. A "task-specific architecture" is a design built by hand for one task.

**2. Less hand-made machinery.** Before BERT, the best question-answering systems and the best sentiment classifiers were different, carefully designed networks. With BERT, the same model plus one small output layer does both. That saves months of engineering per task.

**3. Eleven state-of-the-art results**, plus public code and weights. The public release is a big reason BERT spread so fast: anyone could download the model and fine-tune it on a single GPU. The model we ran above is that release.

> [!DEFINITION] NLP
> Natural language processing: the field of making computers work with human language (text and speech).

## Who's who: the papers BERT is talking to

The introduction names a handful of earlier works again and again. Here they are in one place, so the rest of the series can refer back to them. Each entry is listed in full in the references at the end of this part.

| Short name | Paper | Year | What it is, in one line | Role in the BERT paper |
|---|---|---|---|---|
| Transformer | Vaswani et al., [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) | 2017 | the attention-only network for translation | BERT is its encoder half |
| ELMo | Peters et al. (2018a), [*Deep contextualized word representations*](https://arxiv.org/abs/1802.05365) | 2018 | two one-way LSTM language models, concatenated | the feature-based rival; "shallow" bidirectional |
| OpenAI GPT | Radford et al., [*Improving Language Understanding by Generative Pre-Training*](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) | 2018 | a left-to-right Transformer, fine-tuned per task | the fine-tuning rival; BERT-base copies its size |
| ULMFiT | Howard and Ruder, [*Universal Language Model Fine-tuning for Text Classification*](https://arxiv.org/abs/1801.06146) | 2018 | an LSTM language model fine-tuned per task | an earlier fine-tuning approach |
| Semi-supervised sequence learning | Dai and Le, [*Semi-supervised Sequence Learning*](https://arxiv.org/abs/1511.01432) | 2015 | pre-train an LSTM, then fine-tune it | one of the first pre-train-then-fine-tune papers |
| word2vec | Mikolov et al., [*Distributed Representations of Words and Phrases and their Compositionality*](https://arxiv.org/abs/1310.4546) | 2013 | one fixed vector per word, learned from its neighbours | the classic pre-trained word embeddings |
| GloVe | Pennington et al., [*GloVe: Global Vectors for Word Representation*](https://aclanthology.org/D14-1162/) | 2014 | word vectors from word co-occurrence counts | the other classic word embeddings |
| Skip-thought | Kiros et al., [*Skip-Thought Vectors*](https://arxiv.org/abs/1506.06726) | 2015 | a sentence vector trained to predict nearby sentences | sentence-level pre-training (Part 2) |
| CoVe | McCann et al., [*Learned in Translation: Contextualized Word Vectors*](https://arxiv.org/abs/1708.00107) | 2017 | word vectors from a translation encoder | transfer from a supervised task (Part 2) |
| Cloze | Taylor, [*"Cloze Procedure": A New Tool for Measuring Readability*](https://doi.org/10.1177/107769905303000401) | 1953 | the fill-in-the-blank reading test | the inspiration for the masked LM |

GPT-2 (Radford et al., 2019, [*Language Models are Unsupervised Multitask Learners*](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)), which we ran above, came out a few months after BERT and is not cited in the paper. We use it only because it is a freely available left-to-right model of similar size.

## Figure 3: the three designs side by side {§A.4}

The paper puts its clearest picture of this argument in the appendix (Appendix A.4), so we read it now, where it belongs in the story.

> [!PAPER] · Appendix A.4, Figure 3
> [![Figure 3 of the BERT paper: three diagrams. BERT, a stack of Transformer blocks where every block connects to every position below. OpenAI GPT, a stack of Transformer blocks where each block connects only to positions on its left. ELMo, a left-to-right LSTM and a right-to-left LSTM whose outputs are combined. The caption says only BERT representations are jointly conditioned on both left and right context in all layers](https://ishwarj.com/img/papers/bert/p1-figure3.pngbert/p1-figure3.png)](/img/papers/bert/p1-figure3.png)
>
> **What it says:** BERT uses a bidirectional Transformer; GPT a left-to-right Transformer; ELMo two separately trained LSTMs, one per direction. "Among the three, only BERT representations are jointly conditioned on both left and right context in all layers." Also, BERT and GPT are fine-tuning approaches, while ELMo is feature-based.
>
> **What it means:**
>
> The whole paper is in this one picture, and the arrows tell the story. An arrow means "this output can use that input".
>
> In the BERT panel, every block connects to *every* position in the layer below, so each output knows both sides at every layer. In the GPT panel, each block connects only to positions on its own left. In the ELMo panel, the arrows inside each LSTM chain all point one way; the two directions only meet at the top.
>
> BERT and GPT also have the **same** shape: a stack of Transformer blocks. This is deliberate. The authors made BERT's smaller version the same size as GPT, so a gain of BERT comes from the arrows, not from size.

The symbols in the figure:

- **E₁, E₂, …, E_N** (yellow, bottom) are the input embeddings: one vector per input token.
- **Trm** is one Transformer block (a layer). **Lstm** is one step of an LSTM, an older kind of network that reads one token at a time.
- **T₁, T₂, …, T_N** (green, top) are the output vectors, one per token.

> [!DEFINITION] LSTM
> Long short-term memory: a kind of recurrent neural network, the standard way to read text before Transformers. It reads tokens one at a time, carrying a running memory from each token to the next.

Part 2 opens up that stack of blocks.

> [!TAKEAWAYS] Key takeaways
> - **BERT = Bidirectional Encoder Representations from Transformers**: a Transformer encoder, pre-trained on unlabeled text, that builds every word's representation from both its left and its right, in every layer.
> - **Two steps:** pre-train once (expensive), then fine-tune a copy for each task with one small extra layer (cheap).
> - In 2018 there were **two ways** to reuse a pre-trained model: **feature-based** (ELMo: frozen features fed to a task model) and **fine-tuning** (GPT: train everything a little). BERT fine-tunes.
> - The problem: standard **language models are unidirectional**, so each word learns only from its left. For tasks like question answering, the clue is often on the right.
> - We checked it on real models: with the right side hidden, GPT-2's best guess for "I went to the ____" had probability 0.033; BERT, seeing "to deposit my paycheck", put **0.901** on "bank".
> - BERT's fix is the **masked language model**: hide some tokens and predict them from both sides (a Cloze task), plus **next sentence prediction** for sentence pairs.
> - Concretely, BERT and GPT differ only in the **attention mask** $$M$$: GPT blocks every key to the right of the query ($$-\infty$$ in the upper triangle); BERT blocks nothing. We recomputed a real GPT-2 head by hand and matched the library exactly.
> - In **fine-tuning, all of BERT's weights keep training**, not just the new top layer. Freezing BERT is the separate feature-based approach (Part 5).
> - The results: new state of the art on **eleven tasks**, including GLUE **80.5** (+7.7) and SQuAD v1.1 Test F1 **93.2**.

**Next, in Part 2:** the related work the paper builds on, and the model itself: its layers, its size (and where "110 million parameters" comes from), and how text becomes the numbers BERT reads.

<details>
<summary>Run it yourself</summary>

The script behind the GPT-2 and BERT comparison is [`code/papers/bert/bert_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1.py). It runs on a CPU in under a minute and downloads `bert-base-uncased` (about 440 MB) and GPT-2 (about 550 MB) the first time.

```bash
pip install torch transformers
python bert_part1.py      # prints the table above, writes results/part1.json
```

<figure><img src="/img/papers/bert/part1-run.png" alt="Terminal output of bert_part1.py: for each of three sentences, GPT-2's top five next-word guesses from the left side only, and BERT's top five guesses for the masked word using both sides" loading="lazy" /><figcaption>The real output of bert_part1.py.</figcaption></figure>

</details>

## References

**The BERT paper**

1. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805). NAACL 2019 ([ACL Anthology](https://aclanthology.org/N19-1423/)). arXiv:1810.04805, version 2 (May 2019), which is the version shown in the screenshots.
2. Google Research. [BERT code and pre-trained models](https://github.com/google-research/bert). GitHub, 2018.

**Papers the BERT paper cites in this part**

3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin. [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762). NeurIPS 2017.
4. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer. [*Deep contextualized word representations*](https://arxiv.org/abs/1802.05365) (ELMo; "Peters et al., 2018a" in the paper). NAACL 2018.
5. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever. [*Improving Language Understanding by Generative Pre-Training*](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) (OpenAI GPT). OpenAI technical report, 2018. The BERT paper's reference list gives it the title "Improving language understanding with unsupervised learning".
6. J. Howard, S. Ruder. [*Universal Language Model Fine-tuning for Text Classification*](https://arxiv.org/abs/1801.06146) (ULMFiT). ACL 2018.
7. A. M. Dai, Q. V. Le. [*Semi-supervised Sequence Learning*](https://arxiv.org/abs/1511.01432). NeurIPS 2015.
8. W. L. Taylor. [*"Cloze Procedure": A New Tool for Measuring Readability*](https://doi.org/10.1177/107769905303000401). Journalism Quarterly 30(4), 1953 (cited as "Journalism Bulletin" in the BERT paper).
9. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean. [*Distributed Representations of Words and Phrases and their Compositionality*](https://arxiv.org/abs/1310.4546) (word2vec). NeurIPS 2013.
10. J. Pennington, R. Socher, C. D. Manning. [*GloVe: Global Vectors for Word Representation*](https://aclanthology.org/D14-1162/). EMNLP 2014.
11. R. Kiros, Y. Zhu, R. Salakhutdinov, R. Zemel, A. Torralba, R. Urtasun, S. Fidler. [*Skip-Thought Vectors*](https://arxiv.org/abs/1506.06726). NeurIPS 2015.
12. B. McCann, J. Bradbury, C. Xiong, R. Socher. [*Learned in Translation: Contextualized Word Vectors*](https://arxiv.org/abs/1708.00107) (CoVe). NeurIPS 2017.
13. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei. [*ImageNet: A Large-Scale Hierarchical Image Database*](https://doi.org/10.1109/CVPR.2009.5206848). CVPR 2009.

**Other sources used in this part**

14. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever. [*Language Models are Unsupervised Multitask Learners*](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) (GPT-2). OpenAI, 2019. The `openai-community/gpt2` model used in the code.
15. P. Nayak. [*Understanding searches better than ever before*](https://blog.google/products/search/search-language-understanding-bert/). Google blog, 25 October 2019.
16. Code for this part: [`bert_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1.py) and [`bert_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part1_math.py).

