# Inside BERT: Architecture and Input

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/bert/part-2-architecture-and-input/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/bert/part-2-architecture-and-input.md


In Part 1 we read the paper's big idea: pre-train a Transformer that looks at both sides of every word, then fine-tune it. This part opens the box. First we read Section 2, the history BERT builds on. Then the first half of Section 3: what the model is made of, how big it is, and how text is turned into the numbers it reads.

Everything that can be checked, we check on the real released model, `bert-base-uncased`.

## Related work: a long history of pre-training {§2}

> [!PAPER] · §2
> [![The opening of Section 2, Related Work: there is a long history of pre-training general language representations, and we briefly review the most widely-used approaches in this section](https://ishwarj.com/img/papers/bert/p2-related-intro.png2-related-intro.png)](/img/papers/bert/p2-related-intro.png)
>
> **What it says:** "There is a long history of pre-training general language representations." The section reviews the most used approaches, in three groups.
>
> **What it means:**
>
> No invention starts from zero. A new car still has wheels, seats and an engine, and often only one part is really new. So before the authors describe BERT, they place it in the history of the field.
>
> BERT is not one sudden invention. It combines ideas that already worked: learned word vectors, contextual vectors, fine-tuning, and transfer learning. Knowing them makes clear what is actually new.

The three groups are: **feature-based** approaches (§2.1), **fine-tuning** approaches (§2.2), and **transfer from labelled data** (§2.3). We take them one at a time.

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="Section 2 of the BERT paper as a map. Three lanes over the years 2008 to 2018: feature-based approaches (word embeddings, skip-thought, ELMo), fine-tuning approaches (Collobert and Weston, Dai and Le, ULMFiT, OpenAI GPT) and transfer from labelled data (ImageNet, InferSent, CoVe). All three lead to BERT."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Related work (Section 2): three families of pre-training, all feeding into BERT</text><line class="grid" x1="225.0" y1="56.0" x2="225.0" y2="274.0"/><text class="t-tick" x="225.0" y="50.0" text-anchor="middle">2008</text><line class="grid" x1="304.0" y1="56.0" x2="304.0" y2="274.0"/><text class="t-tick" x="304.0" y="50.0" text-anchor="middle">2010</text><line class="grid" x1="383.1" y1="56.0" x2="383.1" y2="274.0"/><text class="t-tick" x="383.1" y="50.0" text-anchor="middle">2012</text><line class="grid" x1="462.1" y1="56.0" x2="462.1" y2="274.0"/><text class="t-tick" x="462.1" y="50.0" text-anchor="middle">2014</text><line class="grid" x1="541.2" y1="56.0" x2="541.2" y2="274.0"/><text class="t-tick" x="541.2" y="50.0" text-anchor="middle">2016</text><line class="grid" x1="620.2" y1="56.0" x2="620.2" y2="274.0"/><text class="t-tick" x="620.2" y="50.0" text-anchor="middle">2018</text><text class="t-note" x="20.0" y="89.0" text-anchor="start">§2.1 feature-based</text><line class="edge-dim" x1="219.0" y1="84.0" x2="640.0" y2="84.0"/><circle class="node on" cx="304.0" cy="84" r="6"/><text class="t-tick" x="304.0" y="72.0" text-anchor="middle">Turian</text><circle class="node on" cx="422.6" cy="84" r="6"/><text class="t-tick" x="422.6" y="106.0" text-anchor="middle">word2vec</text><circle class="node on" cx="462.1" cy="84" r="6"/><text class="t-tick" x="462.1" y="72.0" text-anchor="middle">GloVe</text><circle class="node on" cx="501.7" cy="84" r="6"/><text class="t-tick" x="501.7" y="106.0" text-anchor="middle">skip-thought</text><circle class="node on" cx="624.2" cy="84" r="6"/><text class="t-tick" x="624.2" y="72.0" text-anchor="middle">ELMo</text><line class="edge" x1="642.0" y1="84.0" x2="668.0" y2="102.0" marker-end="url(#ah)"/><text class="t-note" x="20.0" y="163.0" text-anchor="start">§2.2 fine-tuning</text><line class="edge-dim" x1="219.0" y1="158.0" x2="640.0" y2="158.0"/><circle class="node on" cx="225.0" cy="158" r="6"/><text class="t-tick" x="225.0" y="146.0" text-anchor="middle">Collobert &amp; Weston</text><circle class="node on" cx="511.5" cy="158" r="6"/><text class="t-tick" x="511.5" y="180.0" text-anchor="middle">Dai &amp; Le</text><circle class="node on" cx="620.2" cy="158" r="6"/><text class="t-tick" x="620.2" y="146.0" text-anchor="middle">ULMFiT</text><circle class="node on" cx="638.0" cy="158" r="6"/><text class="t-tick" x="638.0" y="180.0" text-anchor="middle">OpenAI GPT</text><line class="edge" x1="642.0" y1="158.0" x2="668.0" y2="124.0" marker-end="url(#ah)"/><text class="t-note" x="20.0" y="237.0" text-anchor="start">§2.3 labelled transfer</text><line class="edge-dim" x1="219.0" y1="232.0" x2="640.0" y2="232.0"/><circle class="node on" cx="264.5" cy="232" r="6"/><text class="t-tick" x="264.5" y="220.0" text-anchor="middle">ImageNet</text><circle class="node on" cx="462.1" cy="232" r="6"/><text class="t-tick" x="462.1" y="254.0" text-anchor="middle">Yosinski</text><circle class="node on" cx="582.7" cy="232" r="6"/><text class="t-tick" x="582.7" y="220.0" text-anchor="middle">InferSent</text><circle class="node on" cx="604.4" cy="232" r="6"/><text class="t-tick" x="604.4" y="254.0" text-anchor="middle">CoVe</text><line class="edge" x1="642.0" y1="232.0" x2="668.0" y2="146.0" marker-end="url(#ah)"/><rect class="box-on" x="670.0" y="96.0" width="80.0" height="56.0" rx="12"/><text class="t-big" x="710.0" y="121.0" text-anchor="middle">BERT</text><text class="t-tick" x="710.0" y="139.0" text-anchor="middle">2018</text><text class="t-muted" x="20.0" y="300.0" text-anchor="start">Feature-based: frozen vectors feed a separate task model. Fine-tuning: the whole pre-trained network keeps training.</text><text class="t-muted" x="20.0" y="318.0" text-anchor="start">Labelled transfer: pre-train on a big labelled task. BERT takes the fine-tuning recipe and the ImageNet habit, with unlabelled text.</text></svg><figcaption>The related work on one timeline. Feature-based methods (word2vec, GloVe, skip-thought, ELMo) produce frozen vectors for another model. Fine-tuning methods (Collobert and Weston, Dai and Le, ULMFiT, OpenAI GPT) keep training the whole pre-trained network. Labelled transfer (ImageNet, InferSent, CoVe) pre-trains on a big labelled task. BERT takes the fine-tuning recipe and the ImageNet habit, with unlabeled text.</figcaption></figure>

### Word embeddings {§2.1}

> [!PAPER] · §2.1
> [![Section 2.1, Unsupervised Feature-based Approaches: learning widely applicable representations of words has been an active area of research for decades. Pre-trained word embeddings are an integral part of modern NLP systems, offering significant improvements over embeddings learned from scratch](https://ishwarj.com/img/papers/bert/p2-feature-words.png2-feature-words.png)](/img/papers/bert/p2-feature-words.png)
>
> **What it says:** people have learned vectors for words for decades, first without neural networks and then with them (Mikolov et al., 2013, known as word2vec; Pennington et al., 2014, known as GloVe). "Pre-trained word embeddings" give "significant improvements over embeddings learned from scratch".
>
> **What it means:**
>
> Think of a dictionary in which every word has a fixed entry. Someone wrote the entry once, after a lot of reading, and every later reader can look it up. A pre-trained word embedding is such an entry, written as a list of numbers.
>
> This is the oldest form of pre-training: learn one vector per word from a lot of text, then reuse those vectors in other models. "Significant improvements over embeddings learned from scratch" means that the reuse pays. A small task no longer has to learn what words mean from its own few examples.

> [!DEFINITION] Word embedding
> A fixed list of numbers for each word in a vocabulary, learned from a large amount of text so that words used in similar ways get similar lists. word2vec (2013) and GloVe (2014) are the two most famous methods.

> [!DEFINITION] Learned from scratch
> Starting from random numbers and learning only from the task's own (usually small) labelled data. The opposite of starting from pre-trained numbers.

A word embedding is a **lookup table**: one row per word. The word "bank" always gets the same row, whether it is a river bank or a money bank. Keep this weakness in mind; the next two paragraphs are about fixing it.

<figure class="fig"><svg viewBox="0 0 760 292" role="img" aria-label="Three ways to represent a word. Brown clusters, a non-neural method, give each word a path in a binary tree of word groups. word2vec and GloVe give each word one learned vector, the same in every sentence. ELMo and BERT compute a new vector for each sentence, so river bank and savings bank differ. The colours are an illustration, not real values."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Three ways to turn the word "bank" into numbers</text><rect class="box" x="20.0" y="44.0" width="230.0" height="236.0" rx="12"/><text class="t-note" x="135.0" y="68.0" text-anchor="middle">1. Brown clusters (1992)</text><text class="t-muted" x="135.0" y="86.0" text-anchor="middle">non-neural: count word neighbours</text><line class="edge" x1="135.0" y1="110.0" x2="85.0" y2="148.0"/><text class="t-muted" x="102.0" y="129.0" text-anchor="middle">0</text><line class="edge-on" x1="135.0" y1="110.0" x2="185.0" y2="148.0"/><text class="t-muted" x="168.0" y="129.0" text-anchor="middle">1</text><line class="edge" x1="85.0" y1="148.0" x2="60.0" y2="186.0"/><text class="t-muted" x="64.5" y="167.0" text-anchor="middle">0</text><line class="edge" x1="85.0" y1="148.0" x2="110.0" y2="186.0"/><text class="t-muted" x="105.5" y="167.0" text-anchor="middle">1</text><line class="edge-on" x1="185.0" y1="148.0" x2="160.0" y2="186.0"/><text class="t-muted" x="164.5" y="167.0" text-anchor="middle">0</text><line class="edge" x1="185.0" y1="148.0" x2="210.0" y2="186.0"/><text class="t-muted" x="205.5" y="167.0" text-anchor="middle">1</text><line class="edge" x1="60.0" y1="186.0" x2="47.5" y2="224.0"/><text class="t-muted" x="45.8" y="205.0" text-anchor="middle">0</text><line class="edge" x1="60.0" y1="186.0" x2="72.5" y2="224.0"/><text class="t-muted" x="74.2" y="205.0" text-anchor="middle">1</text><line class="edge" x1="110.0" y1="186.0" x2="97.5" y2="224.0"/><text class="t-muted" x="95.8" y="205.0" text-anchor="middle">0</text><line class="edge" x1="110.0" y1="186.0" x2="122.5" y2="224.0"/><text class="t-muted" x="124.2" y="205.0" text-anchor="middle">1</text><line class="edge" x1="160.0" y1="186.0" x2="147.5" y2="224.0"/><text class="t-muted" x="145.8" y="205.0" text-anchor="middle">0</text><line class="edge-on" x1="160.0" y1="186.0" x2="172.5" y2="224.0"/><text class="t-muted" x="174.2" y="205.0" text-anchor="middle">1</text><line class="edge" x1="210.0" y1="186.0" x2="197.5" y2="224.0"/><text class="t-muted" x="195.8" y="205.0" text-anchor="middle">0</text><line class="edge" x1="210.0" y1="186.0" x2="222.5" y2="224.0"/><text class="t-muted" x="224.2" y="205.0" text-anchor="middle">1</text><circle class="node on" cx="135.0" cy="110.0" r="7"/><circle class="node" cx="85.0" cy="148.0" r="7"/><circle class="node on" cx="185.0" cy="148.0" r="7"/><circle class="node" cx="60.0" cy="186.0" r="7"/><circle class="node" cx="110.0" cy="186.0" r="7"/><circle class="node on" cx="160.0" cy="186.0" r="7"/><circle class="node" cx="210.0" cy="186.0" r="7"/><circle class="node" cx="47.5" cy="224.0" r="7"/><circle class="node" cx="72.5" cy="224.0" r="7"/><circle class="node" cx="97.5" cy="224.0" r="7"/><circle class="node" cx="122.5" cy="224.0" r="7"/><circle class="node" cx="147.5" cy="224.0" r="7"/><circle class="node on" cx="172.5" cy="224.0" r="7"/><circle class="node" cx="197.5" cy="224.0" r="7"/><circle class="node" cx="222.5" cy="224.0" r="7"/><text class="t-mono" x="135.0" y="262.0" text-anchor="middle">"bank" = cluster 101</text><text class="t-muted" x="135.0" y="276.0" text-anchor="middle">(with "shore", "branch", ...)</text><rect class="box" x="266.0" y="44.0" width="230.0" height="236.0" rx="12"/><text class="t-note" x="381.0" y="68.0" text-anchor="middle">2. word2vec, GloVe (2013, 2014)</text><text class="t-muted" x="381.0" y="86.0" text-anchor="middle">neural: one learned vector per word</text><text class="t-tick" x="381.0" y="122.0" text-anchor="middle">"river bank"</text><rect class="box-ghost" x="303.5" y="129.0" width="155.0" height="22.0" rx="4"/><g class="mark"><rect class="s1" x="306.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s1" x="325.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s1" x="344.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.35"/></g><g class="mark"><rect class="s1" x="363.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="382.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="401.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s1" x="420.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.27"/></g><g class="mark"><rect class="s1" x="439.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.61"/></g><text class="t-tick" x="381.0" y="192.0" text-anchor="middle">"savings bank"</text><rect class="box-ghost" x="303.5" y="199.0" width="155.0" height="22.0" rx="4"/><g class="mark"><rect class="s1" x="306.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s1" x="325.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s1" x="344.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.35"/></g><g class="mark"><rect class="s1" x="363.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="382.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="401.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s1" x="420.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.27"/></g><g class="mark"><rect class="s1" x="439.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.61"/></g><text class="t-tick" x="381.0" y="262.0" text-anchor="middle">the same row every time</text><text class="t-muted" x="381.0" y="276.0" text-anchor="middle">(a static embedding)</text><rect class="box" x="512.0" y="44.0" width="230.0" height="236.0" rx="12"/><text class="t-note" x="627.0" y="68.0" text-anchor="middle">3. ELMo, BERT (2018)</text><text class="t-muted" x="627.0" y="86.0" text-anchor="middle">contextual: computed per sentence</text><text class="t-tick" x="627.0" y="122.0" text-anchor="middle">"river bank"</text><rect class="box-ghost" x="549.5" y="129.0" width="155.0" height="22.0" rx="4"/><g class="mark"><rect class="s3" x="552.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s3" x="571.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.35"/></g><g class="mark"><rect class="s3" x="590.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s3" x="609.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="628.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s3" x="647.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.27"/></g><g class="mark"><rect class="s3" x="666.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s3" x="685.5" y="132.0" width="16" height="16" rx="2" style="fill-opacity:0.52"/></g><text class="t-tick" x="627.0" y="192.0" text-anchor="middle">"savings bank"</text><rect class="box-ghost" x="549.5" y="199.0" width="155.0" height="22.0" rx="4"/><g class="mark"><rect class="s2" x="552.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s2" x="571.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s2" x="590.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.69"/></g><g class="mark"><rect class="s2" x="609.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.35"/></g><g class="mark"><rect class="s2" x="628.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s2" x="647.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="666.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s2" x="685.5" y="202.0" width="16" height="16" rx="2" style="fill-opacity:0.69"/></g><text class="t-tick" x="627.0" y="262.0" text-anchor="middle">a different vector per sentence</text><text class="t-muted" x="627.0" y="276.0" text-anchor="middle">(real BERT: river vs money cosine 0.43 to 0.48)</text></svg><figcaption>Three generations of word representations. Non-neural: Brown clusters (1992) give each word a code from a tree of word groups built from counts. Neural, static: word2vec and GloVe learn one vector per word. Contextual: ELMo and BERT compute a new vector for every sentence.</figcaption></figure>

The paragraph says word vectors were trained with left-to-right language models and with objectives that "discriminate correct from incorrect words in left and right context". The second phrase is word2vec. Here is its picture and its training objective, from its own papers:

> [!PAPER] Mikolov et al. (2013a), Efficient Estimation of Word Representations in Vector Space · §3.2, Figure 1
> [![Figure 1 of the skip-gram paper: the input word w(t) is projected to a vector, which is used to predict the surrounding words w(t-2), w(t-1), w(t+1), w(t+2); the training objective is to learn word vectors that are good at predicting the nearby words](https://ishwarj.com/img/papers/bert/p2-ext-skipgram-fig1.pngt-skipgram-fig1.png)](/img/papers/bert/p2-ext-skipgram-fig1.png)
>
> **What it says:** in the skip-gram model, one of the two word2vec models, one word goes in, becomes a vector, and that vector is used to predict the words around it, two on each side here. The training objective is "to learn word vector representations that are good at predicting the nearby words".
>
> **What it means:**
>
> You can guess a lot about a person from the friends they keep. Skip-gram applies the same idea to words: a word's vector must be good at predicting the words that stand near it.
>
> This is "left and right context" in 2013: a small window on both sides of the word. The window is the context; there are no layers that mix it. So each word ends up with one vector, the same in every sentence.

> [!PAPER] Mikolov et al. (2013b), Distributed Representations of Words and Phrases · §2.2, Equation (4)
> [![Equation 4 of the word2vec negative sampling paper: the objective log sigma of v prime w O dot v w I plus the sum over k samples from a noise distribution of log sigma of minus v prime w i dot v w I, called negative sampling](https://ishwarj.com/img/papers/bert/p2-ext-skipgram-neg.pngxt-skipgram-neg.png)](/img/papers/bert/p2-ext-skipgram-neg.png)
>
> **What it says:** this is the trick that made skip-gram fast. instead of a softmax over the whole vocabulary, the model learns to tell the real neighbour $$w_O$$ apart from $$k$$ random "noise" words. They "define Negative sampling (NEG)" by this objective.
>
> **What it means:**
>
> Think of a quiz that asks "which word really stood next to *sat*?" Checking every word in the dictionary is slow. So the quiz shows one real neighbour and a few random words, and the model only has to tell them apart.
>
> This is exactly "discriminate correct from incorrect words in left and right context", the phrase in BERT's related work. It is also the paper that BERT cites as Mikolov et al. (2013). The objective below scores that yes-or-no choice.

The objective, symbol by symbol:

$$
\log \sigma\!\left(v'^{\top}_{w_O} v_{w_I}\right) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)}\!\left[\log \sigma\!\left(-v'^{\top}_{w_i} v_{w_I}\right)\right]
$$

where:

- $$w_I$$ is the input (centre) word and $$w_O$$ a real word from its window; $$v_{w_I}$$ and $$v'_{w_O}$$ are their vectors (word2vec keeps two vectors per word, one for each role);
- $$\sigma(x) = 1 / (1 + e^{-x})$$ is the sigmoid, which squeezes any number into a probability between 0 and 1;
- the first term is large when the real pair's dot product is large ("these two go together");
- $$w_1, \dots, w_k$$ are $$k$$ random words drawn from a noise distribution $$P_n(w)$$ (common words drawn more often); the second term is large when their dot products with the centre word are very negative ("these do not go together");
- $$\mathbb{E}$$ means "on average over the random draws". Training makes the whole expression as large as possible.

<figure class="fig"><svg viewBox="0 0 760 326" role="img" aria-label="word2vec skip-gram. The centre word sat is trained to predict the two words on its left and the two on its right. With negative sampling, the model learns to give a high score to real neighbour pairs (sat with cat, sat with on) and a low score to random pairs (sat with banana)."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">word2vec (skip-gram): learn a vector by predicting the words around it</text><rect class="box-1" x="150.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="69.5" text-anchor="middle">the</text><rect class="box-1" x="230.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="265.0" y="69.5" text-anchor="middle">cat</text><rect class="box-on" x="310.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="345.0" y="69.5" text-anchor="middle">sat</text><rect class="box-1" x="390.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="425.0" y="69.5" text-anchor="middle">on</text><rect class="box-1" x="470.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="505.0" y="69.5" text-anchor="middle">the</text><rect class="box" x="550.0" y="50.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="585.0" y="69.5" text-anchor="middle">mat</text><text class="t-muted" x="140.0" y="70.0" text-anchor="end">sentence</text><path class="edge-1" d="M152.0,84.0 L152.0,92.0 L298.0,92.0 L298.0,84.0"/><text class="t-s1" x="225.0" y="110.0" text-anchor="middle">left context (2 words)</text><path class="edge-1" d="M392.0,84.0 L392.0,92.0 L538.0,92.0 L538.0,84.0"/><text class="t-s1" x="465.0" y="110.0" text-anchor="middle">right context (2 words)</text><text class="t-on" x="345.0" y="112.0" text-anchor="middle">centre word</text><text class="t-muted" x="585.0" y="112.0" text-anchor="middle">outside window</text><text class="t-note" x="20.0" y="150.0" text-anchor="start">Training signal: tell real neighbours from random words ("discriminate correct from incorrect words")</text><rect class="box-on" x="150.0" y="168.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="187.5" text-anchor="middle">sat</text><text class="t-muted" x="232.0" y="188.0" text-anchor="middle">+</text><rect class="box-3" x="244.0" y="168.0" width="84.0" height="30.0" rx="6"/><text class="t-tick" x="286.0" y="187.5" text-anchor="middle">cat</text><line class="edge" x1="334.0" y1="183.0" x2="380.0" y2="183.0" marker-end="url(#ah)"/><rect class="box" x="384.0" y="168.0" width="170.0" height="30.0" rx="6"/><text class="t-tick" x="469.0" y="188.0" text-anchor="middle">σ( v′cat · vsat )</text><line class="edge" x1="558.0" y1="183.0" x2="590.0" y2="183.0" marker-end="url(#ah)"/><text class="t-tick" x="600.0" y="188.0" text-anchor="start">target 1: real neighbour</text><rect class="box-on" x="150.0" y="206.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="225.5" text-anchor="middle">sat</text><text class="t-muted" x="232.0" y="226.0" text-anchor="middle">+</text><rect class="box-3" x="244.0" y="206.0" width="84.0" height="30.0" rx="6"/><text class="t-tick" x="286.0" y="225.5" text-anchor="middle">on</text><line class="edge" x1="334.0" y1="221.0" x2="380.0" y2="221.0" marker-end="url(#ah)"/><rect class="box" x="384.0" y="206.0" width="170.0" height="30.0" rx="6"/><text class="t-tick" x="469.0" y="226.0" text-anchor="middle">σ( v′on · vsat )</text><line class="edge" x1="558.0" y1="221.0" x2="590.0" y2="221.0" marker-end="url(#ah)"/><text class="t-tick" x="600.0" y="226.0" text-anchor="start">target 1: real neighbour</text><rect class="box-on" x="150.0" y="244.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="263.5" text-anchor="middle">sat</text><text class="t-muted" x="232.0" y="264.0" text-anchor="middle">+</text><rect class="box-2" x="244.0" y="244.0" width="84.0" height="30.0" rx="6"/><text class="t-tick" x="286.0" y="263.5" text-anchor="middle">banana</text><line class="edge" x1="334.0" y1="259.0" x2="380.0" y2="259.0" marker-end="url(#ah)"/><rect class="box" x="384.0" y="244.0" width="170.0" height="30.0" rx="6"/><text class="t-tick" x="469.0" y="264.0" text-anchor="middle">σ( v′banana · vsat )</text><line class="edge" x1="558.0" y1="259.0" x2="590.0" y2="259.0" marker-end="url(#ah)"/><text class="t-tick" x="600.0" y="264.0" text-anchor="start">target 0: random word</text><rect class="box-on" x="150.0" y="282.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="301.5" text-anchor="middle">sat</text><text class="t-muted" x="232.0" y="302.0" text-anchor="middle">+</text><rect class="box-2" x="244.0" y="282.0" width="84.0" height="30.0" rx="6"/><text class="t-tick" x="286.0" y="301.5" text-anchor="middle">quickly</text><line class="edge" x1="334.0" y1="297.0" x2="380.0" y2="297.0" marker-end="url(#ah)"/><rect class="box" x="384.0" y="282.0" width="170.0" height="30.0" rx="6"/><text class="t-tick" x="469.0" y="302.0" text-anchor="middle">σ( v′quickly · vsat )</text><line class="edge" x1="558.0" y1="297.0" x2="590.0" y2="297.0" marker-end="url(#ah)"/><text class="t-tick" x="600.0" y="302.0" text-anchor="start">target 0: random word</text></svg><figcaption>The skip-gram setup on "the cat sat on the mat". The centre word "sat" has a window of two words on each side. Training pairs it with its real neighbours (target 1) and with random words such as "banana" (target 0), and learns vectors so that sigmoid of the dot product tells them apart.</figcaption></figure>

> [!DEFINITION] Sigmoid
> The function $$\sigma(x) = 1/(1 + e^{-x})$$. It turns any number into a value between 0 and 1: large positive numbers give almost 1, large negative ones almost 0, and 0 gives 0.5. Used whenever a model must output a single "yes" probability.

### Sentence and paragraph embeddings {§2.1}

> [!PAPER] · §2.1
> [![These approaches have been generalized to coarser granularities, such as sentence embeddings or paragraph embeddings. Prior work trained sentence representations by ranking candidate next sentences, generating the next sentence, or with denoising auto-encoder objectives](https://ishwarj.com/img/papers/bert/p2-feature-sentences.pngature-sentences.png)](/img/papers/bert/p2-feature-sentences.png)
>
> **What it says:** the same idea was "generalized to coarser granularities": one vector for a whole **sentence** or **paragraph**. To learn them, earlier work ranked candidate next sentences, generated the next sentence's words, or used "denoising auto-encoder" objectives.
>
> **What it means:**
>
> Word vectors give one entry per word. The next step was one entry per sentence or paragraph, so that one whole thought could be compared with another. "Coarser granularities" simply means bigger pieces of text.
>
> Two of these ideas come back in BERT. Guessing **which sentence comes next** returns as next sentence prediction (Part 3), and **repairing a damaged input** is close to masked language modelling.

> [!DEFINITION] Denoising auto-encoder
> A model that is given a damaged input (some words deleted or swapped) and is trained to rebuild the clean original. BERT's masked language model is similar, but it only has to rebuild the hidden words, not the whole input.

The paragraph packs three earlier sentence objectives into one sentence. Each one is a small idea worth seeing, because BERT borrows from two of them.

**1. Generate the neighbouring sentences (skip-thought).** Kiros et al. (2015) encode a sentence into one vector, then train two decoders to write the sentence before it and the sentence after it.

> [!PAPER] Kiros et al. (2015), Skip-Thought Vectors · §2, Figure 1
> [![Figure 1 of the skip-thought paper: given a triplet of contiguous sentences, the middle sentence I could see the cat on the steps is encoded and the model tries to reconstruct the previous sentence I got back home and the next sentence This was strange](https://ishwarj.com/img/papers/bert/p2-ext-skipthought-fig1.pngkipthought-fig1.png)](/img/papers/bert/p2-ext-skipthought-fig1.png)
>
> **What it says:** this is the model that BERT describes as "left-to-right generation of next sentence words given a representation of the previous sentence". given three sentences in a row, "the sentence $$s_i$$ is encoded and tries to reconstruct the previous sentence $$s_{i-1}$$ and next sentence $$s_{i+1}$$".
>
> **What it means:**
>
> Think of a friend who tells you the middle of a story. If you understand that part, you can guess what happened just before and just after it. Skip-thought trains a sentence vector to do that guessing.
>
> The idea is that a sentence's meaning shows in what comes around it. BERT keeps this idea but turns it into a much cheaper yes/no question (next sentence prediction, Part 3). Writing out a whole sentence word by word is slow; one yes or no is fast.

**2. Rank candidate next sentences.** Jernite et al. (2017) and Logeswaran and Lee (2018) skip the writing. They give the model a sentence and a few candidates, and train it to pick the one that really came next.

> [!PAPER] Logeswaran and Lee (2018), An efficient framework for learning sentence representations · §3, Figure 1
> [![Figure 1 of the quick-thought paper: the conventional approach encodes Spring had come and decodes the next sentence; the proposed approach encodes the sentence and several candidates and a classifier chooses the target sentence from the set of candidate sentences](https://ishwarj.com/img/papers/bert/p2-ext-quickthought-fig1.pngickthought-fig1.png)](/img/papers/bert/p2-ext-quickthought-fig1.png)
>
> **What it says:** this is the "quick-thought" model, one of the two papers BERT cites for ranking next sentences. (a) the conventional approach decodes the next sentence word by word. (b) The proposed approach "replaces the decoder with a classifier which chooses the target sentence from a set of candidate sentences".
>
> **What it means:**
>
> A multiple-choice test is faster to mark than an essay, and it can still test the same knowledge. Quick-thought makes that trade: the model picks the true next sentence instead of writing it.
>
> Choosing beats generating: it is faster and trains the same skill. BERT's next sentence prediction is the simplest version: two candidates (the real next sentence or a random one), one yes/no answer.

**3. Repair a damaged sentence (denoising auto-encoders).** Hill et al. (2016) corrupt a sentence and train an encoder-decoder to rebuild the original.

> [!PAPER] Hill, Cho, Korhonen (2016), Learning Distributed Representations of Sentences from Unlabelled Data · §2
> [![The sequential denoising autoencoder paragraph of Hill et al.: each word is deleted with probability p0 and each non-overlapping bigram is swapped with probability px, and an LSTM encoder-decoder is trained to predict the original sentence from the corrupted version](https://ishwarj.com/img/papers/bert/p2-ext-hill-sdae.png2-ext-hill-sdae.png)](/img/papers/bert/p2-ext-hill-sdae.png)
>
> **What it says:** the "sequential denoising autoencoder" (SDAE) is the model behind the phrase "denoising auto-encoder derived objectives". each word is deleted "with (independent) probability $$p_o$$", neighbouring pairs are swapped with probability $$p_x$$, and an LSTM encoder-decoder learns to "predict (as target) the original source sentence" from the damaged one.
>
> **What it means:**
>
> Think of a page with coffee stains: some words are gone, and two words have changed places. A good reader can still say what the page said. The SDAE is trained to be that reader.
>
> This is the closest ancestor of the masked language model. BERT damages the input too, but only asks for the damaged words back, not the whole sentence (Section 3.1 says so explicitly).
>
> The difference looks small, but it changes the model. An SDAE needs a decoder that writes the whole sentence again. BERT needs only an encoder that reads.

<figure class="fig"><svg viewBox="0 0 760 364" role="img" aria-label="Two older ways to learn sentence vectors. Skip-thought encodes a sentence into one vector and trains two decoders to write the previous and the next sentence from it. Ranking methods replace the decoder with a classifier that picks the true next sentence from a few candidates. The scores shown are an illustration."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Learning sentence vectors from neighbouring sentences</text><text class="t-note" x="20.0" y="56.0" text-anchor="start">Skip-thought (Kiros et al., 2015): encode a sentence, then write its neighbours</text><rect class="box-1" x="20.0" y="96.0" width="170.0" height="32.0" rx="6"/><text class="t-tick" x="105.0" y="116.5" text-anchor="middle">I could see the cat</text><line class="edge" x1="194.0" y1="112.0" x2="226.0" y2="112.0" marker-end="url(#ah)"/><rect class="box-1" x="230.0" y="92.0" width="92.0" height="40.0" rx="10"/><text class="t-note" x="276.0" y="117.0" text-anchor="middle">encoder</text><line class="edge" x1="326.0" y1="112.0" x2="352.0" y2="112.0" marker-end="url(#ah)"/><rect class="box-ghost" x="355.0" y="103.0" width="74.0" height="18.0" rx="4"/><g class="mark"><rect class="s1" x="358.0" y="106.0" width="12" height="12" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s1" x="372.0" y="106.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="386.0" y="106.0" width="12" height="12" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="400.0" y="106.0" width="12" height="12" rx="2" style="fill-opacity:0.78"/></g><g class="mark"><rect class="s1" x="414.0" y="106.0" width="12" height="12" rx="2" style="fill-opacity:0.52"/></g><text class="t-muted" x="392.0" y="96.0" text-anchor="middle">sentence vector</text><line class="edge" x1="432.0" y1="106.0" x2="476.0" y2="84.0" marker-end="url(#ah)"/><line class="edge" x1="432.0" y1="118.0" x2="476.0" y2="140.0" marker-end="url(#ah)"/><rect class="box-2" x="480.0" y="64.0" width="92.0" height="36.0" rx="10"/><text class="t-note" x="526.0" y="87.0" text-anchor="middle">decoder</text><line class="edge" x1="576.0" y1="82.0" x2="596.0" y2="82.0" marker-end="url(#ah)"/><text class="t-tick" x="602.0" y="87.0" text-anchor="start">"I got back home"</text><rect class="box-3" x="480.0" y="122.0" width="92.0" height="36.0" rx="10"/><text class="t-note" x="526.0" y="145.0" text-anchor="middle">decoder</text><line class="edge" x1="576.0" y1="140.0" x2="596.0" y2="140.0" marker-end="url(#ah)"/><text class="t-tick" x="602.0" y="145.0" text-anchor="start">"This was strange"</text><text class="t-muted" x="602.0" y="70.0" text-anchor="start">previous sentence</text><text class="t-muted" x="602.0" y="128.0" text-anchor="start">next sentence</text><text class="t-note" x="20.0" y="196.0" text-anchor="start">Ranking (Jernite et al., 2017; Logeswaran and Lee, 2018): pick the true next sentence</text><rect class="box-1" x="20.0" y="220.0" width="170.0" height="32.0" rx="6"/><text class="t-tick" x="105.0" y="240.5" text-anchor="middle">Spring had come.</text><text class="t-muted" x="105.0" y="268.0" text-anchor="middle">the sentence</text><line class="edge" x1="194.0" y1="236.0" x2="232.0" y2="235.0" marker-end="url(#ah)"/><rect class="box" x="236.0" y="220.0" width="230.0" height="30.0" rx="6"/><text class="t-tick" x="351.0" y="239.5" text-anchor="middle">They were so black.</text><line class="edge" x1="470.0" y1="235.0" x2="496.0" y2="235.0" marker-end="url(#ah)"/><rect class="s1" x="500" y="227" width="11.2" height="16" rx="3" style="fill-opacity:0.8"/><text class="t-val" x="517.2" y="240.0" text-anchor="start">0.08</text><line class="edge" x1="194.0" y1="236.0" x2="232.0" y2="275.0" marker-end="url(#ah)"/><rect class="box-3" x="236.0" y="260.0" width="230.0" height="30.0" rx="6"/><text class="t-tick" x="351.0" y="279.5" text-anchor="middle">And yet his crops didn't grow.</text><line class="edge" x1="470.0" y1="275.0" x2="496.0" y2="275.0" marker-end="url(#ah)"/><rect class="s3" x="500" y="267" width="113.4" height="16" rx="3" style="fill-opacity:0.8"/><text class="t-val" x="619.4" y="280.0" text-anchor="start">0.81</text><line class="edge" x1="194.0" y1="236.0" x2="232.0" y2="315.0" marker-end="url(#ah)"/><rect class="box" x="236.0" y="300.0" width="230.0" height="30.0" rx="6"/><text class="t-tick" x="351.0" y="319.5" text-anchor="middle">He had blue eyes.</text><line class="edge" x1="470.0" y1="315.0" x2="496.0" y2="315.0" marker-end="url(#ah)"/><rect class="s1" x="500" y="307" width="15.4" height="16" rx="3" style="fill-opacity:0.8"/><text class="t-val" x="521.4" y="320.0" text-anchor="start">0.11</text><text class="t-muted" x="500.0" y="352.0" text-anchor="start">classifier scores (illustration)</text></svg><figcaption>Learning sentence vectors from neighbouring sentences. Skip-thought writes the previous and next sentence from one sentence vector. Ranking methods score candidate sentences and pick the true next one (scores here are an illustration).</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 760 424" role="img" aria-label="Four frames. 1: a clean sentence. 2: the noise function of Hill et al. deletes a word and swaps two neighbouring words. 3: their sequential denoising auto-encoder rebuilds the whole clean sentence. 4: BERT also repairs damaged input, but only the hidden word is a target."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Denoising: damage the input, learn to repair it</text><rect class="box" x="20.0" y="40.0" width="720.0" height="62.0" rx="12"/><circle class="node on" cx="38.0" cy="58.0" r="11"/><text class="t-note" x="38.0" y="62.5" text-anchor="middle">1</text><text class="t-title" x="56.0" y="63.0" text-anchor="start">clean sentence S</text><rect class="box" x="300.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="333.0" y="74.5" text-anchor="middle">the</text><rect class="box" x="372.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="405.0" y="74.5" text-anchor="middle">dog</text><rect class="box" x="444.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="477.0" y="74.5" text-anchor="middle">chased</text><rect class="box" x="516.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="549.0" y="74.5" text-anchor="middle">a</text><rect class="box" x="588.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="621.0" y="74.5" text-anchor="middle">red</text><rect class="box" x="660.0" y="56.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="693.0" y="74.5" text-anchor="middle">ball</text><rect class="box" x="20.0" y="110.0" width="720.0" height="92.0" rx="12"/><circle class="node on" cx="38.0" cy="128.0" r="11"/><text class="t-note" x="38.0" y="132.5" text-anchor="middle">2</text><text class="t-title" x="56.0" y="133.0" text-anchor="start">noise: delete "a" (pₒ), swap "dog chased" (pₓ)</text><rect class="box" x="300.0" y="158.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="333.0" y="176.5" text-anchor="middle">the</text><rect class="box-2" x="372.0" y="158.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="405.0" y="176.5" text-anchor="middle">chased</text><rect class="box-2" x="444.0" y="158.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="477.0" y="176.5" text-anchor="middle">dog</text><rect class="box" x="516.0" y="158.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="549.0" y="176.5" text-anchor="middle">red</text><rect class="box" x="588.0" y="158.0" width="66.0" height="28.0" rx="6"/><text class="t-tick" x="621.0" y="176.5" text-anchor="middle">ball</text><text class="t-muted" x="290.0" y="177.0" text-anchor="end">damaged input</text><rect class="box" x="20.0" y="210.0" width="720.0" height="104.0" rx="12"/><circle class="node on" cx="38.0" cy="228.0" r="11"/><text class="t-note" x="38.0" y="232.5" text-anchor="middle">3</text><text class="t-title" x="56.0" y="233.0" text-anchor="start">SDAE: an LSTM encoder-decoder rebuilds ALL of S</text><rect class="box-1" x="50.0" y="252.0" width="110.0" height="34.0" rx="8"/><text class="t-note" x="105.0" y="274.0" text-anchor="middle">encoder</text><line class="edge" x1="164.0" y1="269.0" x2="186.0" y2="269.0" marker-end="url(#ah)"/><rect class="box-1" x="190.0" y="252.0" width="110.0" height="34.0" rx="8"/><text class="t-note" x="245.0" y="274.0" text-anchor="middle">decoder</text><line class="edge" x1="304.0" y1="269.0" x2="326.0" y2="269.0" marker-end="url(#ah)"/><rect class="box-3" x="330.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="361.0" y="273.5" text-anchor="middle">the</text><rect class="box-3" x="398.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="429.0" y="273.5" text-anchor="middle">dog</text><rect class="box-3" x="466.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="497.0" y="273.5" text-anchor="middle">chased</text><rect class="box-3" x="534.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="565.0" y="273.5" text-anchor="middle">a</text><rect class="box-3" x="602.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="633.0" y="273.5" text-anchor="middle">red</text><rect class="box-3" x="670.0" y="256.0" width="62.0" height="26.0" rx="6"/><text class="t-tick" x="701.0" y="273.5" text-anchor="middle">ball</text><text class="t-muted" x="534.0" y="302.0" text-anchor="middle">every word of S is a target</text><rect class="box" x="20.0" y="322.0" width="720.0" height="92.0" rx="12"/><circle class="node on" cx="38.0" cy="340.0" r="11"/><text class="t-note" x="38.0" y="344.5" text-anchor="middle">4</text><text class="t-title" x="56.0" y="345.0" text-anchor="start">BERT masked LM: predict ONLY the hidden word</text><rect class="box" x="50.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="80.0" y="384.5" text-anchor="middle">the</text><rect class="box" x="116.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="146.0" y="384.5" text-anchor="middle">dog</text><rect class="box" x="182.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="212.0" y="384.5" text-anchor="middle">chased</text><rect class="box" x="248.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="278.0" y="384.5" text-anchor="middle">a</text><rect class="box-mask" x="314.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="344.0" y="384.5" text-anchor="middle">[MASK]</text><rect class="box" x="380.0" y="366.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="410.0" y="384.5" text-anchor="middle">ball</text><line class="edge" x1="450.0" y1="380.0" x2="476.0" y2="380.0" marker-end="url(#ah)"/><rect class="box-1" x="480.0" y="364.0" width="90.0" height="32.0" rx="8"/><text class="t-note" x="525.0" y="385.0" text-anchor="middle">BERT</text><line class="edge" x1="574.0" y1="380.0" x2="596.0" y2="380.0" marker-end="url(#ah)"/><rect class="box-3" x="600.0" y="366.0" width="56.0" height="28.0" rx="6"/><text class="t-tick" x="628.0" y="384.5" text-anchor="middle">red</text><text class="t-muted" x="664.0" y="385.0" text-anchor="start">one target</text></svg><figcaption>Denoising, step by step. A clean sentence is damaged (a word deleted, two words swapped). An SDAE rebuilds every word of the original. BERT's masked LM hides one word and predicts only that word.</figcaption></figure>

### ELMo: a word's vector depends on its sentence {§2.1}

> [!PAPER] · §2.1
> [![The ELMo paragraph: ELMo extracts context-sensitive features from a left-to-right and a right-to-left language model. The contextual representation of each token is the concatenation of the left-to-right and right-to-left representations. Melamud et al. 2016 predicted a word from both sides with LSTMs, but their model is feature-based and not deeply bidirectional](https://ishwarj.com/img/papers/bert/p2-elmo.pngrs/bert/p2-elmo.png)](/img/papers/bert/p2-elmo.png)
>
> **What it says:** ELMo extracts "context-sensitive features" from a left-to-right and a right-to-left language model. Each token's vector "is the concatenation of the left-to-right and right-to-left representations". Plugged into existing task models, ELMo set new records in question answering, sentiment analysis and named entity recognition. Melamud et al. (2016, called context2vec) predicted a word from both sides with LSTMs, but, like ELMo, it is "not deeply bidirectional". Fedus et al. (2018) used the Cloze task to improve text generation.
>
> **What it means:**
>
> Remember the word "bank" in "river bank" and in "bank account". A word embedding gives both the same vector. ELMo, the strongest feature-based model of 2018, fixed this "bank" problem: the vector of a word now depends on its sentence.
>
> The fix has a limit, and the paragraph names it: "not deeply bidirectional". ELMo's two readers each go one way, and their results are only glued together at the end. BERT keeps contextual vectors, and makes the two directions mix in every layer instead of only at the end.

> [!DEFINITION] Contextual embedding
> A word vector that is computed fresh for each sentence, from the words around it. The same word in two sentences gets two different vectors. ELMo and BERT both produce contextual embeddings.

<figure class="fig"><svg viewBox="0 0 800 286" role="img" aria-label="ELMo runs a left-to-right LSTM and a right-to-left LSTM over the sentence. The vector for bank is the forward state glued to the backward state. Inside each reader, information flows only one way; the two directions meet only in this final concatenation."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">ELMo: two one-way readers, glued together at the end</text><rect class="box" x="150.0" y="220.0" width="76.0" height="30.0" rx="6"/><text class="t-tick" x="188.0" y="239.5" text-anchor="middle">he</text><rect class="box" x="238.0" y="220.0" width="76.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="239.5" text-anchor="middle">sat</text><rect class="box" x="326.0" y="220.0" width="76.0" height="30.0" rx="6"/><text class="t-tick" x="364.0" y="239.5" text-anchor="middle">on</text><rect class="box" x="414.0" y="220.0" width="76.0" height="30.0" rx="6"/><text class="t-tick" x="452.0" y="239.5" text-anchor="middle">the</text><rect class="box" x="502.0" y="220.0" width="76.0" height="30.0" rx="6"/><text class="t-tick" x="540.0" y="239.5" text-anchor="middle">bank</text><rect class="box-1" x="158.0" y="150.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="188.0" y="170.0" text-anchor="middle">LSTM →</text><rect class="box-2" x="158.0" y="80.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="188.0" y="100.0" text-anchor="middle">← LSTM</text><line class="edge" x1="188.0" y1="218.0" x2="188.0" y2="182.0" marker-end="url(#ah)"/><line class="edge" x1="218.0" y1="165.0" x2="244.0" y2="165.0" marker-end="url(#ah)"/><line class="edge" x1="246.0" y1="95.0" x2="220.0" y2="95.0" marker-end="url(#ah)"/><rect class="box-1" x="246.0" y="150.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="170.0" text-anchor="middle">LSTM →</text><rect class="box-2" x="246.0" y="80.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="276.0" y="100.0" text-anchor="middle">← LSTM</text><line class="edge" x1="276.0" y1="218.0" x2="276.0" y2="182.0" marker-end="url(#ah)"/><line class="edge" x1="306.0" y1="165.0" x2="332.0" y2="165.0" marker-end="url(#ah)"/><line class="edge" x1="334.0" y1="95.0" x2="308.0" y2="95.0" marker-end="url(#ah)"/><rect class="box-1" x="334.0" y="150.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="364.0" y="170.0" text-anchor="middle">LSTM →</text><rect class="box-2" x="334.0" y="80.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="364.0" y="100.0" text-anchor="middle">← LSTM</text><line class="edge" x1="364.0" y1="218.0" x2="364.0" y2="182.0" marker-end="url(#ah)"/><line class="edge" x1="394.0" y1="165.0" x2="420.0" y2="165.0" marker-end="url(#ah)"/><line class="edge" x1="422.0" y1="95.0" x2="396.0" y2="95.0" marker-end="url(#ah)"/><rect class="box-1" x="422.0" y="150.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="452.0" y="170.0" text-anchor="middle">LSTM →</text><rect class="box-2" x="422.0" y="80.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="452.0" y="100.0" text-anchor="middle">← LSTM</text><line class="edge" x1="452.0" y1="218.0" x2="452.0" y2="182.0" marker-end="url(#ah)"/><line class="edge" x1="482.0" y1="165.0" x2="508.0" y2="165.0" marker-end="url(#ah)"/><line class="edge" x1="510.0" y1="95.0" x2="484.0" y2="95.0" marker-end="url(#ah)"/><rect class="box-1" x="510.0" y="150.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="540.0" y="170.0" text-anchor="middle">LSTM →</text><rect class="box-2" x="510.0" y="80.0" width="60.0" height="30.0" rx="6"/><text class="t-tick" x="540.0" y="100.0" text-anchor="middle">← LSTM</text><line class="edge" x1="540.0" y1="218.0" x2="540.0" y2="182.0" marker-end="url(#ah)"/><line class="edge-dim" x1="540.0" y1="150.0" x2="540.0" y2="112.0"/><text class="t-s1" x="140.0" y="170.0" text-anchor="end">left to right</text><text class="t-s2" x="140.0" y="100.0" text-anchor="end">right to left</text><text class="t-note" x="590.0" y="60.0" text-anchor="start">vector for "bank"</text><rect class="box-ghost" x="587.0" y="73.0" width="68.0" height="20.0" rx="4"/><g class="mark"><rect class="s2" x="590.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s2" x="606.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s2" x="622.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s2" x="638.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="653.0" y="73.0" width="68.0" height="20.0" rx="4"/><g class="mark"><rect class="s1" x="656.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="672.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.40"/></g><g class="mark"><rect class="s1" x="688.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><rect class="s1" x="704.0" y="76.0" width="14" height="14" rx="2" style="fill-opacity:0.62"/></g><text class="t-muted" x="590.0" y="112.0" text-anchor="start">← half  +  → half</text><text class="t-muted" x="20.0" y="272.0" text-anchor="start">Concatenated, not mixed: inside each reader, information flows only one way.</text><line class="edge-2" x1="570.0" y1="95.0" x2="586.0" y2="84.0"/><line class="edge-1" x1="570.0" y1="165.0" x2="686.0" y2="92.0"/></svg><figcaption>ELMo's vector for "bank", drawn. A left-to-right LSTM and a right-to-left LSTM each read the sentence on their own. The vector for "bank" is the forward state glued to the backward state. Inside each reader, information flows one way only.</figcaption></figure>

> [!DEFINITION] Sentiment analysis
> Deciding whether a piece of text is positive or negative, for example a movie review.

We can see the difference between a static and a contextual vector in the real BERT. BERT has both kinds inside it: its first step is a lookup table (a static embedding, one row per token), and its output is a contextual vector. I took the word "bank" in four sentences, two about rivers and two about money, and compared the vectors with **cosine similarity**.

> [!DEFINITION] Cosine similarity
> A number from -1 to 1 that says how much two vectors point the same way. 1 means the same direction; values near 0 mean unrelated. It ignores the length of the vectors and only looks at their direction.

As a formula, for two vectors $$a$$ and $$b$$ of the same length:

$$
\cos(a, b) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}, \qquad a \cdot b = \sum_i a_i b_i, \qquad \lVert a \rVert = \sqrt{\textstyle\sum_i a_i^2}
$$

where $$a \cdot b$$ is the dot product and $$\lVert a \rVert$$ the length. A tiny example: $$a = (1, 2, 2)$$ and $$b = (2, 1, 2)$$ give $$a \cdot b = 2 + 2 + 4 = 8$$, $$\lVert a \rVert = \lVert b \rVert = 3$$, so $$\cos = 8 / 9 = 0.889$$. BERT's vectors have 768 numbers instead of 3, but the formula is the same.

```python
import torch, torch.nn.functional as F
from transformers import BertModel, BertTokenizer

tok = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased").eval()

def bank_vector(sentence):
    enc = tok(sentence, return_tensors="pt")
    pos = enc.input_ids[0].tolist().index(tok.vocab["bank"])   # where "bank" is
    return model(**enc).last_hidden_state[0, pos]              # BERT's output vector for it (768 numbers)

a = bank_vector("he sat on the river bank and watched the water.")
b = bank_vector("she opened a savings account at the bank.")
print(F.cosine_similarity(a, b, dim=0))
```

The real output, for all four sentences:

```text
the input (static) vector of "bank" is the same row of the table in every sentence: cosine 1.000
cosine similarity of BERT's output vectors for "bank":
[1] 1.000  0.747  0.477  0.432   he sat on the river bank and watched the water.
[2] 0.747  1.000  0.458  0.479   the bank of the river was covered in mud.
[3] 0.477  0.458  1.000  0.786   she opened a savings account at the bank.
[4] 0.432  0.479  0.786  1.000   the bank approved my loan yesterday.
average, same meaning (river-river, money-money): 0.766
average, different meaning (river-money):         0.462
```

<figure class="fig"><svg viewBox="0 0 760 380" role="img" aria-label="Cosine similarity between BERT output vectors for the word bank in four sentences. The two river sentences are close to each other, the two money sentences are close to each other, and river versus money pairs are much less similar."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="386.0" y="28.0" text-anchor="middle">cosine similarity of the output vectors of "bank"</text><text class="t-tick" x="262.0" y="73.0" text-anchor="end">[1] river bank</text><g class="mark"><title>[1] river bank → [1] river bank: 1.00</title><rect class="cell" x="271.0" y="41.0" width="56.0" height="56.0" rx="2" style="fill-opacity:1.000"/></g><text class="t-cell on" x="299.0" y="73.0" text-anchor="middle">1.00</text><g class="mark"><title>[1] river bank → [2] bank of river: 0.75</title><rect class="cell" x="329.0" y="41.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.747"/></g><text class="t-cell on" x="357.0" y="73.0" text-anchor="middle">0.75</text><g class="mark"><title>[1] river bank → [3] savings bank: 0.48</title><rect class="cell" x="387.0" y="41.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.477"/></g><text class="t-cell" x="415.0" y="73.0" text-anchor="middle">0.48</text><g class="mark"><title>[1] river bank → [4] bank loan: 0.43</title><rect class="cell" x="445.0" y="41.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.432"/></g><text class="t-cell" x="473.0" y="73.0" text-anchor="middle">0.43</text><text class="t-tick" x="262.0" y="131.0" text-anchor="end">[2] bank of river</text><g class="mark"><title>[2] bank of river → [1] river bank: 0.75</title><rect class="cell" x="271.0" y="99.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.747"/></g><text class="t-cell on" x="299.0" y="131.0" text-anchor="middle">0.75</text><g class="mark"><title>[2] bank of river → [2] bank of river: 1.00</title><rect class="cell" x="329.0" y="99.0" width="56.0" height="56.0" rx="2" style="fill-opacity:1.000"/></g><text class="t-cell on" x="357.0" y="131.0" text-anchor="middle">1.00</text><g class="mark"><title>[2] bank of river → [3] savings bank: 0.46</title><rect class="cell" x="387.0" y="99.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.458"/></g><text class="t-cell" x="415.0" y="131.0" text-anchor="middle">0.46</text><g class="mark"><title>[2] bank of river → [4] bank loan: 0.48</title><rect class="cell" x="445.0" y="99.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.479"/></g><text class="t-cell" x="473.0" y="131.0" text-anchor="middle">0.48</text><text class="t-tick" x="262.0" y="189.0" text-anchor="end">[3] savings bank</text><g class="mark"><title>[3] savings bank → [1] river bank: 0.48</title><rect class="cell" x="271.0" y="157.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.477"/></g><text class="t-cell" x="299.0" y="189.0" text-anchor="middle">0.48</text><g class="mark"><title>[3] savings bank → [2] bank of river: 0.46</title><rect class="cell" x="329.0" y="157.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.458"/></g><text class="t-cell" x="357.0" y="189.0" text-anchor="middle">0.46</text><g class="mark"><title>[3] savings bank → [3] savings bank: 1.00</title><rect class="cell" x="387.0" y="157.0" width="56.0" height="56.0" rx="2" style="fill-opacity:1.000"/></g><text class="t-cell on" x="415.0" y="189.0" text-anchor="middle">1.00</text><g class="mark"><title>[3] savings bank → [4] bank loan: 0.79</title><rect class="cell" x="445.0" y="157.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.786"/></g><text class="t-cell on" x="473.0" y="189.0" text-anchor="middle">0.79</text><text class="t-tick" x="262.0" y="247.0" text-anchor="end">[4] bank loan</text><g class="mark"><title>[4] bank loan → [1] river bank: 0.43</title><rect class="cell" x="271.0" y="215.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.432"/></g><text class="t-cell" x="299.0" y="247.0" text-anchor="middle">0.43</text><g class="mark"><title>[4] bank loan → [2] bank of river: 0.48</title><rect class="cell" x="329.0" y="215.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.479"/></g><text class="t-cell" x="357.0" y="247.0" text-anchor="middle">0.48</text><g class="mark"><title>[4] bank loan → [3] savings bank: 0.79</title><rect class="cell" x="387.0" y="215.0" width="56.0" height="56.0" rx="2" style="fill-opacity:0.786"/></g><text class="t-cell on" x="415.0" y="247.0" text-anchor="middle">0.79</text><g class="mark"><title>[4] bank loan → [4] bank loan: 1.00</title><rect class="cell" x="445.0" y="215.0" width="56.0" height="56.0" rx="2" style="fill-opacity:1.000"/></g><text class="t-cell on" x="473.0" y="247.0" text-anchor="middle">1.00</text><text class="t-tick" x="299.0" y="282.0" text-anchor="end" transform="rotate(-50 299.0 282.0)">[1] river bank</text><text class="t-tick" x="357.0" y="282.0" text-anchor="end" transform="rotate(-50 357.0 282.0)">[2] bank of river</text><text class="t-tick" x="415.0" y="282.0" text-anchor="end" transform="rotate(-50 415.0 282.0)">[3] savings bank</text><text class="t-tick" x="473.0" y="282.0" text-anchor="end" transform="rotate(-50 473.0 282.0)">[4] bank loan</text><text class="t-tick" x="530.0" y="70.0" text-anchor="start">river vs river: 0.747</text><text class="t-tick" x="530.0" y="92.0" text-anchor="start">money vs money: 0.786</text><text class="t-tick" x="530.0" y="114.0" text-anchor="start">river vs money: 0.432 to 0.479</text><text class="t-note" x="530.0" y="160.0" text-anchor="start">The input (static) vector</text><text class="t-note" x="530.0" y="180.0" text-anchor="start">of "bank" is one fixed row</text><text class="t-note" x="530.0" y="200.0" text-anchor="start">of the table: similarity</text><text class="t-note" x="530.0" y="220.0" text-anchor="start">1.000 for every pair.</text></svg><figcaption>Cosine similarity of BERT's output vectors for "bank". The two river sentences are close (0.747), the two money sentences are close (0.786), and every river and money pair is far apart (0.432 to 0.479). A static word embedding would give 1.000 everywhere.</figcaption></figure>

The static row of the table cannot tell the two banks apart: it is literally the same vector. After 12 layers of looking at the neighbours, BERT's output vectors split cleanly into a "river" group and a "money" group. This is what "contextual" means, and it is what made ELMo, and then BERT, so useful.

**Use case.** A search engine that knows "bank" in "fishing spots on the river bank" is not about money will not show loan offers for that query.

### Fine-tuning approaches {§2.2}

> [!PAPER] · §2.2
> [![Section 2.2, Unsupervised Fine-tuning Approaches: the first works only pre-trained word embedding parameters. More recently, sentence or document encoders have been pre-trained from unlabeled text and fine-tuned for a supervised downstream task. The advantage is that few parameters need to be learned from scratch. OpenAI GPT achieved previously state-of-the-art results on many sentence-level tasks from the GLUE benchmark](https://ishwarj.com/img/papers/bert/p2-finetune.pngert/p2-finetune.png)](/img/papers/bert/p2-finetune.png)
> [![The paragraph continues on page 3: left-to-right language modeling and auto-encoder objectives have been used for pre-training such models](https://ishwarj.com/img/papers/bert/p2-finetune2.pngrt/p2-finetune2.png)](/img/papers/bert/p2-finetune2.png)
>
> **What it says:** the first works (Collobert and Weston, 2008) "only pre-trained word embedding parameters". Later, whole encoders were pre-trained on unlabeled text and "fine-tuned for a supervised downstream task" (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018). The advantage: "few parameters need to be learned from scratch". This is partly why OpenAI GPT held the previous best results on many GLUE tasks. They were pre-trained with left-to-right language modelling and auto-encoder objectives.
>
> **What it means:**
>
> Think of buying a house instead of building one. The first buyers bought only bricks: early work pre-trained just the word vectors. Later work bought whole houses: the entire encoder was pre-trained, then fine-tuned for each task.
>
> This is the family BERT belongs to. BERT takes this recipe unchanged (pre-train everything, fine-tune everything) and only changes what happens during pre-training.

> [!DEFINITION] Supervised learning
> Learning from labelled examples: inputs paired with the right answers. Fine-tuning on a task is supervised. Pre-training on plain text is called **unsupervised** (or self-supervised), because the "answers" come from the text itself.

The key phrase is "**few parameters need to be learned from scratch**". With the feature-based approach, the whole task model above the features starts from random numbers and must learn from the task's small labelled set. With fine-tuning, almost every weight starts pre-trained; only a tiny output layer is new. Less to learn from scratch means less labelled data is needed. Howard and Ruder's 2018 paper (ULMFiT) showed this works well for text classification with LSTMs; GPT showed it with a Transformer.

### Transfer learning from labelled data {§2.3}

> [!PAPER] · §2.3
> [![Section 2.3, Transfer Learning from Supervised Data: effective transfer from supervised tasks with large datasets, such as natural language inference and machine translation. Computer vision has shown the importance of transfer learning from large pre-trained models, where an effective recipe is to fine-tune models pre-trained with ImageNet](https://ishwarj.com/img/papers/bert/p2-supervised.pngt/p2-supervised.png)](/img/papers/bert/p2-supervised.png)
>
> **What it says:** models can also be pre-trained on a big **labelled** task and transferred: natural language inference (Conneau et al., 2017) and machine translation (McCann et al., 2017). In computer vision, "an effective recipe is to fine-tune models pre-trained with ImageNet".
>
> **What it means:**
>
> In vision, the ImageNet recipe was the model to copy. Nobody trained an image classifier from zero anymore; everyone started from a network pre-trained on ImageNet. A small task, such as finding cracks in photos of roads, then needed far fewer labelled photos.
>
> Language had tried the same with labelled tasks, such as natural language inference and translation. The catch is that labels cost money, so such datasets stay small. BERT brought the ImageNet habit to language, but with **unlabeled** text, which is far more plentiful than any labelled dataset.

> [!DEFINITION] Transfer learning
> Reusing what a model learned on one task to do better on another task. Pre-training and fine-tuning is one form of transfer learning.

> [!DEFINITION] ImageNet
> A large collection of labelled photos (Deng et al., 2009) used to train image classifiers. A network pre-trained on ImageNet became the standard starting point for almost any vision task.

> [!DEFINITION] Machine translation (MT)
> Translating text from one language to another automatically.

<figure class="fig"><svg viewBox="0 0 760 312" role="img" aria-label="Transfer learning. In vision, a network is pre-trained once on ImageNet labels and then fine-tuned for many other image tasks. BERT does the same for language, but its pre-training data is unlabelled text."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">The transfer-learning recipe, in vision and in language</text><text class="t-note" x="20.0" y="60.0" text-anchor="start">vision</text><rect class="box" x="20.0" y="74.0" width="150.0" height="64.0" rx="10"/><text class="t-tick" x="95.0" y="100.0" text-anchor="middle">millions of labelled photos</text><text class="t-muted" x="95.0" y="118.0" text-anchor="middle">ImageNet classes</text><line class="edge" x1="174.0" y1="106.0" x2="214.0" y2="106.0" marker-end="url(#ah)"/><rect class="box-2" x="218.0" y="82.0" width="110.0" height="48.0" rx="12"/><text class="t-big" x="273.0" y="104.0" text-anchor="middle">CNN</text><text class="t-muted" x="273.0" y="122.0" text-anchor="middle">pre-trained</text><line class="edge" x1="330.0" y1="106.0" x2="398.0" y2="78.0" marker-end="url(#ah)"/><rect class="box-2" x="402.0" y="66.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="82.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="78.0" x2="512.0" y2="78.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="82.0" text-anchor="start">fine-tune: spot tumours</text><line class="edge" x1="330.0" y1="106.0" x2="398.0" y2="108.0" marker-end="url(#ah)"/><rect class="box-2" x="402.0" y="96.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="112.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="108.0" x2="512.0" y2="108.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="112.0" text-anchor="start">fine-tune: find cars</text><line class="edge" x1="330.0" y1="106.0" x2="398.0" y2="138.0" marker-end="url(#ah)"/><rect class="box-2" x="402.0" y="126.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="142.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="138.0" x2="512.0" y2="138.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="142.0" text-anchor="start">fine-tune: sort plants</text><text class="t-note" x="20.0" y="180.0" text-anchor="start">language (BERT)</text><rect class="box" x="20.0" y="194.0" width="150.0" height="64.0" rx="10"/><text class="t-tick" x="95.0" y="220.0" text-anchor="middle">3.3B words, no labels</text><text class="t-muted" x="95.0" y="238.0" text-anchor="middle">masked LM + NSP</text><line class="edge" x1="174.0" y1="226.0" x2="214.0" y2="226.0" marker-end="url(#ah)"/><rect class="box-1" x="218.0" y="202.0" width="110.0" height="48.0" rx="12"/><text class="t-big" x="273.0" y="224.0" text-anchor="middle">BERT</text><text class="t-muted" x="273.0" y="242.0" text-anchor="middle">pre-trained</text><line class="edge" x1="330.0" y1="226.0" x2="398.0" y2="198.0" marker-end="url(#ah)"/><rect class="box-1" x="402.0" y="186.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="202.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="198.0" x2="512.0" y2="198.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="202.0" text-anchor="start">fine-tune: sentiment</text><line class="edge" x1="330.0" y1="226.0" x2="398.0" y2="228.0" marker-end="url(#ah)"/><rect class="box-1" x="402.0" y="216.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="232.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="228.0" x2="512.0" y2="228.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="232.0" text-anchor="start">fine-tune: answer questions</text><line class="edge" x1="330.0" y1="226.0" x2="398.0" y2="258.0" marker-end="url(#ah)"/><rect class="box-1" x="402.0" y="246.0" width="88.0" height="24.0" rx="6"/><text class="t-tick" x="446.0" y="262.0" text-anchor="middle">copy</text><line class="edge" x1="492.0" y1="258.0" x2="512.0" y2="258.0" marker-end="url(#ah)"/><text class="t-tick" x="518.0" y="262.0" text-anchor="start">fine-tune: find names</text><text class="t-muted" x="20.0" y="300.0" text-anchor="start">Step 1 is done once on a huge dataset; step 2 starts every new task from those weights instead of from random numbers.</text></svg><figcaption>The transfer-learning recipe in vision and in language. Step 1, done once: pre-train a big network on a huge dataset (ImageNet's 1.2 million labelled photos; BERT's 3.3 billion words of unlabeled text). Step 2, done for every new task: start from a copy of those weights and fine-tune.</figcaption></figure>

## BERT in two steps, with one architecture {§3}

Now Section 3, the description of BERT itself. It opens with the two steps and with the paper's main picture, Figure 1.

> [!PAPER] · §3
> [![The opening of Section 3: there are two steps in our framework, pre-training and fine-tuning. During pre-training the model is trained on unlabeled data over different pre-training tasks. For fine-tuning, the model is first initialized with the pre-trained parameters, and all of the parameters are fine-tuned using labeled data from the downstream tasks. Each downstream task has separate fine-tuned models](https://ishwarj.com/img/papers/bert/p2-bert-intro.pngt/p2-bert-intro.png)](/img/papers/bert/p2-bert-intro.png)
>
> **What it says:** "There are two steps in our framework: pre-training and fine-tuning." For fine-tuning, BERT starts from the pre-trained parameters and "all of the parameters are fine-tuned using labeled data from the downstream tasks". "Each downstream task has separate fine-tuned models", all starting from the same pre-trained parameters.
>
> **What it means:**
>
> Think of a cookery school. Every student first takes the same long general course. Then each one takes a short course for one job: bread, fish or cakes. At the end there are many cooks, each a separate copy of the same training.
>
> This fixes the vocabulary for the rest of the paper. One pre-training run, then one separate copy per task. The word "all" matters again: every parameter of the copy moves during fine-tuning, not only the new top layer.

> [!DEFINITION] Initialized with
> The numbers a model starts from before training. "Initialized with the pre-trained parameters" means the copy starts as an exact clone of pre-trained BERT, not from random numbers.

> [!PAPER] · §3, Figure 1
> [![Figure 1 of the BERT paper. Left, pre-training: an unlabeled sentence A and B pair goes in as [CLS], the tokens of masked sentence A, [SEP], and the tokens of masked sentence B. BERT turns input embeddings E into outputs C and T; C feeds NSP and the T vectors feed Mask LM. Right, fine-tuning: the same BERT is copied for MNLI, NER and SQuAD; for SQuAD a question and a paragraph go in and start and end span outputs come out](https://ishwarj.com/img/papers/bert/p2-figure1.pngbert/p2-figure1.png)](/img/papers/bert/p2-figure1.png)
>
> **What it says:** "Apart from output layers, the same architectures are used in both pre-training and fine-tuning." "During fine-tuning, all parameters are fine-tuned." `[CLS]` is added in front of every input, and `[SEP]` separates two pieces of text, such as a question and its answer.
>
> **What it means:**
>
> This one picture shows the whole system. The left side is the long general course; the right side is the short course for each task. The big box in the middle is the same on both sides.
>
> The question-answering example on the right is the paper's running example for Section 3. A question and a paragraph go in together, separated by `[SEP]`, and the model points at the answer in the paragraph.
>
> Learn to read this figure and the rest of the paper is easy to follow. The list below reads it from bottom to top.

How to read Figure 1, from bottom to top:

- **Bottom (pink):** the input tokens. `[CLS]`, then the tokens of sentence A (Tok 1 to Tok N), then `[SEP]`, then the tokens of sentence B (Tok 1 to Tok M).
- **Yellow:** **E**, the input embedding of each token. We build these by hand later in this part.
- **Blue box:** BERT itself, the stack of layers. Every position is connected to every other (the faint lines).
- **Green (top):** the outputs. **C** is the output for `[CLS]`. **T₁, …, T_N** and **T₁′, …, T_M′** are the outputs for the other tokens.
- **Red arrows (top):** the output layers. In pre-training, C feeds **NSP** (next sentence prediction) and the T vectors feed **Mask LM**. In fine-tuning, the output layer depends on the task: for **SQuAD** (question answering), the T vectors of the paragraph predict where the answer starts and ends.

<figure class="fig"><svg viewBox="0 0 760 360" role="img" aria-label="Figure 1 redrawn. Left, pre-training: a masked sentence pair goes in; C feeds next sentence prediction and the T vectors at masked positions feed the masked language model. Right, fine-tuning for SQuAD: the same encoder with the same starting weights reads a question and a paragraph, and only the output layer changes: start and end scores over the paragraph tokens."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">Pre-training</text><rect class="box-on" x="12.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="35.0" y="318.5" text-anchor="middle">[CLS]</text><rect class="box-1" x="64.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="87.0" y="318.5" text-anchor="middle">my</text><rect class="box-mask" x="116.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="139.0" y="318.5" text-anchor="middle">[MASK]</text><rect class="box" x="168.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="191.0" y="318.5" text-anchor="middle">[SEP]</text><rect class="box-2" x="220.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="243.0" y="318.5" text-anchor="middle">he</text><rect class="box-mask" x="272.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="295.0" y="318.5" text-anchor="middle">[MASK]</text><rect class="box" x="324.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="347.0" y="318.5" text-anchor="middle">[SEP]</text><text class="t-muted" x="190.0" y="346.0" text-anchor="middle">sentence A, sentence B (masked)</text><line class="edge" x1="35.0" y1="298.0" x2="35.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="87.0" y1="298.0" x2="87.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="139.0" y1="298.0" x2="139.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="191.0" y1="298.0" x2="191.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="243.0" y1="298.0" x2="243.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="295.0" y1="298.0" x2="295.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="347.0" y1="298.0" x2="347.0" y2="280.0" marker-end="url(#ah)"/><rect class="box-4" x="12.0" y="250.0" width="360.0" height="28.0" rx="6"/><text class="t-tick" x="192.0" y="269.0" text-anchor="middle">E: input embeddings</text><rect class="box-1" x="12.0" y="166.0" width="360.0" height="72.0" rx="12"/><text class="t-big" x="192.0" y="198.0" text-anchor="middle">BERT</text><text class="t-muted" x="192.0" y="218.0" text-anchor="middle">same encoder, same starting weights</text><line class="edge" x1="192.0" y1="248.0" x2="192.0" y2="240.0" marker-end="url(#ah)"/><rect class="box-on" x="12.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="35.0" y="130.5" text-anchor="middle">C</text><rect class="box-3" x="64.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="87.0" y="130.5" text-anchor="middle">T₁</text><rect class="box-3" x="116.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="139.0" y="130.5" text-anchor="middle">T₂</text><rect class="box-3" x="168.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="191.0" y="130.5" text-anchor="middle">T</text><rect class="box-3" x="220.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="243.0" y="130.5" text-anchor="middle">T₁′</text><rect class="box-3" x="272.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="295.0" y="130.5" text-anchor="middle">T₂′</text><rect class="box-3" x="324.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="347.0" y="130.5" text-anchor="middle">T</text><line class="edge" x1="35.0" y1="164.0" x2="35.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="87.0" y1="164.0" x2="87.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="139.0" y1="164.0" x2="139.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="191.0" y1="164.0" x2="191.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="243.0" y1="164.0" x2="243.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="295.0" y1="164.0" x2="295.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="347.0" y1="164.0" x2="347.0" y2="142.0" marker-end="url(#ah)"/><line class="edge-on" x1="35.0" y1="110.0" x2="35.0" y2="74.0"/><rect class="box-on" x="12.0" y="44.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="35.0" y="63.0" text-anchor="middle">NSP</text><text class="t-muted" x="35.0" y="38.0" text-anchor="middle">IsNext?</text><line class="edge-1" x1="139.0" y1="110.0" x2="139.0" y2="74.0"/><rect class="box-mask" x="109.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="139.0" y="63.0" text-anchor="middle">Mask LM</text><line class="edge-1" x1="295.0" y1="110.0" x2="295.0" y2="74.0"/><rect class="box-mask" x="265.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-tick" x="295.0" y="63.0" text-anchor="middle">Mask LM</text><text class="t-muted" x="139.0" y="38.0" text-anchor="middle">"dog"</text><text class="t-muted" x="295.0" y="38.0" text-anchor="middle">"likes"</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">Fine-tuning (SQuAD)</text><rect class="box-on" x="392.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="415.0" y="318.5" text-anchor="middle">[CLS]</text><rect class="box-1" x="444.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="467.0" y="318.5" text-anchor="middle">who</text><rect class="box-1" x="496.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="519.0" y="318.5" text-anchor="middle">won</text><rect class="box" x="548.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="571.0" y="318.5" text-anchor="middle">[SEP]</text><rect class="box-2" x="600.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="623.0" y="318.5" text-anchor="middle">Ada</text><rect class="box-2" x="652.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="675.0" y="318.5" text-anchor="middle">won</text><rect class="box" x="704.0" y="300.0" width="46.0" height="28.0" rx="6"/><text class="t-muted" x="727.0" y="318.5" text-anchor="middle">[SEP]</text><text class="t-muted" x="570.0" y="346.0" text-anchor="middle">question, paragraph</text><line class="edge" x1="415.0" y1="298.0" x2="415.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="467.0" y1="298.0" x2="467.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="519.0" y1="298.0" x2="519.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="571.0" y1="298.0" x2="571.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="623.0" y1="298.0" x2="623.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="675.0" y1="298.0" x2="675.0" y2="280.0" marker-end="url(#ah)"/><line class="edge" x1="727.0" y1="298.0" x2="727.0" y2="280.0" marker-end="url(#ah)"/><rect class="box-4" x="392.0" y="250.0" width="360.0" height="28.0" rx="6"/><text class="t-tick" x="572.0" y="269.0" text-anchor="middle">E: input embeddings</text><rect class="box-1" x="392.0" y="166.0" width="360.0" height="72.0" rx="12"/><text class="t-big" x="572.0" y="198.0" text-anchor="middle">BERT</text><text class="t-muted" x="572.0" y="218.0" text-anchor="middle">same encoder, same starting weights</text><line class="edge" x1="572.0" y1="248.0" x2="572.0" y2="240.0" marker-end="url(#ah)"/><rect class="box-on" x="392.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="415.0" y="130.5" text-anchor="middle">C</text><rect class="box-3" x="444.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="467.0" y="130.5" text-anchor="middle">T₁</text><rect class="box-3" x="496.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="519.0" y="130.5" text-anchor="middle">T₂</text><rect class="box-3" x="548.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="571.0" y="130.5" text-anchor="middle">T</text><rect class="box-3" x="600.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="623.0" y="130.5" text-anchor="middle">T₁′</text><rect class="box-3" x="652.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="675.0" y="130.5" text-anchor="middle">T₂′</text><rect class="box-3" x="704.0" y="112.0" width="46.0" height="28.0" rx="6"/><text class="t-tick" x="727.0" y="130.5" text-anchor="middle">T</text><line class="edge" x1="415.0" y1="164.0" x2="415.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="467.0" y1="164.0" x2="467.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="519.0" y1="164.0" x2="519.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="571.0" y1="164.0" x2="571.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="623.0" y1="164.0" x2="623.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="675.0" y1="164.0" x2="675.0" y2="142.0" marker-end="url(#ah)"/><line class="edge" x1="727.0" y1="164.0" x2="727.0" y2="142.0" marker-end="url(#ah)"/><line class="edge-1" x1="623.0" y1="110.0" x2="623.0" y2="74.0"/><rect class="box-3" x="597.0" y="44.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="623.0" y="63.0" text-anchor="middle">start</text><line class="edge-1" x1="623.0" y1="110.0" x2="675.0" y2="74.0"/><rect class="box-3" x="673.0" y="44.0" width="52.0" height="28.0" rx="6"/><text class="t-tick" x="699.0" y="63.0" text-anchor="middle">end</text><text class="t-muted" x="635.0" y="38.0" text-anchor="middle">answer span: "Ada"</text><text class="t-muted" x="415.0" y="98.0" text-anchor="middle">C unused</text><line class="edge-dim" x1="378.0" y1="40.0" x2="378.0" y2="340.0"/></svg><figcaption>Figure 1 redrawn with real words. Left, pre-training: "[CLS] my [MASK] [SEP] he [MASK] [SEP]"; C feeds the IsNext question, the T vectors at the masked positions predict "dog" and "likes". Right, fine-tuning on SQuAD: a question and a paragraph go through the same encoder, starting from the same weights, and the paragraph's T vectors predict where the answer "Ada" starts and ends. C is unused there.</figcaption></figure>

> [!DEFINITION] MNLI, NER, SQuAD
> Three of the downstream tasks in Figure 1. **MNLI** (MultiNLI): natural language inference on sentence pairs. **NER**: named entity recognition, a label for every token. **SQuAD**: question answering, find the answer span in a paragraph. Part 4 covers all of them.

> [!PAPER] · §3
> [![A distinctive feature of BERT is its unified architecture across different tasks](https://ishwarj.com/img/papers/bert/p2-unified.pngbert/p2-unified.png)](/img/papers/bert/p2-unified.png)
> [![There is minimal difference between the pre-trained architecture and the final downstream architecture. Then the Model Architecture paragraph: a multi-layer bidirectional Transformer encoder based on the original implementation of Vaswani et al. 2017, almost identical to the original](https://ishwarj.com/img/papers/bert/p2-architecture.pngp2-architecture.png)](/img/papers/bert/p2-architecture.png)
>
> **What it says:** "A distinctive feature of BERT is its unified architecture across different tasks." There is "minimal difference between the pre-trained architecture and the final downstream architecture". The model is "a multi-layer bidirectional Transformer encoder" based on Vaswani et al. (2017), released in the `tensor2tensor` library, and "almost identical to the original".
>
> **What it means:**
>
> Think of one engine that goes into a car, a boat and a generator. Only the parts attached to it change. BERT's encoder is that engine, and only the output layer changes from task to task.
>
> The "Model Architecture" paragraph then says where the engine comes from. BERT invents no new kind of layer. The architecture is the standard 2017 Transformer encoder. Everything new is in how it is trained.

## The model: a stack of Transformer encoder layers {§3}

The paper skips the details of the Transformer and points to Vaswani et al. (2017) and the guide "The Annotated Transformer". Here is the short version, enough to follow the rest of the paper.

> [!DEFINITION] Layer (Transformer block)
> One repeated unit of the model. Each layer takes one vector per token and returns one improved vector per token, of the same size. BERT stacks 12 or 24 identical layers, each with its own weights.

> [!DEFINITION] Hidden size
> How many numbers are in each token's vector inside the model. BERT-base uses 768. It stays the same from the first layer to the last.

> [!DEFINITION] Self-attention head
> One attention calculation: every token compares itself with every other token, decides how much to look at each, and mixes in their information. A layer runs several heads side by side; each can learn a different pattern (one may follow the previous word, another may link a pronoun to its noun). See the [attention series](https://ishwarj.com/writings/attention-1-self-attention/) for the full story.

Every BERT layer has the same two parts:

1. **Self-attention** with A heads: each token gathers information from the other tokens.
2. **A feed-forward network**: each token's vector is processed on its own by two matrix multiplications with a non-linear step between them.

After each part there is an **add-and-normalise** step: the part's output is added to its input (a "residual connection"), then normalised with **LayerNorm**. Written as equations, for the vectors $$h$$ of all tokens entering a layer:

$$
h' = \text{LayerNorm}\big(h + \text{MultiHeadAttention}(h)\big)
$$

$$
\text{output} = \text{LayerNorm}\big(h' + \text{FFN}(h')\big), \qquad \text{FFN}(x) = \text{GELU}(x W_1 + b_1)\, W_2 + b_2
$$

where:

- $$h$$ holds one $$H$$-number vector per token (the input of the layer), and $$h'$$ is the result after attention;
- $$\text{MultiHeadAttention}$$ runs $$A$$ attention heads in parallel and joins their results;
- $$W_1$$ is an $$H \times 4H$$ matrix and $$W_2$$ a $$4H \times H$$ matrix, with bias vectors $$b_1$$ and $$b_2$$: the feed-forward network makes each vector four times wider, then narrows it back;
- $$\text{GELU}$$ is a smooth version of "keep positive numbers, set negative ones to zero" (Part 3 shows its shape);
- $$\text{LayerNorm}$$ rescales each vector to have mean 0 and spread 1, then applies a learned scale and shift.

> [!DEFINITION] Residual connection
> Adding a block's input to its output: output = input + block(input). The block only has to learn a correction. This keeps deep stacks of layers trainable.

> [!DEFINITION] LayerNorm (layer normalisation)
> A step that rescales each token's vector so its numbers have average 0 and spread 1, then multiplies and shifts them by two learned vectors. It keeps the numbers in a stable range from layer to layer.

The paper hands the details to Vaswani et al. and to "The Annotated Transformer" in two footnotes:

> [!PAPER] · §3, footnotes 1 and 2
> [![Footnotes 1 and 2 of the BERT paper: the tensor2tensor library on GitHub, and The Annotated Transformer at nlp.seas.harvard.edu](https://ishwarj.com/img/papers/bert/p2-footnotes12.png/p2-footnotes12.png)](/img/papers/bert/p2-footnotes12.png)
>
> **What it says:** these are the two links behind "the tensor2tensor library" and "excellent guides such as The Annotated Transformer". Footnote 1 is Google's [tensor2tensor](https://github.com/tensorflow/tensor2tensor) code; footnote 2 is [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html), a line-by-line code walk-through of Vaswani et al. by Harvard NLP.
>
> **What it means:**
>
> Footnotes like these tell you where to look instead of repeating the text. The authors skip the Transformer's details on purpose, because the code and a good guide already exist.
>
> BERT's code started from that Transformer code, so the layer we describe next is exactly the 2017 one.

### Inside one layer: attention, with real numbers

The heart of each layer is the attention equation of Vaswani et al.:

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · §3.2.1, Equation (1)
> [![Section 3.2.1 of Attention Is All You Need: queries are packed together into a matrix Q, keys and values into matrices K and V, and the matrix of outputs is Attention of Q, K, V equals softmax of Q K transpose over square root of d k, times V](https://ishwarj.com/img/papers/bert/p2-ext-vaswani-eq1.pngext-vaswani-eq1.png)](/img/papers/bert/p2-ext-vaswani-eq1.png)
>
> **What it says:** this is scaled dot-product attention, computed for all tokens at once: "we compute the attention function on a set of queries simultaneously, packed together into a matrix $$Q$$", and likewise $$K$$ and $$V$$, then $$\text{softmax}(QK^T / \sqrt{d_k})V$$.
>
> **What it means:**
>
> Think of a library search. Each token writes a question (a query), and every token shows a label (a key). The best matching labels decide which contents (values) each token takes home. Packing all the questions into one matrix lets a computer answer them all in one step.
>
> This one line is most of what BERT computes. BERT uses it with no mask (Part 1), so every token may match every other token.

Where do $$Q$$, $$K$$ and $$V$$ come from? Each is the layer input $$X$$ (one row of $$H = 768$$ numbers per token) multiplied by a learned matrix:

$$
Q = X W^Q, \qquad K = X W^K, \qquad V = X W^V, \qquad \text{head} = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
$$

where, for one head of BERT-base and a text of $$n$$ tokens:

- $$X$$ is $$n \times 768$$;
- $$W^Q$$, $$W^K$$, $$W^V$$ are each $$768 \times 64$$, so $$Q$$, $$K$$, $$V$$ are $$n \times 64$$ (here $$d_k = 64$$);
- $$QK^\top$$ is $$n \times n$$: one score for every pair of tokens;
- softmax works row by row, so each row of weights adds up to 1;
- multiplying by $$V$$ gives the head's output, $$n \times 64$$: for each token, a weighted mix of all tokens' value rows.

<figure class="fig"><svg viewBox="0 0 760 334" role="img" aria-label="The shapes inside one attention head. The input X (n by 768) times Wq (768 by 64) gives Q (n by 64); K and V are made the same way. Q times K transposed gives an n by n grid of scores, divided by 8 and passed through softmax so each row sums to 1. The weights times V give the head output Z, n by 64."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">One attention head of BERT-base, shape by shape (n tokens)</text><rect class="box-ghost" x="36.0" y="66.0" width="126.0" height="56.0" rx="5"/><rect class="s4" x="40.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="50.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="60.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="70.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="80.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="90.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="100.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="110.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="120.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="130.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="140.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="150.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="40.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="50.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="60.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="70.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="80.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="90.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="100.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="110.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="120.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="130.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="140.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="150.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="40.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="50.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="60.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="70.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="80.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="90.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="100.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="110.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="120.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="130.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="140.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="150.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="40.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="50.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="60.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="70.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="80.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="90.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="100.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="110.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="120.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="130.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="140.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="150.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="40.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="50.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="60.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="70.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="80.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="90.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="100.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="110.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="120.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="130.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="140.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="150.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><text class="t-math" x="99.0" y="58.0" text-anchor="middle">X</text><text class="t-tick" x="99.0" y="138.0" text-anchor="middle">n × 768</text><text class="t-big" x="172.0" y="98.0" text-anchor="middle">×</text><rect class="box-ghost" x="184.0" y="56.0" width="46.0" height="126.0" rx="5"/><rect class="s1" x="188.0" y="60.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="198.0" y="60.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="208.0" y="60.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="218.0" y="60.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="188.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="198.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="208.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="218.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="188.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="198.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="208.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="218.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="188.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="198.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="208.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="218.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="188.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="198.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="208.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="218.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="188.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="198.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="208.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="218.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="188.0" y="120.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="198.0" y="120.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="208.0" y="120.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="218.0" y="120.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="188.0" y="130.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="198.0" y="130.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="208.0" y="130.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="218.0" y="130.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="188.0" y="140.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="198.0" y="140.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="208.0" y="140.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="218.0" y="140.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="188.0" y="150.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="198.0" y="150.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="208.0" y="150.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="218.0" y="150.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="188.0" y="160.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="198.0" y="160.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="208.0" y="160.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="218.0" y="160.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="188.0" y="170.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="198.0" y="170.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="208.0" y="170.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="218.0" y="170.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><text class="t-math" x="207.0" y="48.0" text-anchor="middle">Wq</text><text class="t-tick" x="207.0" y="198.0" text-anchor="middle">768 × 64</text><text class="t-big" x="242.0" y="98.0" text-anchor="middle">=</text><rect class="box-ghost" x="256.0" y="66.0" width="46.0" height="56.0" rx="5"/><rect class="s1" x="260.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="270.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="280.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="290.0" y="70.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="260.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="270.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="280.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="290.0" y="80.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="260.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="270.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="280.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="290.0" y="90.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="260.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="270.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="280.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="290.0" y="100.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="260.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="270.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="280.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="290.0" y="110.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><text class="t-math" x="279.0" y="58.0" text-anchor="middle">Q</text><text class="t-tick" x="279.0" y="138.0" text-anchor="middle">n × 64</text><text class="t-tick" x="380.0" y="76.0" text-anchor="start">the same for K = X Wk and V = X Wv</text><text class="t-muted" x="380.0" y="96.0" text-anchor="start">(each W is 768 × 64: one slice of</text><text class="t-muted" x="380.0" y="112.0" text-anchor="start">the 768 × 768 query, key, value matrices)</text><rect class="box-ghost" x="36.0" y="196.0" width="46.0" height="56.0" rx="5"/><rect class="s1" x="40.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="50.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="60.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="70.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="40.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="50.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="60.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="70.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="40.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="50.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="60.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="70.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="40.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="50.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="60.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="70.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="40.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="50.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="60.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="70.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><text class="t-math" x="59.0" y="188.0" text-anchor="middle">Q</text><text class="t-tick" x="59.0" y="268.0" text-anchor="middle">n × 64</text><text class="t-big" x="92.0" y="226.0" text-anchor="middle">×</text><rect class="box-ghost" x="102.0" y="204.0" width="56.0" height="46.0" rx="5"/><rect class="s3" x="106.0" y="208.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="116.0" y="208.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="126.0" y="208.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="136.0" y="208.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="146.0" y="208.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="106.0" y="218.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="116.0" y="218.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="126.0" y="218.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="136.0" y="218.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="146.0" y="218.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="106.0" y="228.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="116.0" y="228.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="126.0" y="228.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="136.0" y="228.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="146.0" y="228.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="106.0" y="238.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="116.0" y="238.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="126.0" y="238.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="136.0" y="238.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="146.0" y="238.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><text class="t-math" x="130.0" y="196.0" text-anchor="middle">Kᵀ</text><text class="t-tick" x="130.0" y="266.0" text-anchor="middle">64 × n</text><text class="t-big" x="170.0" y="226.0" text-anchor="middle">=</text><rect class="box-ghost" x="182.0" y="196.0" width="56.0" height="56.0" rx="5"/><rect class="s4" x="186.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="196.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="206.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="216.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="226.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="186.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="196.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="206.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="216.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="226.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="186.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="196.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="206.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="216.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="226.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="186.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="196.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="206.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="216.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="226.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="186.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="196.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="206.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="216.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="226.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><text class="t-tick" x="210.0" y="188.0" text-anchor="middle">scores</text><text class="t-tick" x="210.0" y="268.0" text-anchor="middle">n × n</text><line class="edge" x1="248.0" y1="222.0" x2="274.0" y2="222.0" marker-end="url(#ah)"/><text class="t-tick" x="262.0" y="210.0" text-anchor="middle">÷ 8</text><text class="t-muted" x="262.0" y="244.0" text-anchor="middle">softmax</text><rect class="box-ghost" x="278.0" y="196.0" width="56.0" height="56.0" rx="5"/><rect class="cell" x="282.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="cell" x="292.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="cell" x="302.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="cell" x="312.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="cell" x="322.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="cell" x="282.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="cell" x="292.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="cell" x="302.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="cell" x="312.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="cell" x="322.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="cell" x="282.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="cell" x="292.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="cell" x="302.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="cell" x="312.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="cell" x="322.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="cell" x="282.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="cell" x="292.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="cell" x="302.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="cell" x="312.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="cell" x="322.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="cell" x="282.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="cell" x="292.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="cell" x="302.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="cell" x="312.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="cell" x="322.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><text class="t-tick" x="306.0" y="188.0" text-anchor="middle">weights</text><text class="t-tick" x="306.0" y="268.0" text-anchor="middle">n × n, rows sum to 1</text><text class="t-big" x="344.0" y="226.0" text-anchor="middle">×</text><rect class="box-ghost" x="354.0" y="196.0" width="46.0" height="56.0" rx="5"/><rect class="s2" x="358.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="368.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="378.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="388.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="358.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="368.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="378.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="388.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="358.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="368.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="378.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="388.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="358.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="368.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="378.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="388.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="358.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="368.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="378.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="388.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><text class="t-math" x="377.0" y="188.0" text-anchor="middle">V</text><text class="t-tick" x="377.0" y="268.0" text-anchor="middle">n × 64</text><text class="t-big" x="412.0" y="226.0" text-anchor="middle">=</text><rect class="box-ghost" x="424.0" y="196.0" width="46.0" height="56.0" rx="5"/><rect class="s3" x="428.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="438.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="448.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="458.0" y="200.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="428.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="438.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="448.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="458.0" y="210.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="428.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="438.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="448.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="458.0" y="220.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="428.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="438.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="448.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="458.0" y="230.0" width="8" height="8" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="428.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="438.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="448.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="458.0" y="240.0" width="8" height="8" rx="2" style="fill-opacity:0.55"/><text class="t-math" x="447.0" y="188.0" text-anchor="middle">Z</text><text class="t-tick" x="447.0" y="268.0" text-anchor="middle">n × 64</text><text class="t-muted" x="20.0" y="320.0" text-anchor="start">Each row of the weights says how much one token looks at every token. Z, the head's output, mixes the value rows with those weights.</text></svg><figcaption>One attention head of BERT-base, shape by shape, for a text of n tokens. X (n × 768) times W_Q (768 × 64) gives Q (n × 64); the same for K and V. Q times Kᵀ gives an n × n table of scores; divided by 8 and passed through softmax, it becomes weights; the weights times V give the head's output Z (n × 64).</figcaption></figure>

**A worked example small enough to check by hand.** Three tokens, $$d_k = 4$$, with simple numbers chosen for the example ([`bert_part2_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part2_math.py)):

```text
Q K^T:
  the      1.000  0.000  2.000
  kid      0.000  3.000  2.000
  smiles   1.000  2.000  3.000
/ sqrt(d_k):
  the      0.500  0.000  1.000
  kid      0.000  1.500  1.000
  smiles   0.500  1.000  1.500
softmax:
  the      0.307  0.186  0.506
  kid      0.122  0.547  0.331
  smiles   0.186  0.307  0.506
output = weights V:
  the      0.814  0.693
  kid      0.453  0.878
  smiles   0.693  0.814
row sums of the weights: [1.0, 1.0, 1.0]
```

Check the row for "kid": the scaled scores are 0, 1.5 and 1.0, and

$$
\frac{e^{0}}{e^{0} + e^{1.5} + e^{1.0}} = \frac{1}{1 + 4.482 + 2.718} = \frac{1}{8.200} = 0.122
$$

and likewise $$4.482 / 8.200 = 0.547$$ and $$2.718 / 8.200 = 0.331$$. "kid" looks mostly at itself, then at "smiles", and all three weights add up to 1.

<figure class="fig"><svg viewBox="0 0 760 260" role="img" aria-label="The tiny attention example computed in code. Left: the scaled scores. Middle: after softmax each row sums to 1; "kid" puts 0.547 on itself. Right: each output row is a weighted average of the value rows."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">The tiny example: 3 tokens, dₖ = 4</text><text class="t-title" x="137.2" y="52.0" text-anchor="middle">scores Q Kᵀ / 2</text><text class="t-muted" x="137.2" y="70.0" text-anchor="middle">key (column)</text><text class="t-tick" x="93.2" y="88.0" text-anchor="middle">the</text><text class="t-tick" x="137.2" y="88.0" text-anchor="middle">kid</text><text class="t-tick" x="181.2" y="88.0" text-anchor="middle">smiles</text><text class="t-tick" x="63.2" y="122.0" text-anchor="end">the</text><g class="mark"><title>the attends to the: 0.33</title><rect class="s4" x="72.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.33"/></g><g class="mark"><title>the attends to kid: 0.00</title><rect class="s4" x="116.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.12"/></g><g class="mark"><title>the attends to smiles: 0.67</title><rect class="s4" x="160.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.67"/></g><text class="t-tick" x="63.2" y="166.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to the: 0.00</title><rect class="s4" x="72.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.12"/></g><g class="mark"><title>kid attends to kid: 1.00</title><rect class="s4" x="116.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:1.00"/></g><g class="mark"><title>kid attends to smiles: 0.67</title><rect class="s4" x="160.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.67"/></g><text class="t-tick" x="63.2" y="210.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to the: 0.33</title><rect class="s4" x="72.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.33"/></g><g class="mark"><title>smiles attends to kid: 0.67</title><rect class="s4" x="116.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.67"/></g><g class="mark"><title>smiles attends to smiles: 1.00</title><rect class="s4" x="160.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:1.00"/></g><text class="t-muted" x="20.0" y="248.0" text-anchor="start">query (row)</text><text class="t-cell" x="106.0" y="122.0" text-anchor="middle">0.5</text><text class="t-cell" x="150.0" y="122.0" text-anchor="middle">0.0</text><text class="t-cell" x="194.0" y="122.0" text-anchor="middle">1.0</text><text class="t-cell" x="106.0" y="166.0" text-anchor="middle">0.0</text><text class="t-cell" x="150.0" y="166.0" text-anchor="middle">1.5</text><text class="t-cell" x="194.0" y="166.0" text-anchor="middle">1.0</text><text class="t-cell" x="106.0" y="210.0" text-anchor="middle">0.5</text><text class="t-cell" x="150.0" y="210.0" text-anchor="middle">1.0</text><text class="t-cell" x="194.0" y="210.0" text-anchor="middle">1.5</text><line class="edge" x1="240.0" y1="162.0" x2="284.0" y2="162.0" marker-end="url(#ah)"/><text class="t-muted" x="262.0" y="152.0" text-anchor="middle">softmax</text><text class="t-muted" x="262.0" y="182.0" text-anchor="middle">per row</text><text class="t-title" x="407.2" y="52.0" text-anchor="middle">weights (rows sum to 1)</text><text class="t-muted" x="407.2" y="70.0" text-anchor="middle">key (column)</text><text class="t-tick" x="363.2" y="88.0" text-anchor="middle">the</text><text class="t-tick" x="407.2" y="88.0" text-anchor="middle">kid</text><text class="t-tick" x="451.2" y="88.0" text-anchor="middle">smiles</text><text class="t-tick" x="333.2" y="122.0" text-anchor="end">the</text><g class="mark"><title>the attends to the: 0.31</title><rect class="cell" x="342.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.31"/></g><text class="t-cell" x="363.2" y="122.0" text-anchor="middle">0.31</text><g class="mark"><title>the attends to kid: 0.19</title><rect class="cell" x="386.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.19"/></g><text class="t-cell" x="407.2" y="122.0" text-anchor="middle">0.19</text><g class="mark"><title>the attends to smiles: 0.51</title><rect class="cell" x="430.7" y="97.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.51"/></g><text class="t-cell" x="451.2" y="122.0" text-anchor="middle">0.51</text><text class="t-tick" x="333.2" y="166.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to the: 0.12</title><rect class="cell" x="342.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="363.2" y="166.0" text-anchor="middle">0.12</text><g class="mark"><title>kid attends to kid: 0.55</title><rect class="cell" x="386.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.55"/></g><text class="t-cell" x="407.2" y="166.0" text-anchor="middle">0.55</text><g class="mark"><title>kid attends to smiles: 0.33</title><rect class="cell" x="430.7" y="141.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.33"/></g><text class="t-cell" x="451.2" y="166.0" text-anchor="middle">0.33</text><text class="t-tick" x="333.2" y="210.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to the: 0.19</title><rect class="cell" x="342.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.19"/></g><text class="t-cell" x="363.2" y="210.0" text-anchor="middle">0.19</text><g class="mark"><title>smiles attends to kid: 0.31</title><rect class="cell" x="386.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.31"/></g><text class="t-cell" x="407.2" y="210.0" text-anchor="middle">0.31</text><g class="mark"><title>smiles attends to smiles: 0.51</title><rect class="cell" x="430.7" y="185.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.51"/></g><text class="t-cell" x="451.2" y="210.0" text-anchor="middle">0.51</text><text class="t-muted" x="290.0" y="248.0" text-anchor="start">query (row)</text><line class="edge" x1="510.0" y1="162.0" x2="548.0" y2="162.0" marker-end="url(#ah)"/><text class="t-muted" x="529.0" y="152.0" text-anchor="middle">× V</text><text class="t-tick" x="640.0" y="70.0" text-anchor="middle">output (2 numbers each)</text><text class="t-tick" x="570.0" y="122.0" text-anchor="end">the</text><g class="mark"><title>0.8137</title><rect class="s1" x="580.0" y="102.0" width="32" height="32" rx="3" style="fill-opacity:0.95"/></g><g class="mark"><title>0.6928</title><rect class="s1" x="616.0" y="102.0" width="32" height="32" rx="3" style="fill-opacity:0.83"/></g><text class="t-cell" x="596.0" y="122.0" text-anchor="middle">0.81</text><text class="t-cell" x="632.0" y="122.0" text-anchor="middle">0.69</text><text class="t-tick" x="570.0" y="166.0" text-anchor="end">kid</text><g class="mark"><title>0.4535</title><rect class="s1" x="580.0" y="146.0" width="32" height="32" rx="3" style="fill-opacity:0.56"/></g><g class="mark"><title>0.8780</title><rect class="s1" x="616.0" y="146.0" width="32" height="32" rx="3" style="fill-opacity:0.95"/></g><text class="t-cell" x="596.0" y="166.0" text-anchor="middle">0.45</text><text class="t-cell" x="632.0" y="166.0" text-anchor="middle">0.88</text><text class="t-tick" x="570.0" y="210.0" text-anchor="end">smiles</text><g class="mark"><title>0.6928</title><rect class="s1" x="580.0" y="190.0" width="32" height="32" rx="3" style="fill-opacity:0.83"/></g><g class="mark"><title>0.8137</title><rect class="s1" x="616.0" y="190.0" width="32" height="32" rx="3" style="fill-opacity:0.95"/></g><text class="t-cell" x="596.0" y="210.0" text-anchor="middle">0.69</text><text class="t-cell" x="632.0" y="210.0" text-anchor="middle">0.81</text></svg><figcaption>The tiny example as a picture. Left: the scaled scores. Middle: softmax turns each row into weights that add up to 1. Right: each token's output is the weighted mix of the value rows.</figcaption></figure>

**Why divide by $$\sqrt{d_k}$$?** If the numbers in $$q$$ and $$k$$ are random with spread 1, their dot product (a sum of $$d_k$$ products) has spread about $$\sqrt{d_k}$$. With $$d_k = 64$$ that is 8, and softmax over numbers that large picks one key and ignores the rest. Measured:

```text
10,000 random pairs, each number drawn with mean 0 and spread 1
spread (standard deviation) of q.k:          8.002   (about sqrt(64) = 8)
spread of q.k / sqrt(64):                     1.000
softmax WITHOUT the division (scores x 8):    [ 0.530,  0.023,  0.004,  0.385,  0.000,  0.001,  0.000,  0.057]   max 0.530
softmax WITH the division:                    [ 0.208,  0.140,  0.114,  0.199,  0.068,  0.091,  0.023,  0.157]   max 0.208
```

<figure class="fig"><svg viewBox="0 0 760 280" role="img" aria-label="Softmax of the same eight scores. Without dividing by 8, the spread of dot products of 64-number vectors is about 8, and softmax puts 0.53 on one key and almost nothing on several. With the division the largest weight is 0.21."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Why divide by √64 = 8: the same 8 scores, softmax with and without the division</text><text class="t-note" x="30.0" y="52.0" text-anchor="start">without ÷ 8 (spread 8)</text><g class="mark"><title>key 1: 0.530</title><rect class="s2" x="30.0" y="81.5" width="30" height="132.5" rx="3"/></g><text class="t-tick" x="45.0" y="75.5" text-anchor="middle">0.53</text><text class="t-muted" x="45.0" y="230.0" text-anchor="middle">1</text><g class="mark"><title>key 2: 0.023</title><rect class="s2" x="70.0" y="208.2" width="30" height="5.8" rx="3"/></g><text class="t-tick" x="85.0" y="202.2" text-anchor="middle">0.02</text><text class="t-muted" x="85.0" y="230.0" text-anchor="middle">2</text><g class="mark"><title>key 3: 0.004</title><rect class="s2" x="110.0" y="213.0" width="30" height="1.0" rx="3"/></g><text class="t-tick" x="125.0" y="207.0" text-anchor="middle">0.00</text><text class="t-muted" x="125.0" y="230.0" text-anchor="middle">3</text><g class="mark"><title>key 4: 0.385</title><rect class="s2" x="150.0" y="117.8" width="30" height="96.2" rx="3"/></g><text class="t-tick" x="165.0" y="111.8" text-anchor="middle">0.39</text><text class="t-muted" x="165.0" y="230.0" text-anchor="middle">4</text><g class="mark"><title>key 5: 0.000</title><rect class="s2" x="190.0" y="214.0" width="30" height="1.0" rx="3"/></g><text class="t-tick" x="205.0" y="208.0" text-anchor="middle">0.00</text><text class="t-muted" x="205.0" y="230.0" text-anchor="middle">5</text><g class="mark"><title>key 6: 0.001</title><rect class="s2" x="230.0" y="213.8" width="30" height="1.0" rx="3"/></g><text class="t-tick" x="245.0" y="207.8" text-anchor="middle">0.00</text><text class="t-muted" x="245.0" y="230.0" text-anchor="middle">6</text><g class="mark"><title>key 7: 0.000</title><rect class="s2" x="270.0" y="214.0" width="30" height="1.0" rx="3"/></g><text class="t-tick" x="285.0" y="208.0" text-anchor="middle">0.00</text><text class="t-muted" x="285.0" y="230.0" text-anchor="middle">7</text><g class="mark"><title>key 8: 0.057</title><rect class="s2" x="310.0" y="199.8" width="30" height="14.2" rx="3"/></g><text class="t-tick" x="325.0" y="193.8" text-anchor="middle">0.06</text><text class="t-muted" x="325.0" y="230.0" text-anchor="middle">8</text><line class="axis" x1="26.0" y1="214.0" x2="344.0" y2="214.0"/><text class="t-note" x="400.0" y="52.0" text-anchor="start">with ÷ 8 (spread 1)</text><g class="mark"><title>key 1: 0.208</title><rect class="s1" x="400.0" y="162.0" width="30" height="52.0" rx="3"/></g><text class="t-tick" x="415.0" y="156.0" text-anchor="middle">0.21</text><text class="t-muted" x="415.0" y="230.0" text-anchor="middle">1</text><g class="mark"><title>key 2: 0.140</title><rect class="s1" x="440.0" y="179.0" width="30" height="35.0" rx="3"/></g><text class="t-tick" x="455.0" y="173.0" text-anchor="middle">0.14</text><text class="t-muted" x="455.0" y="230.0" text-anchor="middle">2</text><g class="mark"><title>key 3: 0.114</title><rect class="s1" x="480.0" y="185.5" width="30" height="28.5" rx="3"/></g><text class="t-tick" x="495.0" y="179.5" text-anchor="middle">0.11</text><text class="t-muted" x="495.0" y="230.0" text-anchor="middle">3</text><g class="mark"><title>key 4: 0.199</title><rect class="s1" x="520.0" y="164.2" width="30" height="49.8" rx="3"/></g><text class="t-tick" x="535.0" y="158.2" text-anchor="middle">0.20</text><text class="t-muted" x="535.0" y="230.0" text-anchor="middle">4</text><g class="mark"><title>key 5: 0.068</title><rect class="s1" x="560.0" y="197.0" width="30" height="17.0" rx="3"/></g><text class="t-tick" x="575.0" y="191.0" text-anchor="middle">0.07</text><text class="t-muted" x="575.0" y="230.0" text-anchor="middle">5</text><g class="mark"><title>key 6: 0.091</title><rect class="s1" x="600.0" y="191.2" width="30" height="22.8" rx="3"/></g><text class="t-tick" x="615.0" y="185.2" text-anchor="middle">0.09</text><text class="t-muted" x="615.0" y="230.0" text-anchor="middle">6</text><g class="mark"><title>key 7: 0.023</title><rect class="s1" x="640.0" y="208.2" width="30" height="5.8" rx="3"/></g><text class="t-tick" x="655.0" y="202.2" text-anchor="middle">0.02</text><text class="t-muted" x="655.0" y="230.0" text-anchor="middle">7</text><g class="mark"><title>key 8: 0.157</title><rect class="s1" x="680.0" y="174.8" width="30" height="39.2" rx="3"/></g><text class="t-tick" x="695.0" y="168.8" text-anchor="middle">0.16</text><text class="t-muted" x="695.0" y="230.0" text-anchor="middle">8</text><line class="axis" x1="396.0" y1="214.0" x2="714.0" y2="214.0"/><text class="t-muted" x="30.0" y="250.0" text-anchor="start">Unscaled, one key takes 0.53 and three get almost 0: softmax is close to "pick one", and its gradients are tiny.</text><text class="t-muted" x="30.0" y="268.0" text-anchor="start">Scaled, the weights stay spread out (largest 0.21), so the model can still learn which keys matter.</text></svg><figcaption>The same eight scores through softmax, without and with the division by 8. Without it, one key takes 0.53 and three get almost nothing, so learning stalls. With it, the weights stay spread out and every key still gets some signal.</figcaption></figure>

**Many heads.** One head learns one way of looking. BERT-base runs $$A = 12$$ heads side by side, each with its own $$W^Q, W^K, W^V$$ of $$768 \times 64$$, and joins their outputs:

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · §3.2.2
> [![Section 3.2.2 of Attention Is All You Need: multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions; MultiHead of Q, K, V equals Concat of head 1 to head h times W O, where head i is Attention of Q W i Q, K W i K, V W i V](https://ishwarj.com/img/papers/bert/p2-ext-vaswani-mha.pngext-vaswani-mha.png)](/img/papers/bert/p2-ext-vaswani-mha.png)
>
> **What it says:** in multi-head attention, the version every Transformer uses, multiple heads let the model "jointly attend to information from different representation subspaces at different positions". The outputs are concatenated and multiplied by $$W^O$$.
>
> **What it means:**
>
> One reader can follow one thread of a story at a time: the plot, or who said what. Twelve readers can follow twelve threads at once. Each head is such a reader, with its own way of looking.
>
> With $$A$$ heads of $$H/A$$ numbers each, the joined output is again $$H$$ wide: $$12 \times 64 = 768$$. That is why BERT's $$H$$ is always a multiple of $$A$$.

$$
\text{MultiHead}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_A)\, W^O
$$

where each $$\text{head}_i$$ is $$n \times 64$$, the concatenation is $$n \times 768$$, and $$W^O$$ is $$768 \times 768$$.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="Multi-head attention in BERT-base. Twelve heads run side by side, each with its own query, key and value matrices of 768 by 64, and each returns 64 numbers per token. Their outputs are concatenated into 768 numbers per token and multiplied by Wo, a 768 by 768 matrix."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Multi-head attention: 12 heads of 64 numbers, joined back into 768</text><rect class="box" x="20.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="40.0" y="66.0" text-anchor="middle">h1</text><g class="mark"><rect class="s1" x="33.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="33.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="33.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="33.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="33.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="66.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="86.0" y="66.0" text-anchor="middle">h2</text><g class="mark"><rect class="s2" x="79.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="79.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="79.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="79.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="79.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="112.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="132.0" y="66.0" text-anchor="middle">h3</text><g class="mark"><rect class="s3" x="125.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="125.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="125.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="125.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="125.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="158.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="178.0" y="66.0" text-anchor="middle">h4</text><g class="mark"><rect class="s4" x="171.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="171.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="171.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="171.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="171.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="204.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="224.0" y="66.0" text-anchor="middle">h5</text><g class="mark"><rect class="s1" x="217.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="217.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="217.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="217.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="217.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="250.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="270.0" y="66.0" text-anchor="middle">h6</text><g class="mark"><rect class="s2" x="263.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="263.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="263.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="263.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="263.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="296.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="316.0" y="66.0" text-anchor="middle">h7</text><g class="mark"><rect class="s3" x="309.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="309.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="309.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="309.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="309.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="342.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="362.0" y="66.0" text-anchor="middle">h8</text><g class="mark"><rect class="s4" x="355.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="355.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="355.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="355.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="355.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="388.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="408.0" y="66.0" text-anchor="middle">h9</text><g class="mark"><rect class="s1" x="401.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="401.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="401.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="401.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="401.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="434.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="454.0" y="66.0" text-anchor="middle">h10</text><g class="mark"><rect class="s2" x="447.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="447.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="447.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="447.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="447.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="480.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="500.0" y="66.0" text-anchor="middle">h11</text><g class="mark"><rect class="s3" x="493.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="493.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="493.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="493.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="493.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="526.0" y="46.0" width="40.0" height="30.0" rx="6"/><text class="t-tick" x="546.0" y="66.0" text-anchor="middle">h12</text><g class="mark"><rect class="s4" x="539.0" y="90.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="539.0" y="104.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="539.0" y="118.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="539.0" y="132.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="539.0" y="146.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><text class="t-tick" x="580.0" y="64.0" text-anchor="start">each head: its own</text><text class="t-tick" x="580.0" y="80.0" text-anchor="start">Wq, Wk, Wv (768 × 64)</text><text class="t-tick" x="580.0" y="112.0" text-anchor="start">output: n × 64</text><line class="edge" x1="20.0" y1="176.0" x2="568.0" y2="176.0"/><line class="edge" x1="294.0" y1="176.0" x2="294.0" y2="196.0" marker-end="url(#ah)"/><g class="mark"><rect class="s1" x="20.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="34.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="48.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s2" x="66.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s2" x="80.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="94.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s3" x="112.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s3" x="126.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="140.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s4" x="158.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s4" x="172.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="186.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s1" x="204.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="218.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="232.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s2" x="250.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s2" x="264.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="278.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s3" x="296.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s3" x="310.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="324.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s4" x="342.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s4" x="356.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="370.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s1" x="388.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="402.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="416.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s2" x="434.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s2" x="448.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="462.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s3" x="480.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s3" x="494.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="508.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><g class="mark"><rect class="s4" x="526.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s4" x="540.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="554.0" y="202.0" width="12" height="12" rx="2" style="fill-opacity:0.66"/></g><text class="t-tick" x="580.0" y="212.0" text-anchor="start">Concat: n × 768</text><text class="t-muted" x="580.0" y="228.0" text-anchor="start">(12 × 64 = 768)</text><line class="edge" x1="294.0" y1="220.0" x2="294.0" y2="242.0" marker-end="url(#ah)"/><rect class="box-1" x="194.0" y="246.0" width="200.0" height="30.0" rx="8"/><text class="t-note" x="294.0" y="266.0" text-anchor="middle">× Wo (768 × 768)</text><line class="edge" x1="294.0" y1="278.0" x2="294.0" y2="298.0" marker-end="url(#ah)"/><rect class="box-3" x="144.0" y="302.0" width="300.0" height="26.0" rx="6"/><text class="t-tick" x="294.0" y="320.0" text-anchor="middle">MultiHead output: n × 768</text></svg><figcaption>Multi-head attention in BERT-base. Twelve heads, each producing 64 numbers per token, are joined side by side into 768 numbers and mixed by W_O (768 × 768).</figcaption></figure>

Here is a real head of the real model on "the kid smiles":

```text
tokens: ['[CLS]', 'the', 'kid', 'smiles', '[SEP]']  (n = 5)
X (input embeddings) (5, 768);  W_Q (768, 768) holds 12 heads of 64 columns
per head: Q (5, 64), K (5, 64), V (5, 64); Q K^T (5, 5); output (5, 64)
12 heads joined: (5, 768); after W_O (768, 768): (5, 768)
head 1 of layer 1, attention weights:
             [CLS]     the     kid  smiles   [SEP]
  [CLS]      0.108   0.246   0.063   0.076   0.506
  the        0.198   0.212   0.175   0.217   0.198
  kid        0.152   0.088   0.139   0.281   0.340
  smiles     0.144   0.160   0.185   0.235   0.276
  [SEP]      0.189   0.200   0.091   0.171   0.348
```

<figure class="fig"><svg viewBox="0 0 760 360" role="img" aria-label="Attention weights of head 1 in layer 1 of the real bert-base-uncased, for the input [CLS] the kid smiles [SEP]. Every token may look at every token. The row for kid puts 0.340 on [SEP] and 0.281 on smiles."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="331.2" y="32.0" text-anchor="middle">real bert-base-uncased, layer 1, head 1</text><text class="t-muted" x="331.2" y="50.0" text-anchor="middle">key (column)</text><text class="t-tick" x="227.2" y="68.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="279.2" y="68.0" text-anchor="middle">the</text><text class="t-tick" x="331.2" y="68.0" text-anchor="middle">kid</text><text class="t-tick" x="383.2" y="68.0" text-anchor="middle">smiles</text><text class="t-tick" x="435.2" y="68.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="193.2" y="106.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]: 0.11</title><rect class="cell" x="202.7" y="77.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="227.2" y="106.0" text-anchor="middle">0.11</text><g class="mark"><title>[CLS] attends to the: 0.25</title><rect class="cell" x="254.7" y="77.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.25"/></g><text class="t-cell" x="279.2" y="106.0" text-anchor="middle">0.25</text><g class="mark"><title>[CLS] attends to kid: 0.06</title><rect class="cell" x="306.7" y="77.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="331.2" y="106.0" text-anchor="middle">0.06</text><g class="mark"><title>[CLS] attends to smiles: 0.08</title><rect class="cell" x="358.7" y="77.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="383.2" y="106.0" text-anchor="middle">0.08</text><g class="mark"><title>[CLS] attends to [SEP]: 0.51</title><rect class="cell" x="410.7" y="77.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.51"/></g><text class="t-cell" x="435.2" y="106.0" text-anchor="middle">0.51</text><text class="t-tick" x="193.2" y="158.0" text-anchor="end">the</text><g class="mark"><title>the attends to [CLS]: 0.20</title><rect class="cell" x="202.7" y="129.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.20"/></g><text class="t-cell" x="227.2" y="158.0" text-anchor="middle">0.20</text><g class="mark"><title>the attends to the: 0.21</title><rect class="cell" x="254.7" y="129.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.21"/></g><text class="t-cell" x="279.2" y="158.0" text-anchor="middle">0.21</text><g class="mark"><title>the attends to kid: 0.18</title><rect class="cell" x="306.7" y="129.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.18"/></g><text class="t-cell" x="331.2" y="158.0" text-anchor="middle">0.18</text><g class="mark"><title>the attends to smiles: 0.22</title><rect class="cell" x="358.7" y="129.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.22"/></g><text class="t-cell" x="383.2" y="158.0" text-anchor="middle">0.22</text><g class="mark"><title>the attends to [SEP]: 0.20</title><rect class="cell" x="410.7" y="129.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.20"/></g><text class="t-cell" x="435.2" y="158.0" text-anchor="middle">0.20</text><text class="t-tick" x="193.2" y="210.0" text-anchor="end">kid</text><g class="mark"><title>kid attends to [CLS]: 0.15</title><rect class="cell" x="202.7" y="181.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.15"/></g><text class="t-cell" x="227.2" y="210.0" text-anchor="middle">0.15</text><g class="mark"><title>kid attends to the: 0.09</title><rect class="cell" x="254.7" y="181.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="279.2" y="210.0" text-anchor="middle">0.09</text><g class="mark"><title>kid attends to kid: 0.14</title><rect class="cell" x="306.7" y="181.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.14"/></g><text class="t-cell" x="331.2" y="210.0" text-anchor="middle">0.14</text><g class="mark"><title>kid attends to smiles: 0.28</title><rect class="cell" x="358.7" y="181.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.28"/></g><text class="t-cell" x="383.2" y="210.0" text-anchor="middle">0.28</text><g class="mark"><title>kid attends to [SEP]: 0.34</title><rect class="cell" x="410.7" y="181.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.34"/></g><text class="t-cell" x="435.2" y="210.0" text-anchor="middle">0.34</text><text class="t-tick" x="193.2" y="262.0" text-anchor="end">smiles</text><g class="mark"><title>smiles attends to [CLS]: 0.14</title><rect class="cell" x="202.7" y="233.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.14"/></g><text class="t-cell" x="227.2" y="262.0" text-anchor="middle">0.14</text><g class="mark"><title>smiles attends to the: 0.16</title><rect class="cell" x="254.7" y="233.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.16"/></g><text class="t-cell" x="279.2" y="262.0" text-anchor="middle">0.16</text><g class="mark"><title>smiles attends to kid: 0.19</title><rect class="cell" x="306.7" y="233.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.19"/></g><text class="t-cell" x="331.2" y="262.0" text-anchor="middle">0.19</text><g class="mark"><title>smiles attends to smiles: 0.24</title><rect class="cell" x="358.7" y="233.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.24"/></g><text class="t-cell" x="383.2" y="262.0" text-anchor="middle">0.24</text><g class="mark"><title>smiles attends to [SEP]: 0.28</title><rect class="cell" x="410.7" y="233.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.28"/></g><text class="t-cell" x="435.2" y="262.0" text-anchor="middle">0.28</text><text class="t-tick" x="193.2" y="314.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]: 0.19</title><rect class="cell" x="202.7" y="285.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.19"/></g><text class="t-cell" x="227.2" y="314.0" text-anchor="middle">0.19</text><g class="mark"><title>[SEP] attends to the: 0.20</title><rect class="cell" x="254.7" y="285.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.20"/></g><text class="t-cell" x="279.2" y="314.0" text-anchor="middle">0.20</text><g class="mark"><title>[SEP] attends to kid: 0.09</title><rect class="cell" x="306.7" y="285.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.12"/></g><text class="t-cell" x="331.2" y="314.0" text-anchor="middle">0.09</text><g class="mark"><title>[SEP] attends to smiles: 0.17</title><rect class="cell" x="358.7" y="285.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.17"/></g><text class="t-cell" x="383.2" y="314.0" text-anchor="middle">0.17</text><g class="mark"><title>[SEP] attends to [SEP]: 0.35</title><rect class="cell" x="410.7" y="285.5" width="49.0" height="49.0" rx="3" style="fill-opacity:0.35"/></g><text class="t-cell" x="435.2" y="314.0" text-anchor="middle">0.35</text><text class="t-muted" x="150.0" y="356.0" text-anchor="start">query (row)</text><text class="t-note" x="560.0" y="100.0" text-anchor="start">Every cell is allowed:</text><text class="t-tick" x="560.0" y="120.0" text-anchor="start">no mask, both directions.</text><text class="t-note" x="560.0" y="150.0" text-anchor="start">Rows sum to 1.</text><text class="t-tick" x="560.0" y="170.0" text-anchor="start">"kid" looks most at</text><text class="t-tick" x="560.0" y="188.0" text-anchor="start">[SEP] (0.340) and</text><text class="t-tick" x="560.0" y="206.0" text-anchor="start">"smiles" (0.281).</text><text class="t-tick" x="560.0" y="236.0" text-anchor="start">Weights near 0.2 =</text><text class="t-tick" x="560.0" y="254.0" text-anchor="start">nearly uniform.</text></svg><figcaption>Head 1 of layer 1 of the real bert-base-uncased. Every square is allowed (no mask) and every row adds up to 1. "kid" looks most at [SEP] (0.340) and at "smiles" (0.281), the word on its right.</figcaption></figure>

### Inside one layer: the feed-forward network

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · §3.3, Equation (2)
> [![Section 3.3 of Attention Is All You Need: each layer contains a fully connected feed-forward network applied to each position separately and identically, two linear transformations with a ReLU activation in between, FFN of x equals max of 0 and x W1 plus b1, times W2, plus b2](https://ishwarj.com/img/papers/bert/p2-ext-vaswani-ffn.pngext-vaswani-ffn.png)](/img/papers/bert/p2-ext-vaswani-ffn.png)
>
> **What it says:** the second part of every Transformer layer is a feed-forward network "applied to each position separately and identically": "two linear transformations with a ReLU activation in between".
>
> **What it means:**
>
> Attention is a meeting where the tokens share information. The feed-forward network is the time after the meeting, when each token works alone on what it heard. The same weights are used for every position.
>
> BERT uses the same shape, with two changes: the inner size is $$4H$$ (footnote 3, below) and GELU replaces ReLU (Part 3).

<figure class="fig"><svg viewBox="0 0 760 256" role="img" aria-label="The feed-forward network of one BERT-base layer, applied to each token on its own. The 768 numbers are multiplied by W1 into 3,072, passed through GELU, and multiplied by W2 back to 768. In layer 1, for the token kid, only 5.2 percent of the 3,072 numbers were positive before GELU."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">The feed-forward network: wider, GELU, narrower (one token at a time)</text><text class="t-tick" x="20.0" y="64.0" text-anchor="start">"kid" after attention</text><rect class="box-ghost" x="17.0" y="71.0" width="245.0" height="15.0" rx="4"/><g class="mark"><rect class="s1" x="20.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="30.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="40.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="50.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="60.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="70.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="80.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="90.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="100.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="110.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="120.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="130.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="140.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="150.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="160.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="170.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="180.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="190.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="200.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="210.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="220.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="230.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="240.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="250.0" y="74.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><text class="t-val" x="139.5" y="102.0" text-anchor="middle">768</text><line class="edge" x1="260.0" y1="80.0" x2="290.0" y2="80.0" marker-end="url(#ah)"/><rect class="box" x="294.0" y="62.0" width="80.0" height="36.0" rx="8"/><text class="t-tick" x="334.0" y="85.0" text-anchor="middle">× W₁ + b₁</text><line class="edge" x1="378.0" y1="80.0" x2="400.0" y2="80.0" marker-end="url(#ah)"/><text class="t-tick" x="404.0" y="64.0" text-anchor="start">wide: 3,072 numbers (4H)</text><rect class="box-ghost" x="401.0" y="71.0" width="313.0" height="12.0" rx="4"/><g class="mark"><rect class="s4" x="404.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="411.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="418.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="425.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="432.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="439.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="446.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="453.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="460.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="467.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="474.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="481.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="488.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="495.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="502.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="509.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="516.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="523.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="530.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="537.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="544.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="551.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="558.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="565.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="572.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="579.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="586.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="593.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="600.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="607.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="614.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="621.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="628.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="635.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="642.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="649.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="656.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="663.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><g class="mark"><rect class="s4" x="670.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="677.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><rect class="s4" x="684.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s4" x="691.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.33"/></g><g class="mark"><rect class="s4" x="698.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.43"/></g><g class="mark"><rect class="s4" x="705.0" y="74.0" width="6" height="6" rx="2" style="fill-opacity:0.29"/></g><line class="edge" x1="560.0" y1="102.0" x2="560.0" y2="130.0" marker-end="url(#ah)"/><text class="t-note" x="570.0" y="120.0" text-anchor="start">GELU</text><text class="t-tick" x="404.0" y="152.0" text-anchor="start">after GELU: only 5.2% were positive before it</text><rect class="box-ghost" x="401.0" y="159.0" width="313.0" height="12.0" rx="4"/><g class="mark"><rect class="s4" x="404.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="411.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="418.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="425.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="432.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="439.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="446.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="453.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="460.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="467.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="474.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="481.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="488.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="495.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="502.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="509.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="516.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="523.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="530.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="537.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="544.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="551.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="558.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="565.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="572.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="579.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="586.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="593.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="600.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="607.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="614.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="621.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="628.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="635.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="642.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="649.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="656.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="663.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="670.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="677.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="684.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="691.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="698.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><g class="mark"><rect class="s4" x="705.0" y="162.0" width="6" height="6" rx="2" style="fill-opacity:0.22"/></g><line class="edge" x1="400.0" y1="168.0" x2="378.0" y2="168.0" marker-end="url(#ah)"/><rect class="box" x="294.0" y="150.0" width="80.0" height="36.0" rx="8"/><text class="t-tick" x="334.0" y="173.0" text-anchor="middle">× W₂ + b₂</text><line class="edge" x1="290.0" y1="168.0" x2="262.0" y2="168.0" marker-end="url(#ah)"/><rect class="box-ghost" x="17.0" y="159.0" width="245.0" height="15.0" rx="4"/><g class="mark"><rect class="s3" x="20.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="30.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="40.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="50.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="60.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="70.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="80.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="90.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="100.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="110.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="120.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="130.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="140.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="150.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="160.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="170.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="180.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="190.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="200.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="210.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="220.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="230.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="240.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="250.0" y="162.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><text class="t-val" x="139.5" y="190.0" text-anchor="middle">back to 768</text><text class="t-muted" x="20.0" y="226.0" text-anchor="start">W₁: 768 × 3072    W₂: 3072 × 768    the same weights for every position in a layer</text><text class="t-muted" x="20.0" y="244.0" text-anchor="start">Real numbers (layer 1, "kid"): first 6 of the 3,072 before GELU -1.33, -0.57, -4.07, -1.99, -1.90, -0.82</text></svg><figcaption>The feed-forward network for one token. "kid" (768 numbers after attention) is widened to 3,072 by W₁, passed through GELU, and narrowed back to 768 by W₂. The same weights are used for every position in the layer. In this real example only 5.2% of the 3,072 numbers were positive before GELU.</figcaption></figure>

Real numbers, for "kid" in layer 1:

```text
in (768,) -> x W1 + b1 (3072,) -> GELU -> x W2 + b2 (768,)
first 6 of the 3072 numbers before GELU: [-1.334, -0.569, -4.067, -1.991, -1.904, -0.816]
the same 6 after GELU:                   [-0.122, -0.162, -0.000, -0.046, -0.054, -0.169]
share of the 3072 that are positive before GELU: 0.052
```

Negative inputs come out of GELU small but not exactly zero (-1.334 becomes -0.122), unlike ReLU, which would make them all 0. Very negative inputs (-4.067) do become almost exactly 0.

<figure class="fig"><svg viewBox="0 0 760 410" role="img" aria-label="The BERT-base encoder. Tokens become input embeddings, pass up through 12 identical layers of self-attention and feed-forward, and come out as one 768-number vector per token: C for [CLS] and T for the others."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><g transform="translate(-60,0)"><rect class="box-2" x="150.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="379.5" text-anchor="middle">[CLS]</text><rect class="box" x="230.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="265.0" y="379.5" text-anchor="middle">my</text><rect class="box" x="310.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="345.0" y="379.5" text-anchor="middle">dog</text><rect class="box" x="390.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="425.0" y="379.5" text-anchor="middle">is</text><rect class="box" x="470.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="505.0" y="379.5" text-anchor="middle">cute</text><rect class="box-2" x="550.0" y="360.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="585.0" y="379.5" text-anchor="middle">[SEP]</text><text class="t-muted" x="140.0" y="380.0" text-anchor="end">tokens</text><rect class="box-4" x="140.0" y="300.0" width="490.0" height="34.0" rx="8"/><text class="t-tick" x="385.0" y="322.0" text-anchor="middle">input embeddings: token + segment + position (768 numbers each)</text><line class="edge" x1="185.0" y1="358.0" x2="185.0" y2="336.0" marker-end="url(#ah)"/><line class="edge" x1="265.0" y1="358.0" x2="265.0" y2="336.0" marker-end="url(#ah)"/><line class="edge" x1="345.0" y1="358.0" x2="345.0" y2="336.0" marker-end="url(#ah)"/><line class="edge" x1="425.0" y1="358.0" x2="425.0" y2="336.0" marker-end="url(#ah)"/><line class="edge" x1="505.0" y1="358.0" x2="505.0" y2="336.0" marker-end="url(#ah)"/><line class="edge" x1="585.0" y1="358.0" x2="585.0" y2="336.0" marker-end="url(#ah)"/><rect class="box-1" x="140.0" y="236.0" width="490.0" height="44.0" rx="10"/><text class="t-note" x="150.0" y="254.0" text-anchor="start">layer 1</text><rect class="box" x="240.0" y="244.0" width="180.0" height="28.0" rx="6"/><text class="t-tick" x="330.0" y="262.0" text-anchor="middle">self-attention, 12 heads</text><rect class="box" x="432.0" y="244.0" width="188.0" height="28.0" rx="6"/><text class="t-tick" x="526.0" y="262.0" text-anchor="middle">feed-forward 768→3072→768</text><rect class="box-1" x="140.0" y="176.0" width="490.0" height="44.0" rx="10"/><text class="t-note" x="150.0" y="194.0" text-anchor="start">layer 2</text><rect class="box" x="240.0" y="184.0" width="180.0" height="28.0" rx="6"/><text class="t-tick" x="330.0" y="202.0" text-anchor="middle">self-attention, 12 heads</text><rect class="box" x="432.0" y="184.0" width="188.0" height="28.0" rx="6"/><text class="t-tick" x="526.0" y="202.0" text-anchor="middle">feed-forward 768→3072→768</text><rect class="box-1" x="140.0" y="116.0" width="490.0" height="44.0" rx="10"/><text class="t-note" x="150.0" y="134.0" text-anchor="start">layer 12</text><rect class="box" x="240.0" y="124.0" width="180.0" height="28.0" rx="6"/><text class="t-tick" x="330.0" y="142.0" text-anchor="middle">self-attention, 12 heads</text><rect class="box" x="432.0" y="124.0" width="188.0" height="28.0" rx="6"/><text class="t-tick" x="526.0" y="142.0" text-anchor="middle">feed-forward 768→3072→768</text><line class="edge" x1="385.0" y1="298.0" x2="385.0" y2="282.0" marker-end="url(#ah)"/><line class="edge" x1="385.0" y1="234.0" x2="385.0" y2="222.0" marker-end="url(#ah)"/><text class="t-muted" x="150.0" y="172.5" text-anchor="start">(9 more layers in between)</text><rect class="box-3" x="150.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="185.0" y="85.5" text-anchor="middle">C</text><rect class="box-3" x="230.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="265.0" y="85.5" text-anchor="middle">T₁</text><rect class="box-3" x="310.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="345.0" y="85.5" text-anchor="middle">T₂</text><rect class="box-3" x="390.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="425.0" y="85.5" text-anchor="middle">T₃</text><rect class="box-3" x="470.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="505.0" y="85.5" text-anchor="middle">T₄</text><rect class="box-3" x="550.0" y="66.0" width="70.0" height="30.0" rx="6"/><text class="t-tick" x="585.0" y="85.5" text-anchor="middle">T₅</text><text class="t-muted" x="140.0" y="86.0" text-anchor="end">outputs</text><line class="edge" x1="185.0" y1="114.0" x2="185.0" y2="98.0" marker-end="url(#ah)"/><line class="edge" x1="265.0" y1="114.0" x2="265.0" y2="98.0" marker-end="url(#ah)"/><line class="edge" x1="345.0" y1="114.0" x2="345.0" y2="98.0" marker-end="url(#ah)"/><line class="edge" x1="425.0" y1="114.0" x2="425.0" y2="98.0" marker-end="url(#ah)"/><line class="edge" x1="505.0" y1="114.0" x2="505.0" y2="98.0" marker-end="url(#ah)"/><line class="edge" x1="585.0" y1="114.0" x2="585.0" y2="98.0" marker-end="url(#ah)"/></g><text class="t-note" x="590.0" y="84.0" text-anchor="start">H = 768</text><text class="t-tick" x="590.0" y="102.0" text-anchor="start">numbers per token</text><text class="t-tick" x="590.0" y="120.0" text-anchor="start">at every level</text><text class="t-note" x="590.0" y="186.0" text-anchor="start">L = 12 layers</text><text class="t-tick" x="590.0" y="204.0" text-anchor="start">stacked</text><text class="t-note" x="590.0" y="250.0" text-anchor="start">A = 12 heads</text><text class="t-tick" x="590.0" y="268.0" text-anchor="start">in every layer</text><text class="t-title" x="20.0" y="30.0" text-anchor="start">BERT-base: L = 12, H = 768, A = 12</text></svg><figcaption>The BERT-base encoder. Tokens become input embeddings, go up through 12 identical layers (each with 12-head self-attention and a feed-forward network), and come out as one 768-number vector per token: C for [CLS] and T for every other token.</figcaption></figure>

To check that this short description is complete, I recomputed layer 1 of the real `bert-base-uncased` by hand, using only its weights and the equations above:

```python
x = model.embeddings(input_ids=ids, token_type_ids=seg)[0]     # (10 tokens, 768)
lay = model.encoder.layer[0]
A, d = 12, 64                                                  # 12 heads of 64 numbers
q = lay.attention.self.query(x).view(-1, A, d).transpose(0, 1) # (12, 10, 64)
k = lay.attention.self.key(x).view(-1, A, d).transpose(0, 1)
v = lay.attention.self.value(x).view(-1, A, d).transpose(0, 1)
att = torch.softmax(q @ k.transpose(1, 2) / d ** 0.5, dim=-1)  # no mask: both directions
heads = (att @ v).transpose(0, 1).reshape(-1, 768)             # join the 12 heads
h1 = lay.attention.output.LayerNorm(x + lay.attention.output.dense(heads))
out = lay.output.LayerNorm(h1 + lay.output.dense(F.gelu(lay.intermediate.dense(h1))))
```

```text
== 7. layer 1 of BERT, recomputed by hand ==
  12 heads of 64 numbers each; every attention row sums to 1: True
  max |difference|, our layer vs the model's layer 1: 8.34e-07
```

The largest difference from the model's own layer is $$8.34 \times 10^{-7}$$, the size of normal rounding errors in 32-bit arithmetic. So the equations above are the whole layer: attention, add and normalise, feed-forward, add and normalise.

## Two model sizes: L, H and A {§3}

> [!PAPER] · §3
> [![We denote the number of layers as L, the hidden size as H, and the number of self-attention heads as A. BERT-BASE: L=12, H=768, A=12, Total Parameters=110M. BERT-LARGE: L=24, H=1024, A=16, Total Parameters=340M. BERT-BASE was chosen to have the same model size as OpenAI GPT. BERT uses bidirectional self-attention, while the GPT Transformer uses constrained self-attention where every token can only attend to context to its left](https://ishwarj.com/img/papers/bert/p2-sizes.pngs/bert/p2-sizes.png)](/img/papers/bert/p2-sizes.png)
>
> **What it says:** **L** is the number of layers, **H** the hidden size, **A** the number of attention heads. Two sizes: **BERT-base** (L=12, H=768, A=12, 110M parameters) and **BERT-large** (L=24, H=1024, A=16, 340M parameters). BERT-base "was chosen to have the same model size as OpenAI GPT for comparison purposes". The one critical difference: "bidirectional self-attention" in BERT versus "constrained self-attention where every token can only attend to context to its left" in GPT.
>
> **What it means:**
>
> To compare two cooks fairly, give them the same kitchen and the same food. BERT-base was made the same size as GPT for this reason. So when BERT-base wins, the size is not the reason.
>
> That leaves one difference: the direction of attention. BERT-large then asks a second question: what happens when the same design gets bigger? Part 5 answers it.

| | Layers L | Hidden size H | Heads A | Numbers per head H/A | Feed-forward size 4H | Parameters (paper) |
|---|---|---|---|---|---|---|
| BERT-base | 12 | 768 | 12 | 64 | 3,072 | 110M |
| BERT-large | 24 | 1,024 | 16 | 64 | 4,096 | 340M |

> [!DEFINITION] Parameter
> One learned number inside the model (a weight or a bias). "110M parameters" means about 110 million such numbers, all set by training.

The feed-forward size comes from a footnote:

> [!PAPER] · §3, footnote 3
> [![Footnote 3: In all cases we set the feed-forward/filter size to be 4H, i.e., 3072 for the H = 768 and 4096 for the H = 1024](https://ishwarj.com/img/papers/bert/p2-footnote3.pngrt/p2-footnote3.png)](/img/papers/bert/p2-footnote3.png)
>
> **What it says:** the feed-forward ("filter") size is always 4H: 3,072 for BERT-base and 4,096 for BERT-large.
>
> **What it means:**
>
> One rule, "always 4H", fixes a whole part of the model. Once you know H, you know the size of the feed-forward network, so the paper needs no extra table for it.
>
> With this number we can compute the exact parameter count ourselves, which we do next.

And footnote 4 explains two names you will see everywhere:

> [!PAPER] · §3, footnote 4
> [![Footnote 4, first line: We note that in the literature the bidirectional Trans-](https://ishwarj.com/img/papers/bert/p2-footnote4a.pngt/p2-footnote4a.png)](/img/papers/bert/p2-footnote4a.png)
> [![Footnote 4, continued on page 4: former is often referred to as a Transformer encoder while the left-context-only version is referred to as a Transformer decoder since it can be used for text generation](https://ishwarj.com/img/papers/bert/p2-footnote4.pngrt/p2-footnote4.png)](/img/papers/bert/p2-footnote4.png)
>
> **What it says:** the bidirectional Transformer "is often referred to as a **Transformer encoder**", and the left-context-only version "as a **Transformer decoder** since it can be used for text generation".
>
> **What it means:**
>
> The names come from what each version can do. A model that can only look left can **write** text one token at a time, because it never needs the future. People call it a decoder. A model that looks both ways can **understand** text better, but cannot simply write the next word. People call it an encoder.
>
> This is why BERT is called an *encoder* model and GPT a *decoder* model. The layers are almost the same; the difference is the attention mask. The trade-off between writing and understanding is why BERT is used for understanding tasks and GPT-style models for generation.

The mask is easiest to see as a grid. Each row is a token doing the looking; each column a token being looked at.

<figure class="fig"><svg viewBox="0 0 760 362" role="img" aria-label="Attention masks. In BERT every row is full: each token may attend to all tokens. In GPT the upper triangle is blocked, so each token attends only to itself and tokens on its left."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="245.0" y="26.0" text-anchor="middle">BERT: every token sees every token</text><text class="t-tick" x="120.0" y="84.0" text-anchor="end">[CLS]</text><rect class="box-1" x="126.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="60.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="120.0" y="124.0" text-anchor="end">my</text><rect class="box-1" x="126.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="100.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="120.0" y="164.0" text-anchor="end">dog</text><rect class="box-1" x="126.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="140.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="120.0" y="204.0" text-anchor="end">is</text><rect class="box-1" x="126.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="180.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="120.0" y="244.0" text-anchor="end">cute</text><rect class="box-1" x="126.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="220.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="120.0" y="284.0" text-anchor="end">[SEP]</text><rect class="box-1" x="126.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="166.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="206.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="246.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="286.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="326.0" y="260.0" width="36.0" height="36.0" rx="4"/><text class="t-muted" x="246.0" y="322.0" text-anchor="middle">columns: the token being looked at</text><text class="t-title" x="615.0" y="26.0" text-anchor="middle">GPT: each token sees only its left</text><text class="t-tick" x="490.0" y="84.0" text-anchor="end">[CLS]</text><rect class="box-1" x="496.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="536.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="576.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="616.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="656.0" y="60.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="696.0" y="60.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="490.0" y="124.0" text-anchor="end">my</text><rect class="box-1" x="496.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="536.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="576.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="616.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="656.0" y="100.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="696.0" y="100.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="490.0" y="164.0" text-anchor="end">dog</text><rect class="box-1" x="496.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="536.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="576.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="616.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="656.0" y="140.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="696.0" y="140.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="490.0" y="204.0" text-anchor="end">is</text><rect class="box-1" x="496.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="536.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="576.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="616.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="656.0" y="180.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="696.0" y="180.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="490.0" y="244.0" text-anchor="end">cute</text><rect class="box-1" x="496.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="536.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="576.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="616.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="656.0" y="220.0" width="36.0" height="36.0" rx="4"/><rect class="box-ghost" x="696.0" y="220.0" width="36.0" height="36.0" rx="4"/><text class="t-tick" x="490.0" y="284.0" text-anchor="end">[SEP]</text><rect class="box-1" x="496.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="536.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="576.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="616.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="656.0" y="260.0" width="36.0" height="36.0" rx="4"/><rect class="box-1" x="696.0" y="260.0" width="36.0" height="36.0" rx="4"/><text class="t-muted" x="616.0" y="322.0" text-anchor="middle">columns: the token being looked at</text><text class="t-tick" x="20.0" y="348.0" text-anchor="start">Rows: the token doing the looking. Filled square: allowed. Dashed square: blocked by the mask.</text></svg><figcaption>Attention masks. In BERT (left) every square is allowed: every token can attend to every token. In GPT (right) the upper triangle is blocked, so each token attends only to itself and to the tokens on its left. That one triangle is the difference between an encoder and a decoder.</figcaption></figure>

### Where "110 million" comes from {§3}

The paper gives the parameter counts as rounded totals. We can rebuild them from L, H, A, the 4H footnote and the vocabulary size. Every count below is checked against the real model.

**Embeddings.** Three lookup tables of $$H$$ numbers per row, plus one LayerNorm (a scale and a shift, $$2H$$):

$$
\underbrace{30{,}522 \times 768}_{\text{tokens}} + \underbrace{512 \times 768}_{\text{positions}} + \underbrace{2 \times 768}_{\text{segments}} + \underbrace{2 \times 768}_{\text{LayerNorm}} = 23{,}837{,}184
$$

**One layer.** Attention has four $$H \times H$$ matrices (query, key, value, output), each with a bias of $$H$$. The feed-forward network has an $$H \times 4H$$ matrix with bias $$4H$$ and a $$4H \times H$$ matrix with bias $$H$$. Two LayerNorms add $$4H$$:

$$
\underbrace{4(H^2 + H)}_{\text{attention}} + \underbrace{2 \cdot 4H^2 + 4H + H}_{\text{feed-forward}} + \underbrace{4H}_{\text{2 LayerNorms}} = 2{,}362{,}368 + 4{,}722{,}432 + 3{,}072 = 7{,}087{,}872
$$

where $$H = 768$$. Twelve layers give $$12 \times 7{,}087{,}872 = 85{,}054{,}464$$.

**Pooler.** One more $$H \times H$$ matrix with a bias, $$H^2 + H = 590{,}592$$ (more on it at the end of this part).

**Total:** $$23{,}837{,}184 + 85{,}054{,}464 + 590{,}592 = 109{,}482{,}240$$.

The script computes the same formula and also counts every parameter in the downloaded model. For BERT-large, it builds the model's shape from its configuration file on PyTorch's "meta" device, which creates the layers without storing any weights, so nothing big is downloaded:

```python
from transformers import BertConfig, BertModel
import torch

c = BertConfig.from_pretrained("bert-large-uncased")   # only the small config file
with torch.device("meta"):                             # shapes only, no memory used for weights
    large = BertModel(c)
print(sum(p.numel() for p in large.parameters()))
```

```text
bert-base-uncased: L=12 H=768 A=12 feed-forward=3072
  embeddings   23,837,184   one layer   7,087,872   x12 =   85,054,464   pooler   590,592
  formula total  109,482,240   counted in the model  109,482,240   same: True
  without the pooler 108,891,648
bert-large-uncased: L=24 H=1024 A=16 feed-forward=4096
  embeddings   31,782,912   one layer  12,596,224   x24 =  302,309,376   pooler 1,049,600
  formula total  335,141,888   counted in the model  335,141,888   same: True
  without the pooler 334,092,288
```

<figure class="fig"><svg viewBox="0 0 760 240" role="img" aria-label="Parameters of BERT-base by part: embeddings 23.84 million, attention 28.35 million, feed-forward 56.67 million, layer norms 0.04 million, pooler 0.59 million."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="40.0" text-anchor="start">Where the 109,482,240 parameters of BERT-base live</text><text class="t-tick" x="202.0" y="71.0" text-anchor="end">embeddings</text><g class="mark"><title>embeddings: 23.84 M</title><rect class="s1" x="210.0" y="53.0" width="151.0" height="26.0" rx="3"/></g><text class="t-val" x="367.0" y="71.0" text-anchor="start">23.84 M</text><text class="t-tick" x="202.0" y="105.0" text-anchor="end">attention, 12 layers</text><g class="mark"><title>attention, 12 layers: 28.35 M</title><rect class="s1" x="210.0" y="87.0" width="179.5" height="26.0" rx="3"/></g><text class="t-val" x="395.5" y="105.0" text-anchor="start">28.35 M</text><text class="t-tick" x="202.0" y="139.0" text-anchor="end">feed-forward, 12 layers</text><g class="mark"><title>feed-forward, 12 layers: 56.67 M</title><rect class="s1" x="210.0" y="121.0" width="358.9" height="26.0" rx="3"/></g><text class="t-val" x="574.9" y="139.0" text-anchor="start">56.67 M</text><text class="t-tick" x="202.0" y="173.0" text-anchor="end">layer norms, 12 layers</text><g class="mark"><title>layer norms, 12 layers: 0.04 M</title><rect class="s1" x="210.0" y="155.0" width="2.0" height="26.0" rx="3"/></g><text class="t-val" x="218.0" y="173.0" text-anchor="start">0.04 M</text><text class="t-tick" x="202.0" y="207.0" text-anchor="end">pooler</text><g class="mark"><title>pooler: 0.59 M</title><rect class="s1" x="210.0" y="189.0" width="3.7" height="26.0" rx="3"/></g><text class="t-val" x="219.7" y="207.0" text-anchor="start">0.59 M</text></svg><figcaption>Where BERT-base's 109,482,240 parameters live. The feed-forward networks hold about half (56.67 million), attention about a quarter (28.35 million), the embedding tables about a fifth (23.84 million).</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 760 290" role="img" aria-label="Parameter counts split by part. BERT-base 109.5 million: embeddings 23.8, attention 28.3, feed-forward 56.7. BERT-large 335.1 million: embeddings 31.8, attention 100.8, feed-forward 201.4. OpenAI GPT, which has the same number of layers, hidden size and heads as BERT-base, has 116.5 million, mostly because of its larger vocabulary."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Parameters term by term, in millions: BERT-base, BERT-large and OpenAI GPT</text><text class="t-note" x="20.0" y="80.0" text-anchor="start">bert-base</text><g class="mark"><title>bert-base, embeddings: 23,837,184</title><rect class="s4" x="140.0" y="60" width="39.3" height="30" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><title>bert-base, attention: 28,348,416</title><rect class="s1" x="179.3" y="60" width="46.7" height="30" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><title>bert-base, feed-forward: 56,669,184</title><rect class="s2" x="226.0" y="60" width="93.3" height="30" rx="2" style="fill-opacity:0.85"/></g><text class="t-cell on" x="272.6" y="80.0" text-anchor="middle">feed-forward 56.7</text><g class="mark"><title>bert-base, LayerNorms: 36,864</title><rect class="s3" x="319.3" y="60" width="1.0" height="30" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><title>bert-base, pooler: 590,592</title><rect class="s3" x="319.4" y="60" width="1.0" height="30" rx="2" style="fill-opacity:0.85"/></g><text class="t-val" x="328.3" y="80.0" text-anchor="start">109.5 M</text><text class="t-note" x="20.0" y="150.0" text-anchor="start">bert-large</text><g class="mark"><title>bert-large, embeddings: 31,782,912</title><rect class="s4" x="140.0" y="130" width="52.3" height="30" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><title>bert-large, attention: 100,761,600</title><rect class="s1" x="192.3" y="130" width="166.0" height="30" rx="2" style="fill-opacity:0.85"/></g><text class="t-cell on" x="275.3" y="150.0" text-anchor="middle">attention 100.8</text><g class="mark"><title>bert-large, feed-forward: 201,449,472</title><rect class="s2" x="358.3" y="130" width="331.8" height="30" rx="2" style="fill-opacity:0.85"/></g><text class="t-cell on" x="524.2" y="150.0" text-anchor="middle">feed-forward 201.4</text><g class="mark"><title>bert-large, LayerNorms: 98,304</title><rect class="s3" x="690.1" y="130" width="1.0" height="30" rx="2" style="fill-opacity:0.85"/></g><g class="mark"><title>bert-large, pooler: 1,049,600</title><rect class="s3" x="690.3" y="130" width="1.7" height="30" rx="2" style="fill-opacity:0.85"/></g><text class="t-val" x="700.0" y="150.0" text-anchor="start">335.1 M</text><text class="t-note" x="20.0" y="220.0" text-anchor="start">OpenAI GPT</text><rect class="s4" x="140" y="200" width="191.9" height="30" rx="2" style="fill-opacity:0.5"/><text class="t-val" x="339.9" y="220.0" text-anchor="start">116.5 M</text><text class="t-tick" x="140.0" y="248.0" text-anchor="start">L=12, H=768, 12 heads, vocabulary 40,478</text><text class="t-muted" x="20.0" y="276.0" text-anchor="start">BERT-base copies GPT's L, H and A. The totals differ by 7 million mainly because GPT's vocabulary is larger (40,478 vs 30,522 tokens).</text></svg><figcaption>The same count as bars, for BERT-base and BERT-large, with OpenAI GPT for comparison. In both BERTs the feed-forward networks are the biggest part. GPT has the same L, H and A as BERT-base, but 116.5 million parameters, mostly because its vocabulary is larger (40,478 tokens against 30,522).</figcaption></figure>

So the formula and the real models agree exactly: **109,482,240** for BERT-base and **335,141,888** for BERT-large. The paper's "110M" and "340M" are both what you get by rounding these to the nearest 10 million (109.5 → 110, 335.1 → 340). The paper does not say how it rounded, so treat this as arithmetic, not as the authors' stated method.

Two things stand out in the picture:

- **The feed-forward networks are the biggest part**, about 52% of all parameters. Making them 4H wide is expensive.
- **The token table alone is about a fifth** of BERT-base: $$30{,}522 \times 768 = 23{,}440{,}896$$ numbers just to look up one vector per token. Later models (ALBERT, in Part 6) attack exactly this.

## Input: sentences and sequences {§3}

> [!PAPER] · §3, Input/Output Representations
> [![Input/Output Representations: the input representation can unambiguously represent both a single sentence and a pair of sentences in one token sequence. A sentence can be an arbitrary span of contiguous text, rather than an actual linguistic sentence. A sequence refers to the input token sequence to BERT, which may be a single sentence or two sentences packed together](https://ishwarj.com/img/papers/bert/p2-input.pngs/bert/p2-input.png)](/img/papers/bert/p2-input.png)
>
> **What it says:** one input format must handle both a single sentence and a pair, such as ⟨Question, Answer⟩. In this paper, "a 'sentence' can be an arbitrary span of contiguous text, rather than an actual linguistic sentence". A "sequence" is the whole input: one sentence, or two packed together.
>
> **What it means:**
>
> In everyday speech, a sentence ends with a full stop. In this paper, a "sentence" is any piece of text cut out in one go, and it may be a whole paragraph. A "sequence" is what BERT actually reads: one such piece, or two packed together.
>
> The authors need these words because many tasks come in pairs, such as a question and a passage. One input format handles both cases, so no task needs a second design.

> [!DEFINITION] Contiguous
> Next to each other, without gaps. A contiguous span of text is a piece cut out in one go, from some word to some later word.

## WordPiece: how text becomes tokens {§3}

> [!PAPER] · §3, Input/Output Representations
> [![We use WordPiece embeddings with a 30,000 token vocabulary. The first token of every sequence is always a special classification token [CLS], whose final hidden state is used as the aggregate sequence representation for classification tasks. Sentence pairs are separated with a special token [SEP], and a learned embedding is added to every token indicating whether it belongs to sentence A or sentence B. The input embedding is E, the final hidden vector of [CLS] is C in R to the H, and the final hidden vector of the i-th token is T i in R to the H](https://ishwarj.com/img/papers/bert/p2-wordpiece.pngrt/p2-wordpiece.png)](/img/papers/bert/p2-wordpiece.png)
>
> **What it says:** (1) "WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary". (2) Every sequence starts with `[CLS]`, whose final vector is "the aggregate sequence representation for classification tasks". (3) Two sentences are separated by `[SEP]`. (4) A "learned embedding" on every token says whether it belongs to sentence A or B. (5) Names: **E** for input embeddings, $$C \in \mathbb{R}^H$$ for the final vector of `[CLS]`, $$T_i \in \mathbb{R}^H$$ for the final vector of token $$i$$.
>
> **What it means:**
>
> This paragraph is short but dense: five ideas in nine lines. It works like the key on a map. Each symbol is introduced once, and the rest of the paper uses it without explanation.
>
> These are the symbols used in every formula from here to the end of the paper. $$C$$ is the one vector that speaks for the whole input, and each $$T_i$$ speaks for one token. Sentence-level tasks read $$C$$, and token-level tasks read the $$T_i$$.

We take the five ideas one at a time, starting with WordPiece.

> [!DEFINITION] Vocabulary
> The fixed list of all tokens a model knows, each with an id number. Any text must be cut into pieces from this list before the model can read it.

> [!DEFINITION] WordPiece
> A way to cut words into pieces from a fixed vocabulary of about 30,000 entries (Wu et al., 2016, from Google's translation system). Common words are one piece. Rare words are split into smaller pieces; a piece that continues a word starts with `##`. "embeddings" becomes `em ##bed ##ding ##s`. No word is ever "unknown" as long as its characters are in the vocabulary.

Why not just use whole words? Because there are too many: names, typos, technical terms, word forms. A whole-word vocabulary would either be enormous or would map many words to "unknown". Why not single letters? Because sequences would get very long. WordPiece is the middle path: about 30,000 pieces cover everything.

The released tokenizer cuts each word with a simple rule, which its code comment calls "a greedy longest-match-first algorithm": take the longest piece at the start of the word that is in the vocabulary, then repeat on the rest with a `##` in front. Here it is in a few lines of Python:

```python
def wordpiece(word, vocab, unk="[UNK]"):
    pieces, start = [], 0
    while start < len(word):
        end = len(word)
        while end > start:                       # try the longest piece first
            piece = ("##" if start > 0 else "") + word[start:end]
            if piece in vocab:
                break
            end -= 1
        if end == start:                         # not even one character matched
            return [unk]
        pieces.append(piece)
        start = end
    return pieces
```

Before this step, the real tokenizer lowercases the text (this is the "uncased" model: "`Uncased` means that the text has been lowercased before WordPiece tokenization", says the release README), strips accents, and splits on spaces and punctuation. I ran our function on every word of pages 1 to 9 of the BERT paper and compared with the real tokenizer:

```text
our WordPiece function vs the real tokenizer, pages 1-9 of the BERT paper:
  8,525 words -> 9,878 pieces (ours), 9,878 pieces (real); identical: True
  distinct words that needed more than one piece: 416
```

Identical, piece for piece. (One detail: the paper's text contains the literal strings `[CLS]`, `[SEP]` and `[MASK]`, which the real tokenizer keeps as special tokens before WordPiece runs. The script removes their brackets so only the WordPiece step is compared.)

Some real splits:

```text
  playing         -> playing
  likes           -> likes
  embeddings      -> em ##bed ##ding ##s
  unaffable       -> una ##ffa ##ble
  tokenization    -> token ##ization
  bidirectional   -> bid ##ire ##ction ##al
```

Two honest notes. First, the pieces are chosen by frequency, not by meaning: "bidirectional" becomes `bid ##ire ##ction ##al`, which a person would never choose. Second, Figure 2 of the paper (below) shows "playing" split into `play ##ing`, but in the released uncased vocabulary "playing" is a single token. The figure illustrates the idea; it is not the output of the real tokenizer. (Even the example in the tokenizer's own code comment, "unaffable" → `un ##aff ##able`, differs from what the released vocabulary gives: `una ##ffa ##ble`.)

The paper cites Wu et al. (2016) for WordPiece. Their own example shows the idea:

> [!PAPER] Wu et al. (2016), Google's Neural Machine Translation System · §4
> [![The WordPiece example from the Google neural machine translation paper: Jet makers feud over seat width with big orders at stake becomes _J et _makers _fe ud _over _seat _width _with _big _orders _at _stake, where the underscore is a special character added to mark the beginning of a word](https://ishwarj.com/img/papers/bert/p2-ext-wu-wordpiece.pngxt-wu-wordpiece.png)](/img/papers/bert/p2-ext-wu-wordpiece.png)
>
> **What it says:** in the WordPiece model, built for Google's translation system, "Jet" becomes two pieces, `_J` and `et`, and "feud" becomes `_fe` and `ud`; the other words stay whole. "_" "is a special character added to mark the beginning of a word".
>
> **What it means:**
>
> Think of a box of toy bricks. A few thousand standard bricks can build any shape. WordPiece keeps common words as single bricks and builds rare words from smaller ones.
>
> BERT uses the same idea with the opposite marker: it marks the pieces that *continue* a word (`##`) instead of the ones that start one.

Here is the greedy rule above, run on "embeddings", with every lookup it makes:

```text
18 lookups, 4 pieces: em ##bed ##ding ##s
  embeddings     not in the vocabulary
  embedding      not in the vocabulary
  ...            (6 more misses)
  em             in the vocabulary -> keep
  ##beddings     not in the vocabulary
  ...            (4 more misses)
  ##bed          in the vocabulary -> keep
  ##dings        not in the vocabulary
  ##ding         in the vocabulary -> keep
  ##s            in the vocabulary -> keep
```

<figure class="fig"><svg viewBox="0 0 760 328" role="img" aria-label="WordPiece on the word embeddings, step by step. Each frame tries the longest remaining piece first and shortens it until it is in the vocabulary. It takes 18 lookups to find the four pieces em, ##bed, ##ding and ##s."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Greedy longest-match-first on "embeddings": 18 lookups, 4 pieces</text><rect class="box" x="20.0" y="44.0" width="720.0" height="62.0" rx="12"/><circle class="node on" cx="38.0" cy="62.0" r="11"/><text class="t-note" x="38.0" y="66.5" text-anchor="middle">1</text><text class="t-title" x="56.0" y="67.0" text-anchor="start">try 9 pieces, longest first</text><rect class="box" x="300.0" y="56.0" width="90.0" height="28.0" rx="6"/><text class="t-muted" x="345.0" y="74.5" text-anchor="middle">embeddings</text><line class="bad-line" x1="304.0" y1="70.0" x2="386.0" y2="70.0"/><rect class="box" x="398.0" y="56.0" width="82.4" height="28.0" rx="6"/><text class="t-muted" x="439.2" y="74.5" text-anchor="middle">embedding</text><line class="bad-line" x1="402.0" y1="70.0" x2="476.4" y2="70.0"/><rect class="box-ghost" x="488.4" y="56.0" width="21.6" height="28.0" rx="6"/><text class="t-muted" x="499.2" y="74.5" text-anchor="middle">…</text><rect class="box-3" x="518.0" y="56.0" width="29.2" height="28.0" rx="6"/><text class="t-mono" x="532.6" y="74.5" text-anchor="middle">em</text><text class="t-tick" x="36.0" y="94.0" text-anchor="start">kept so far: em</text><rect class="box" x="20.0" y="114.0" width="720.0" height="62.0" rx="12"/><circle class="node on" cx="38.0" cy="132.0" r="11"/><text class="t-note" x="38.0" y="136.5" text-anchor="middle">2</text><text class="t-title" x="56.0" y="137.0" text-anchor="start">try 6 pieces, longest first</text><rect class="box" x="300.0" y="126.0" width="90.0" height="28.0" rx="6"/><text class="t-muted" x="345.0" y="144.5" text-anchor="middle">##beddings</text><line class="bad-line" x1="304.0" y1="140.0" x2="386.0" y2="140.0"/><rect class="box" x="398.0" y="126.0" width="82.4" height="28.0" rx="6"/><text class="t-muted" x="439.2" y="144.5" text-anchor="middle">##bedding</text><line class="bad-line" x1="402.0" y1="140.0" x2="476.4" y2="140.0"/><rect class="box-ghost" x="488.4" y="126.0" width="21.6" height="28.0" rx="6"/><text class="t-muted" x="499.2" y="144.5" text-anchor="middle">…</text><rect class="box-3" x="518.0" y="126.0" width="52.0" height="28.0" rx="6"/><text class="t-mono" x="544.0" y="144.5" text-anchor="middle">##bed</text><text class="t-tick" x="36.0" y="164.0" text-anchor="start">kept so far: em ##bed</text><rect class="box" x="20.0" y="184.0" width="720.0" height="62.0" rx="12"/><circle class="node on" cx="38.0" cy="202.0" r="11"/><text class="t-note" x="38.0" y="206.5" text-anchor="middle">3</text><text class="t-title" x="56.0" y="207.0" text-anchor="start">try 2 pieces, longest first</text><rect class="box" x="300.0" y="196.0" width="67.2" height="28.0" rx="6"/><text class="t-muted" x="333.6" y="214.5" text-anchor="middle">##dings</text><line class="bad-line" x1="304.0" y1="210.0" x2="363.2" y2="210.0"/><rect class="box-3" x="375.2" y="196.0" width="59.6" height="28.0" rx="6"/><text class="t-mono" x="405.0" y="214.5" text-anchor="middle">##ding</text><text class="t-tick" x="36.0" y="234.0" text-anchor="start">kept so far: em ##bed ##ding</text><rect class="box" x="20.0" y="254.0" width="720.0" height="62.0" rx="12"/><circle class="node on" cx="38.0" cy="272.0" r="11"/><text class="t-note" x="38.0" y="276.5" text-anchor="middle">4</text><text class="t-title" x="56.0" y="277.0" text-anchor="start">try 1 piece, longest first</text><rect class="box-3" x="300.0" y="266.0" width="36.8" height="28.0" rx="6"/><text class="t-mono" x="318.4" y="284.5" text-anchor="middle">##s</text><text class="t-tick" x="36.0" y="304.0" text-anchor="start">kept so far: em ##bed ##ding ##s</text></svg><figcaption>Greedy longest-match-first on "embeddings", in four rounds. Each round tries the longest remaining piece first and shortens it one letter at a time until a piece is in the vocabulary: em, then ##bed, then ##ding, then ##s.</figcaption></figure>

Interesting detail: "embedding" (singular) is not in the vocabulary either, so the plural is not simply `embedding ##s`. The rule is greedy: it never goes back to try a different split, even if a nicer one exists.

### The vocabulary: "30,000" is 30,522 {§3}

The paper says "a 30,000 token vocabulary". The released `vocab.txt` of `bert-base-uncased` has **30,522** lines. I sorted every entry into groups:

| Group | Entries | Examples |
|---|---|---|
| special tokens | 5 | `[PAD]` `[UNK]` `[CLS]` `[SEP]` `[MASK]` |
| unused placeholders | 994 | `[unused0]` `[unused1]` … |
| single characters | 997 | `!` `"` `#` `a` … |
| `##` + one character | 997 | `##s` `##a` `##e` … |
| `##` pieces, longer | 4,831 | `##ing` `##ed` `##er` `##ly` … |
| whole tokens of 2+ characters | 22,698 | `the` `of` `and` `in` … |
| **total** | **30,522** | |

So "30,000" is a round number. 994 entries are empty placeholder slots (`[unused0]` to `[unused993]`) that the paper does not mention, and 5 are special tokens. The special tokens have fixed ids: `[PAD]`=0, `[UNK]`=100, `[CLS]`=101, `[SEP]`=102, `[MASK]`=103.

> [!DEFINITION] [PAD] and [UNK]
> `[PAD]` fills the empty places when several sequences of different lengths are put in one batch. `[UNK]` ("unknown") stands for a piece of text that cannot be built from the vocabulary at all, for example a character the vocabulary does not contain.

## [CLS], [SEP] and the two segments {§3}

Back to the paragraph. Three special tokens shape every input:

> [!DEFINITION] [CLS]
> "Classification" token. Always the first token of every sequence. It is not a word, so it has no meaning of its own; through self-attention it can gather information from the whole sequence. Its final vector, **C**, is used for whole-sequence tasks like sentiment or next sentence prediction.

> [!DEFINITION] [SEP]
> "Separator" token. Marks the end of sentence A, and the end of sentence B if there is one.

> [!DEFINITION] Segment embedding
> A learned vector added to every token to say which sentence it belongs to: vector $$E_A$$ for every token of sentence A (including the first `[SEP]`), vector $$E_B$$ for every token of sentence B. There are only two of these vectors in the whole model.

> [!DEFINITION] R to the H
> $$\mathbb{R}^H$$ is the set of all lists of $$H$$ real numbers. $$C \in \mathbb{R}^H$$ just says C is a vector of $$H$$ numbers (768 for BERT-base).

Here is what the real tokenizer makes of the pair from Figure 2, `my dog is cute` and `he likes playing`:

```text
Figure 2 pair ("my dog is cute", "he likes playing"):
  tokens:     [CLS]      my     dog      is    cute   [SEP]      he   likes playing   [SEP]
  ids:          101    2026    3899    2003   10140     102    2002    7777    2652     102
  segment:        A       A       A       A       A       A       B       B       B       B
  position:       0       1       2       3       4       5       6       7       8       9
```

Two ways tell the sentences apart, exactly as the paper says: the `[SEP]` token between them, and the segment row (A for the first six tokens, B for the last four).

## Three embeddings, added together {§3}

> [!PAPER] · §3, Input/Output Representations
> [![For a given token, its input representation is constructed by summing the corresponding token, segment, and position embeddings. A visualization of this construction can be seen in Figure 2](https://ishwarj.com/img/papers/bert/p2-sum.pngers/bert/p2-sum.png)](/img/papers/bert/p2-sum.png)
>
> **What it says:** a token's input representation is made "by summing the corresponding token, segment, and position embeddings".
>
> **What it means:**
>
> Think of the label on a parcel: what is inside, which order it belongs to, and its place in the queue. BERT gives every token the same three facts as three vectors, and adds them into one.
>
> This is the very first computation BERT does. Everything else builds on these vectors.

> [!PAPER] · §3, Figure 2
> [![Figure 2 of the BERT paper: the input [CLS] my dog is cute [SEP] he likes play ##ing [SEP]. Below each token: its token embedding E with the token as subscript, plus a segment embedding E A for the first six tokens and E B for the rest, plus a position embedding E 0 to E 10](https://ishwarj.com/img/papers/bert/p2-figure2.pngbert/p2-figure2.png)](/img/papers/bert/p2-figure2.png)
>
> **What it says:** "The input embeddings are the sum of the token embeddings, the segmentation embeddings and the position embeddings." Each column is one token; each row is one kind of embedding.
>
> **What it means:**
>
> Read each column from top to bottom. The token "dog" gets three rows: its own row of the token table, $$E_A$$ for sentence A, and the row for position 2. The three are added into one vector.
>
> Three tables, one lookup in each, one addition. That is the whole input layer. Adding, instead of joining the vectors side by side, keeps the result at $$H$$ numbers.

> [!DEFINITION] Position embedding
> A learned vector for each position in the sequence: one for position 0, one for position 1, and so on. Attention by itself does not know word order ("dog bites man" and "man bites dog" would look the same), so the position vector is added to tell the model where each token sits. BERT learns a table of 512 of them, which is why its inputs are at most 512 tokens long.

As an equation, the input embedding of the token at position $$i$$ is:

$$
E_i = \text{LayerNorm}\big(\, \text{Tok}[\,t_i\,] + \text{Seg}[\,s_i\,] + \text{Pos}[\,i\,] \,\big)
$$

where:

- $$t_i$$ is the token's id (for example 3899 for "dog"), and $$\text{Tok}$$ is the token table, 30,522 rows of 768 numbers;
- $$s_i$$ is the segment (A or B), and $$\text{Seg}$$ is the segment table, 2 rows of 768 numbers;
- $$i$$ is the position (0 to 511), and $$\text{Pos}$$ is the position table, 512 rows of 768 numbers;
- $$\text{Tok}[\,t_i\,]$$ means "row $$t_i$$ of the table", a simple lookup;
- $$\text{LayerNorm}$$ is the same normalisation as inside the layers.

<figure class="fig"><svg viewBox="0 0 760 310" role="img" aria-label="BERT input representation for a real sentence pair. Each token adds three learned vectors: its token embedding, a segment embedding (A or B) and a position embedding (0 to 9). The sum is normalised and sent to layer 1."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="30.0" text-anchor="start">The real input for the pair ("my dog is cute", "he likes playing")</text><rect class="box" x="150.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="176.0" y="71.5" text-anchor="middle">[CLS]</text><rect class="box" x="208.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="234.0" y="71.5" text-anchor="middle">my</text><rect class="box" x="266.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="292.0" y="71.5" text-anchor="middle">dog</text><rect class="box" x="324.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="350.0" y="71.5" text-anchor="middle">is</text><rect class="box" x="382.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="408.0" y="71.5" text-anchor="middle">cute</text><rect class="box" x="440.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="466.0" y="71.5" text-anchor="middle">[SEP]</text><rect class="box" x="498.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="524.0" y="71.5" text-anchor="middle">he</text><rect class="box" x="556.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="582.0" y="71.5" text-anchor="middle">likes</text><rect class="box" x="614.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="640.0" y="71.5" text-anchor="middle">playing</text><rect class="box" x="672.0" y="52.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="698.0" y="71.5" text-anchor="middle">[SEP]</text><text class="t-note" x="140.0" y="72.0" text-anchor="end">input</text><rect class="box-4" x="150.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="176.0" y="123.5" text-anchor="middle">101</text><rect class="box-4" x="208.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="234.0" y="123.5" text-anchor="middle">2026</text><rect class="box-4" x="266.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="292.0" y="123.5" text-anchor="middle">3899</text><rect class="box-4" x="324.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="350.0" y="123.5" text-anchor="middle">2003</text><rect class="box-4" x="382.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="408.0" y="123.5" text-anchor="middle">10140</text><rect class="box-4" x="440.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="466.0" y="123.5" text-anchor="middle">102</text><rect class="box-4" x="498.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="524.0" y="123.5" text-anchor="middle">2002</text><rect class="box-4" x="556.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="582.0" y="123.5" text-anchor="middle">7777</text><rect class="box-4" x="614.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="640.0" y="123.5" text-anchor="middle">2652</text><rect class="box-4" x="672.0" y="104.0" width="52.0" height="30.0" rx="6"/><text class="t-tick" x="698.0" y="123.5" text-anchor="middle">102</text><text class="t-note" x="140.0" y="124.0" text-anchor="end">token id</text><rect class="box-3" x="150.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="176.0" y="175.5" text-anchor="middle">A</text><rect class="box-3" x="208.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="234.0" y="175.5" text-anchor="middle">A</text><rect class="box-3" x="266.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="292.0" y="175.5" text-anchor="middle">A</text><rect class="box-3" x="324.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="350.0" y="175.5" text-anchor="middle">A</text><rect class="box-3" x="382.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="408.0" y="175.5" text-anchor="middle">A</text><rect class="box-3" x="440.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="466.0" y="175.5" text-anchor="middle">A</text><rect class="box-1" x="498.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="524.0" y="175.5" text-anchor="middle">B</text><rect class="box-1" x="556.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="582.0" y="175.5" text-anchor="middle">B</text><rect class="box-1" x="614.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="640.0" y="175.5" text-anchor="middle">B</text><rect class="box-1" x="672.0" y="156.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="698.0" y="175.5" text-anchor="middle">B</text><text class="t-note" x="140.0" y="176.0" text-anchor="end">segment</text><text class="t-muted" x="176.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="234.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="292.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="350.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="408.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="466.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="524.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="582.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="640.0" y="150.0" text-anchor="middle">+</text><text class="t-muted" x="698.0" y="150.0" text-anchor="middle">+</text><rect class="box" x="150.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="176.0" y="227.5" text-anchor="middle">0</text><rect class="box" x="208.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="234.0" y="227.5" text-anchor="middle">1</text><rect class="box" x="266.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="292.0" y="227.5" text-anchor="middle">2</text><rect class="box" x="324.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="350.0" y="227.5" text-anchor="middle">3</text><rect class="box" x="382.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="408.0" y="227.5" text-anchor="middle">4</text><rect class="box" x="440.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="466.0" y="227.5" text-anchor="middle">5</text><rect class="box" x="498.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="524.0" y="227.5" text-anchor="middle">6</text><rect class="box" x="556.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="582.0" y="227.5" text-anchor="middle">7</text><rect class="box" x="614.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="640.0" y="227.5" text-anchor="middle">8</text><rect class="box" x="672.0" y="208.0" width="52.0" height="30.0" rx="6"/><text class="t-mono" x="698.0" y="227.5" text-anchor="middle">9</text><text class="t-note" x="140.0" y="228.0" text-anchor="end">position</text><text class="t-muted" x="176.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="234.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="292.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="350.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="408.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="466.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="524.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="582.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="640.0" y="202.0" text-anchor="middle">+</text><text class="t-muted" x="698.0" y="202.0" text-anchor="middle">+</text><text class="t-note" x="140.0" y="280.0" text-anchor="end">sum</text><rect class="box-on" x="150.0" y="260.0" width="574.0" height="30.0" rx="8"/><text class="t-tick" x="437.0" y="280.0" text-anchor="middle">add the three 768-number vectors, then LayerNorm (and dropout in training)</text></svg><figcaption>The real input for the Figure 2 pair. Each token looks up three vectors (its token id, its segment, its position), the three are added, then normalised. Note that the real tokenizer keeps "playing" as one token, so there are 10 tokens, not the 11 drawn in the paper.</figcaption></figure>

Let us follow one token through this step with real numbers: "dog" in the Figure 2 pair (token id 3899, segment A, position 2). Its input vector is

$$
E_{\text{dog}} = \text{LayerNorm}\big(\text{Tok}[3899] + \text{Seg}[A] + \text{Pos}[2]\big)
$$

where $$\text{Tok}$$ is the $$30{,}522 \times 768$$ token table, $$\text{Seg}$$ the $$2 \times 768$$ segment table (row 0 for A, row 1 for B), $$\text{Pos}$$ the $$512 \times 768$$ position table, and all three are learned during pre-training. The first 6 of the 768 numbers:

```text
Tok[3899]           [-0.015,  0.012,  0.009, -0.014, -0.024, -0.009]  ...
Seg[A]              [ 0.000,  0.011,  0.004,  0.002,  0.001, -0.011]  ...
Pos[2]              [-0.011, -0.002, -0.012, -0.022, -0.010,  0.012]  ...
sum                 [-0.026,  0.021,  0.001, -0.035, -0.034, -0.008]  ...
mean of all 768 numbers of the sum: -0.0185; spread (standard deviation): 0.0517
(sum - mean)/spread [-0.141,  0.773,  0.382, -0.309, -0.292,  0.200]  ...
x gamma + beta      [-0.156,  0.665,  0.352, -0.178, -0.324,  0.166]  ...   (LayerNorm output)
model.embeddings    [-0.156,  0.665,  0.352, -0.178, -0.324,  0.166]  ...   max |difference| 0.0e+00
```

LayerNorm, written out for one vector $$x$$ of $$H$$ numbers:

$$
\mu = \frac{1}{H}\sum_{i=1}^{H} x_i, \qquad \sigma = \sqrt{\frac{1}{H}\sum_{i=1}^{H} (x_i - \mu)^2}, \qquad \text{LayerNorm}(x)_i = \gamma_i \, \frac{x_i - \mu}{\sigma} + \beta_i
$$

where $$\mu$$ is the mean, $$\sigma$$ the spread, and $$\gamma$$, $$\beta$$ are two learned vectors of $$H$$ numbers (a scale and a shift). Check the second number by hand: $$(0.021 - (-0.0185)) / 0.0517 = 0.764$$, close to the printed 0.773 (the printed inputs are rounded to three decimals). Our hand computation matches the model's own embedding layer exactly (difference 0.0).

<figure class="fig"><svg viewBox="0 0 760 372" role="img" aria-label="The input embedding of the token dog in the Figure 2 pair, using the real tables of bert-base-uncased. The first six numbers of the token row, the segment A row and the position 2 row are added, then LayerNorm subtracts the mean, divides by the spread and applies a learned scale and shift."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">The input embedding of "dog", number by number (first 6 of 768)</text><text class="t-tick" x="160.0" y="66.0" text-anchor="end">Tok[3899]  "dog"</text><g class="mark"><title>-0.0149</title><rect class="s2" x="185.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.64"/></g><text class="t-tick" x="200.0" y="92.0" text-anchor="middle">-0.015</text><g class="mark"><title>0.0124</title><rect class="s1" x="221.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.56"/></g><text class="t-tick" x="236.0" y="92.0" text-anchor="middle">0.012</text><g class="mark"><title>0.0091</title><rect class="s1" x="257.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.45"/></g><text class="t-tick" x="272.0" y="92.0" text-anchor="middle">0.009</text><g class="mark"><title>-0.0143</title><rect class="s2" x="293.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="308.0" y="92.0" text-anchor="middle">-0.014</text><g class="mark"><title>-0.0241</title><rect class="s2" x="329.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.95"/></g><text class="t-tick" x="344.0" y="92.0" text-anchor="middle">-0.024</text><g class="mark"><title>-0.0088</title><rect class="s2" x="365.0" y="48.0" width="30" height="30" rx="3" style="fill-opacity:0.44"/></g><text class="t-tick" x="380.0" y="92.0" text-anchor="middle">-0.009</text><text class="t-muted" x="410.0" y="66.0" text-anchor="start">token table, row 3899</text><text class="t-muted" x="405.0" y="82.0" text-anchor="start"></text><text class="t-big" x="170.0" y="120.0" text-anchor="middle">+</text><text class="t-tick" x="160.0" y="122.0" text-anchor="end">Seg[A]</text><g class="mark"><title>0.0004</title><rect class="s1" x="185.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.18"/></g><text class="t-tick" x="200.0" y="148.0" text-anchor="middle">0.000</text><g class="mark"><title>0.0110</title><rect class="s1" x="221.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.95"/></g><text class="t-tick" x="236.0" y="148.0" text-anchor="middle">0.011</text><g class="mark"><title>0.0037</title><rect class="s1" x="257.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.42"/></g><text class="t-tick" x="272.0" y="148.0" text-anchor="middle">0.004</text><g class="mark"><title>0.0015</title><rect class="s1" x="293.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.26"/></g><text class="t-tick" x="308.0" y="148.0" text-anchor="middle">0.002</text><g class="mark"><title>0.0006</title><rect class="s1" x="329.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.19"/></g><text class="t-tick" x="344.0" y="148.0" text-anchor="middle">0.001</text><g class="mark"><title>-0.0109</title><rect class="s2" x="365.0" y="104.0" width="30" height="30" rx="3" style="fill-opacity:0.94"/></g><text class="t-tick" x="380.0" y="148.0" text-anchor="middle">-0.011</text><text class="t-muted" x="410.0" y="122.0" text-anchor="start">segment table, row 0</text><text class="t-muted" x="405.0" y="138.0" text-anchor="start"></text><text class="t-big" x="170.0" y="176.0" text-anchor="middle">+</text><text class="t-tick" x="160.0" y="178.0" text-anchor="end">Pos[2]</text><g class="mark"><title>-0.0113</title><rect class="s2" x="185.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.57"/></g><text class="t-tick" x="200.0" y="204.0" text-anchor="middle">-0.011</text><g class="mark"><title>-0.0020</title><rect class="s2" x="221.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.22"/></g><text class="t-tick" x="236.0" y="204.0" text-anchor="middle">-0.002</text><g class="mark"><title>-0.0116</title><rect class="s2" x="257.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.58"/></g><text class="t-tick" x="272.0" y="204.0" text-anchor="middle">-0.012</text><g class="mark"><title>-0.0217</title><rect class="s2" x="293.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.95"/></g><text class="t-tick" x="308.0" y="204.0" text-anchor="middle">-0.022</text><g class="mark"><title>-0.0101</title><rect class="s2" x="329.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.52"/></g><text class="t-tick" x="344.0" y="204.0" text-anchor="middle">-0.010</text><g class="mark"><title>0.0116</title><rect class="s1" x="365.0" y="160.0" width="30" height="30" rx="3" style="fill-opacity:0.58"/></g><text class="t-tick" x="380.0" y="204.0" text-anchor="middle">0.012</text><text class="t-muted" x="410.0" y="178.0" text-anchor="start">position table, row 2</text><text class="t-muted" x="405.0" y="194.0" text-anchor="start"></text><line class="axis" x1="185.0" y1="208.0" x2="395.0" y2="208.0"/><text class="t-big" x="170.0" y="232.0" text-anchor="middle">=</text><text class="t-note" x="160.0" y="234.0" text-anchor="end">sum</text><g class="mark"><title>-0.0258</title><rect class="s2" x="185.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.75"/></g><text class="t-tick" x="200.0" y="260.0" text-anchor="middle">-0.026</text><g class="mark"><title>0.0215</title><rect class="s1" x="221.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.65"/></g><text class="t-tick" x="236.0" y="260.0" text-anchor="middle">0.021</text><g class="mark"><title>0.0013</title><rect class="s1" x="257.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.18"/></g><text class="t-tick" x="272.0" y="260.0" text-anchor="middle">0.001</text><g class="mark"><title>-0.0345</title><rect class="s2" x="293.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.95"/></g><text class="t-tick" x="308.0" y="260.0" text-anchor="middle">-0.035</text><g class="mark"><title>-0.0336</title><rect class="s2" x="329.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.93"/></g><text class="t-tick" x="344.0" y="260.0" text-anchor="middle">-0.034</text><g class="mark"><title>-0.0082</title><rect class="s2" x="365.0" y="216.0" width="30" height="30" rx="3" style="fill-opacity:0.34"/></g><text class="t-tick" x="380.0" y="260.0" text-anchor="middle">-0.008</text><text class="t-muted" x="410.0" y="234.0" text-anchor="start">add the three</text><text class="t-muted" x="405.0" y="250.0" text-anchor="start"></text><line class="edge" x1="290.0" y1="258.0" x2="290.0" y2="278.0" marker-end="url(#ah)"/><text class="t-note" x="160.0" y="296.0" text-anchor="end">LayerNorm(sum)</text><g class="mark"><title>-0.1563</title><rect class="s2" x="185.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.34"/></g><text class="t-tick" x="200.0" y="322.0" text-anchor="middle">-0.156</text><g class="mark"><title>0.6647</title><rect class="s1" x="221.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.95"/></g><text class="t-tick" x="236.0" y="322.0" text-anchor="middle">0.665</text><g class="mark"><title>0.3523</title><rect class="s1" x="257.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.57"/></g><text class="t-tick" x="272.0" y="322.0" text-anchor="middle">0.352</text><g class="mark"><title>-0.1776</title><rect class="s2" x="293.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.36"/></g><text class="t-tick" x="308.0" y="322.0" text-anchor="middle">-0.178</text><g class="mark"><title>-0.3239</title><rect class="s2" x="329.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.54"/></g><text class="t-tick" x="344.0" y="322.0" text-anchor="middle">-0.324</text><g class="mark"><title>0.1662</title><rect class="s1" x="365.0" y="278.0" width="30" height="30" rx="3" style="fill-opacity:0.35"/></g><text class="t-tick" x="380.0" y="322.0" text-anchor="middle">0.166</text><text class="t-muted" x="410.0" y="296.0" text-anchor="start">subtract mean -0.0185, divide by 0.0517, scale, shift</text><text class="t-muted" x="405.0" y="312.0" text-anchor="start"></text><text class="t-muted" x="20.0" y="340.0" text-anchor="start">Blue: positive. Orange: negative. Darker: larger in size (each row has its own scale).</text><text class="t-muted" x="20.0" y="358.0" text-anchor="start">Lengths of the full 768-number vectors: Tok 1.069, Seg 0.897, Pos 0.494, sum 1.522, after LayerNorm 16.659.</text></svg><figcaption>The three lookups for "dog", number by number (first 6 of 768). Blue is positive, orange negative, darker is larger. They are added, then LayerNorm rescales the sum to mean 0 and spread 1 and applies the learned scale and shift.</figcaption></figure>

Notice the sizes: the three vectors are small (lengths 1.069, 0.897 and 0.494), and the sum has length 1.522. After LayerNorm the vector has length 16.659: LayerNorm does not keep vectors short, it puts every token on the same footing, whatever the size of its raw lookups.

The paper says "summing". The released model also applies **LayerNorm** after the sum (and **dropout** during training), which the paper does not mention in this paragraph. To check exactly what happens, I rebuilt the input embeddings from the model's own three tables and compared them with the model's embedding layer:

```python
E = model.embeddings
s = E.word_embeddings(ids) + E.token_type_embeddings(seg) + E.position_embeddings(pos)
mine = E.LayerNorm(s)
theirs = E(input_ids=ids, token_type_ids=seg)
```

```text
== 4. input embedding = LayerNorm(token + segment + position) ==
  tables: token (30522, 768), segment (2, 768), position (512, 768)
  output shape (1, 10, 768) (1 sequence, 10 tokens, 768 numbers each)
  max |difference|, our LayerNorm(sum) vs the model: 0.00e+00
  max |difference|, plain sum without LayerNorm vs the model: 9.15
```

> [!DEFINITION] Dropout
> During training, randomly setting some numbers to zero (10% of them in BERT) so the model cannot rely too much on any single number. It is switched off when the model is used, which is why our check (done in evaluation mode) has no dropout.

With LayerNorm, the difference is exactly zero: this is the model's embedding layer, bit for bit. Without LayerNorm, values differ by up to 9.15. So the precise rule is "sum, then LayerNorm". The three tables themselves have the shapes we used in the parameter count: (30522, 768), (2, 768) and (512, 768).

A note on positions: the original Transformer (Vaswani et al., 2017) used fixed sine and cosine patterns for positions. BERT instead **learns** its position vectors; they are ordinary weights in a table of 512 rows. Appendix A.2 says the last 10% of pre-training used length 512 "to learn the positional embeddings" (Part 3 covers this).

**Use case.** Every BERT-based system, from a spam filter to a search ranker, starts with exactly these three lookups. When an input is longer than 512 tokens, there is no position vector for token 513, so long documents must be cut into pieces. This is one of BERT's practical limits (Part 6).

## The outputs: C, Tᵢ and the pooler {§3}

After 12 layers, every token has an output vector of 768 numbers: **C** for `[CLS]`, and $$T_i$$ for token $$i$$. The paper uses C for whole-sequence tasks and the T vectors for token-level tasks.

There is one honest detail here. The paper defines C as the final hidden vector of `[CLS]`. The released code adds one more small layer on top of it, called the **pooler**: a 768 × 768 matrix and a `tanh`. Its comment in `modeling.py` says:

```python
# We "pool" the model by simply taking the hidden state corresponding
# to the first token. We assume that this has been pre-trained
first_token_tensor = tf.squeeze(self.sequence_output[:, 0:1, :], axis=1)
self.pooled_output = tf.layers.dense(first_token_tensor, config.hidden_size,
                                     activation=tf.tanh, ...)   # (initializer argument shortened)
```

That is the 590,592 parameters of the "pooler" in our count. I checked the formula on the downloaded model:

```text
== 6. the [CLS] vector C and the released pooler ==
  C = final hidden vector of [CLS], shape (768,)
  pooler = tanh(W C + b), W shape (768, 768); max |difference| vs model.pooler_output: 0.00e+00
```

> [!DEFINITION] tanh
> A smooth function that squeezes any number into the range from -1 to 1. Large positive numbers become close to 1, large negative numbers close to -1.

So in the released model, $$\text{pooled} = \tanh(W C + b)$$. In the original pre-training code, the next sentence prediction layer reads this pooled vector (`model.get_pooled_output()` in `run_pretraining.py`), and so does the classifier in `run_classifier.py`. So the pooler is trained during pre-training, through next sentence prediction (Part 3). For understanding the paper, think of it as part of the small output layer on top of C.

> [!TAKEAWAYS] Key takeaways
> - BERT builds on a long line of ideas: **word embeddings** (one fixed vector per word), **contextual embeddings** like ELMo (a vector that depends on the sentence), **fine-tuning** whole pre-trained models (ULMFiT, GPT), and the **ImageNet** habit of transfer learning.
> - In real BERT, "bank" in river sentences and in money sentences gets output vectors with cosine similarity **0.43 to 0.48**, while same-meaning pairs reach **0.75 to 0.79**. The static input row is identical everywhere.
> - The architecture is the **standard Transformer encoder**: L layers, each with A-head self-attention and a 4H-wide feed-forward network, with add-and-LayerNorm after each. Our hand-written layer matched the real one to $$8.34 \times 10^{-7}$$.
> - **BERT-base**: L=12, H=768, A=12. **BERT-large**: L=24, H=1024, A=16. BERT-base matches GPT's size; the difference is the attention mask: **encoder** (both directions) versus **decoder** (left only).
> - The parameter count, by formula and by counting the real models: **109,482,240** (base) and **335,141,888** (large); the paper rounds to 110M and 340M. About half sits in the feed-forward networks.
> - Text becomes **WordPiece** tokens with greedy longest-match; our short version matched the real tokenizer on 9,878 pieces. The "30,000" vocabulary is **30,522** entries, 994 of them unused placeholders.
> - Every input is `[CLS]` sentence A `[SEP]` (sentence B `[SEP]`). Each token's input is **token + segment + position** embeddings, then LayerNorm (exactly matched: difference 0). Positions are learned, up to **512**.
> - Outputs are **C** (for `[CLS]`) and $$T_i$$ (for each token). The released code adds a small **pooler** (dense + tanh) on top of C.

**Next, in Part 3:** how BERT is pre-trained. The masked language model with its strange 80/10/10 rule, next sentence prediction, the data, and the training recipe, all run on the real model.

<details>
<summary>Run it yourself</summary>

Every number in this part comes from [`code/papers/bert/bert_part2.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part2.py). It runs on a CPU in under a minute. It downloads `bert-base-uncased` (about 440 MB) and only the configuration file of `bert-large-uncased`. The WordPiece check reads the BERT paper PDF from `~/.cache/papers/1810.04805.pdf`, which `paper_shots.py` downloads.

```bash
pip install torch transformers pymupdf
python bert_part2.py      # prints everything below, writes results/part2.json
```

<figure><img src="/img/papers/bert/part2-run.png" alt="Terminal output of bert_part2.py: parameter counts by formula and by the real models, the vocabulary groups, WordPiece splits and the check against the real tokenizer, the Figure 2 pair with ids, segments and positions, the embedding check, the bank cosine similarities, the pooler check and the hand-computed layer" loading="lazy" /><figcaption>The real output of bert_part2.py.</figcaption></figure>

</details>

## References

**The BERT paper**

1. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805). NAACL 2019 ([ACL Anthology](https://aclanthology.org/N19-1423/)).
2. Google Research. [BERT code and pre-trained models](https://github.com/google-research/bert): `modeling.py` (the pooler), `tokenization.py` (WordPiece), `README.md`.

**Papers the BERT paper cites in this part**

3. P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. Della Pietra, J. C. Lai. [*Class-Based n-gram Models of Natural Language*](https://aclanthology.org/J92-4003/) (Brown clusters). Computational Linguistics 18(4), 1992.
4. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean. [*Distributed Representations of Words and Phrases and their Compositionality*](https://arxiv.org/abs/1310.4546) (word2vec, negative sampling). NeurIPS 2013.
5. J. Pennington, R. Socher, C. D. Manning. [*GloVe: Global Vectors for Word Representation*](https://aclanthology.org/D14-1162/). EMNLP 2014.
6. R. Kiros, Y. Zhu, R. Salakhutdinov, R. Zemel, A. Torralba, R. Urtasun, S. Fidler. [*Skip-Thought Vectors*](https://arxiv.org/abs/1506.06726). NeurIPS 2015.
7. Y. Jernite, S. R. Bowman, D. Sontag. [*Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning*](https://arxiv.org/abs/1705.00557). arXiv 2017.
8. L. Logeswaran, H. Lee. [*An efficient framework for learning sentence representations*](https://arxiv.org/abs/1803.02893) (quick-thought). ICLR 2018.
9. F. Hill, K. Cho, A. Korhonen. [*Learning Distributed Representations of Sentences from Unlabelled Data*](https://arxiv.org/abs/1602.03483) (SDAE). NAACL 2016.
10. M. E. Peters et al. [*Deep contextualized word representations*](https://arxiv.org/abs/1802.05365) (ELMo). NAACL 2018.
11. O. Melamud, J. Goldberger, I. Dagan. [*context2vec: Learning Generic Context Embedding with Bidirectional LSTM*](https://aclanthology.org/K16-1006/). CoNLL 2016.
12. W. Fedus, I. Goodfellow, A. M. Dai. [*MaskGAN: Better Text Generation via Filling in the ______*](https://arxiv.org/abs/1801.07736). ICLR 2018.
13. R. Collobert, J. Weston. [*A unified architecture for natural language processing: deep neural networks with multitask learning*](https://doi.org/10.1145/1390156.1390177). ICML 2008.
14. A. M. Dai, Q. V. Le. [*Semi-supervised Sequence Learning*](https://arxiv.org/abs/1511.01432). NeurIPS 2015.
15. J. Howard, S. Ruder. [*Universal Language Model Fine-tuning for Text Classification*](https://arxiv.org/abs/1801.06146) (ULMFiT). ACL 2018.
16. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever. [*Improving Language Understanding by Generative Pre-Training*](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) (OpenAI GPT). OpenAI, 2018.
17. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, A. Bordes. [*Supervised Learning of Universal Sentence Representations from Natural Language Inference Data*](https://arxiv.org/abs/1705.02364) (InferSent). EMNLP 2017.
18. B. McCann, J. Bradbury, C. Xiong, R. Socher. [*Learned in Translation: Contextualized Word Vectors*](https://arxiv.org/abs/1708.00107) (CoVe). NeurIPS 2017.
19. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei. [*ImageNet: A Large-Scale Hierarchical Image Database*](https://doi.org/10.1109/CVPR.2009.5206848). CVPR 2009.
20. J. Yosinski, J. Clune, Y. Bengio, H. Lipson. [*How transferable are features in deep neural networks?*](https://arxiv.org/abs/1411.1792). NeurIPS 2014.
21. A. Vaswani et al. [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762). NeurIPS 2017.
22. Harvard NLP (A. Rush). [*The Annotated Transformer*](http://nlp.seas.harvard.edu/2018/04/03/attention.html). 2018. Footnote 2 of the paper.
23. Y. Wu et al. [*Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation*](https://arxiv.org/abs/1609.08144) (WordPiece). arXiv 2016.

**Other sources used in this part**

24. T. Mikolov, K. Chen, G. Corrado, J. Dean. [*Efficient Estimation of Word Representations in Vector Space*](https://arxiv.org/abs/1301.3781) (the skip-gram figure). ICLR workshop 2013.
25. Code for this part: [`bert_part2.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part2.py) and [`bert_part2_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part2_math.py).

