# Ablations: What Really Matters

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/bert/part-5-ablations/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/bert/part-5-ablations.md


Part 4 showed that BERT wins almost everywhere. But a win does not tell you *why*. BERT changed many things at once compared with OpenAI GPT: two new pre-training tasks, bidirectional attention, more data, bigger batches, tuned learning rates. Which of these actually mattered?

This part reads the paper's answer: Section 5 (the ablation studies), plus Appendix A.4 and Appendix C, which hold the rest of the evidence. Every table row gets explained, and I check what can be checked in code.

## What an ablation study is {§5}

> [!PAPER] · §5
> [![The opening of Section 5, Ablation Studies: in this section, we perform ablation experiments over a number of facets of BERT in order to better understand their relative importance](https://ishwarj.com/img/papers/bert/p5-intro.pngs/bert/p5-intro.png)](/img/papers/bert/p5-intro.png)
>
> **What it says:** "we perform ablation experiments over a number of facets of BERT in order to better understand their relative importance." More ablations are in Appendix C.
>
> **What it means:**
>
> A car that wins a race does not tell you which part won it: the engine, the tyres or the driver. To find out, you change one part at a time and race again. That is an ablation.
>
> Section 4 showed that BERT is good. This section asks why. It turns "BERT is good" into "BERT is good *because of* these specific choices". More ablations, about training length and masking, are in Appendix C.

> [!DEFINITION] Ablation study
> An experiment where you remove (or replace) one part of a system, keep everything else the same, and measure how much worse it gets. The word comes from medicine, where ablation means removing tissue. If removing a part costs a lot, that part mattered.

> [!DEFINITION] Dev set (development set)
> A set of labeled examples kept aside from training, used to compare versions of a model and pick settings. Almost every number in this part is a Dev set score, not an official Test score like those of Part 4 (Table 7 is the one exception, and it shows both). Ablations use Dev scores partly because official test servers limit submissions: the paper made only one GLUE test submission per model (its footnote 9).

The key rule of a good ablation comes next: change **one** thing at a time.

## Effect of the pre-training tasks {§5.1}

> [!PAPER] · §5.1
> [![Section 5.1: two pre-training objectives evaluated using exactly the same pre-training data, fine-tuning scheme, and hyperparameters as BERT-base. No NSP is a bidirectional model trained with the masked LM but without next sentence prediction. LTR and No NSP is a left-context-only model trained as a standard left-to-right LM, with the left-only constraint also applied at fine-tuning, and no NSP. This is directly comparable to OpenAI GPT, but using a larger training dataset, the BERT input representation and the BERT fine-tuning scheme](https://ishwarj.com/img/papers/bert/p5-setup.pngs/bert/p5-setup.png)](/img/papers/bert/p5-setup.png)
>
> **What it says:** two new models are pre-trained "using exactly the same pre-training data, fine-tuning scheme, and hyperparameters as BERT-base". **No NSP** keeps the masked LM but drops next sentence prediction. **LTR & No NSP** is a plain left-to-right language model with no NSP, and the left-only rule is kept during fine-tuning too. That second model "is directly comparable to OpenAI GPT", except for BERT's larger data, input format and fine-tuning recipe.
>
> **What it means:**
>
> Each of these models is a full pre-training run, so each one is expensive. The authors design them as a ladder. Each model differs from the one before by exactly one change, so each change in score can be blamed on that one change.
>
> One detail deserves attention. For **LTR & No NSP**, "the left-only constraint was also applied at fine-tuning, because removing it introduced a pre-train/fine-tune mismatch that degraded downstream performance". You cannot train a model to look only left and then suddenly let it look right. It never learned what to do with that information, and it got worse, not better.
>
> The last sentence links back to Part 1. LTR & No NSP is GPT's recipe with BERT's data, input and fine-tuning. Appendix A.4, later in this part, shows why that matters.

> [!DEFINITION] Hyperparameters
> The settings you choose before training rather than learn: batch size, learning rate, number of steps, dropout rate. "Same hyperparameters" means the only difference between two runs is the one thing being tested.

> [!DEFINITION] LTR and RTL
> **LTR** means left-to-right: a language model that predicts each token from the tokens before it (Part 1). **RTL** means right-to-left: the same thing reading backwards.

<figure class="fig"><svg viewBox="0 0 760 134" role="img" aria-label="The four models of Table 5 in order. Each one changes a single thing from the one before: drop NSP, then switch to a left-to-right model, then add a BiLSTM."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="18.0" text-anchor="start">Each row of Table 5 changes one thing from the row before</text><rect class="box-1" x="14.0" y="34.0" width="132.0" height="58.0" rx="10"/><text class="t-note" x="80.0" y="59.0" text-anchor="middle">BERT-base</text><text class="t-tick" x="80.0" y="78.0" text-anchor="middle">MLM + NSP</text><line class="edge" x1="150.0" y1="63.0" x2="196.0" y2="63.0" marker-end="url(#ah)"/><text class="t-muted" x="173.0" y="52.0" text-anchor="middle">drop</text><text class="t-muted" x="173.0" y="84.0" text-anchor="middle">NSP</text><rect class="box-3" x="200.0" y="34.0" width="132.0" height="58.0" rx="10"/><text class="t-note" x="266.0" y="59.0" text-anchor="middle">No NSP</text><text class="t-tick" x="266.0" y="78.0" text-anchor="middle">MLM only</text><line class="edge" x1="336.0" y1="63.0" x2="382.0" y2="63.0" marker-end="url(#ah)"/><text class="t-muted" x="359.0" y="52.0" text-anchor="middle">MLM to</text><text class="t-muted" x="359.0" y="84.0" text-anchor="middle">LTR</text><rect class="box-2" x="386.0" y="34.0" width="132.0" height="58.0" rx="10"/><text class="t-note" x="452.0" y="59.0" text-anchor="middle">LTR &amp; No NSP</text><text class="t-tick" x="452.0" y="78.0" text-anchor="middle">left-to-right only</text><line class="edge" x1="522.0" y1="63.0" x2="568.0" y2="63.0" marker-end="url(#ah)"/><text class="t-muted" x="545.0" y="52.0" text-anchor="middle">add a</text><text class="t-muted" x="545.0" y="84.0" text-anchor="middle">BiLSTM</text><rect class="box-4" x="572.0" y="34.0" width="132.0" height="58.0" rx="10"/><text class="t-note" x="638.0" y="59.0" text-anchor="middle">+ BiLSTM</text><text class="t-tick" x="638.0" y="78.0" text-anchor="middle">LTR, BiLSTM on top</text><text class="t-tick" x="14.0" y="122.0" text-anchor="start">Same pre-training data, same fine-tuning recipe, same hyperparameters: only the named change differs.</text></svg><figcaption>The four models of Table 5. Each row changes exactly one thing from the row before, so each difference in scores can be blamed on that one change.</figcaption></figure>

What differs between the four models is easiest to see in their attention masks and pre-training heads:

<figure class="fig"><svg viewBox="0 0 760 562" role="img" aria-label="The four models of Table 5. BERT-base and No NSP let every token attend to every token; No NSP drops the next-sentence head. LTR and No NSP lets each token see only itself and the tokens to its left, like GPT. Plus BiLSTM adds a randomly initialised BiLSTM on top of that model during fine-tuning."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box-1" x="10.0" y="10.0" width="366.0" height="262.0" rx="14"/><text class="t-title" x="26.0" y="36.0" text-anchor="start">BERT-base</text><text class="t-tick" x="26.0" y="54.0" text-anchor="start">masked LM + NSP</text><text class="t-muted" x="139.0" y="84.0" text-anchor="middle"></text><text class="t-tick" x="79.0" y="102.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="109.0" y="102.0" text-anchor="middle">my</text><text class="t-tick" x="139.0" y="102.0" text-anchor="middle">dog</text><text class="t-tick" x="169.0" y="102.0" text-anchor="middle">is</text><text class="t-tick" x="199.0" y="102.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="56.0" y="129.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]</title><rect class="cell" x="65.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to my</title><rect class="cell" x="95.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to dog</title><rect class="cell" x="125.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to is</title><rect class="cell" x="155.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to [SEP]</title><rect class="cell" x="185.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="56.0" y="159.0" text-anchor="end">my</text><g class="mark"><title>my attends to [CLS]</title><rect class="cell" x="65.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to my</title><rect class="cell" x="95.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to dog</title><rect class="cell" x="125.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to is</title><rect class="cell" x="155.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to [SEP]</title><rect class="cell" x="185.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="56.0" y="189.0" text-anchor="end">dog</text><g class="mark"><title>dog attends to [CLS]</title><rect class="cell" x="65.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to my</title><rect class="cell" x="95.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to dog</title><rect class="cell" x="125.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to is</title><rect class="cell" x="155.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to [SEP]</title><rect class="cell" x="185.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="56.0" y="219.0" text-anchor="end">is</text><g class="mark"><title>is attends to [CLS]</title><rect class="cell" x="65.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to my</title><rect class="cell" x="95.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to dog</title><rect class="cell" x="125.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to is</title><rect class="cell" x="155.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to [SEP]</title><rect class="cell" x="185.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="56.0" y="249.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]</title><rect class="cell" x="65.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to my</title><rect class="cell" x="95.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to dog</title><rect class="cell" x="125.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to is</title><rect class="cell" x="155.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to [SEP]</title><rect class="cell" x="185.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="18.0" y="280.0" text-anchor="start"></text><text class="t-muted" x="26.0" y="78.0" text-anchor="start">who may look at whom</text><text class="t-muted" x="240.0" y="78.0" text-anchor="start">pre-training heads</text><rect class="box-on" x="240.0" y="96.0" width="120.0" height="26.0" rx="7"/><text class="t-note" x="300.0" y="114.0" text-anchor="middle">MLM</text><rect class="box-on" x="240.0" y="130.0" width="120.0" height="26.0" rx="7"/><text class="t-note" x="300.0" y="148.0" text-anchor="middle">NSP on C</text><rect class="box-1" x="390.0" y="10.0" width="366.0" height="262.0" rx="14"/><text class="t-title" x="406.0" y="36.0" text-anchor="start">No NSP</text><text class="t-tick" x="406.0" y="54.0" text-anchor="start">masked LM only</text><text class="t-muted" x="519.0" y="84.0" text-anchor="middle"></text><text class="t-tick" x="459.0" y="102.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="489.0" y="102.0" text-anchor="middle">my</text><text class="t-tick" x="519.0" y="102.0" text-anchor="middle">dog</text><text class="t-tick" x="549.0" y="102.0" text-anchor="middle">is</text><text class="t-tick" x="579.0" y="102.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="436.0" y="129.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]</title><rect class="cell" x="445.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to my</title><rect class="cell" x="475.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to dog</title><rect class="cell" x="505.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to is</title><rect class="cell" x="535.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[CLS] attends to [SEP]</title><rect class="cell" x="565.5" y="111.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="436.0" y="159.0" text-anchor="end">my</text><g class="mark"><title>my attends to [CLS]</title><rect class="cell" x="445.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to my</title><rect class="cell" x="475.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to dog</title><rect class="cell" x="505.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to is</title><rect class="cell" x="535.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to [SEP]</title><rect class="cell" x="565.5" y="141.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="436.0" y="189.0" text-anchor="end">dog</text><g class="mark"><title>dog attends to [CLS]</title><rect class="cell" x="445.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to my</title><rect class="cell" x="475.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to dog</title><rect class="cell" x="505.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to is</title><rect class="cell" x="535.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to [SEP]</title><rect class="cell" x="565.5" y="171.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="436.0" y="219.0" text-anchor="end">is</text><g class="mark"><title>is attends to [CLS]</title><rect class="cell" x="445.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to my</title><rect class="cell" x="475.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to dog</title><rect class="cell" x="505.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to is</title><rect class="cell" x="535.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to [SEP]</title><rect class="cell" x="565.5" y="201.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="436.0" y="249.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]</title><rect class="cell" x="445.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to my</title><rect class="cell" x="475.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to dog</title><rect class="cell" x="505.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to is</title><rect class="cell" x="535.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to [SEP]</title><rect class="cell" x="565.5" y="231.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="398.0" y="280.0" text-anchor="start"></text><text class="t-muted" x="406.0" y="78.0" text-anchor="start">who may look at whom</text><text class="t-muted" x="620.0" y="78.0" text-anchor="start">pre-training heads</text><rect class="box-on" x="620.0" y="96.0" width="120.0" height="26.0" rx="7"/><text class="t-note" x="680.0" y="114.0" text-anchor="middle">MLM</text><rect class="box-ghost" x="620.0" y="130.0" width="120.0" height="26.0" rx="7"/><text class="t-muted" x="680.0" y="148.0" text-anchor="middle">NSP removed</text><line class="bad-line" x1="626.0" y1="143.0" x2="734.0" y2="143.0"/><rect class="box-2" x="10.0" y="286.0" width="366.0" height="262.0" rx="14"/><text class="t-title" x="26.0" y="312.0" text-anchor="start">LTR &amp; No NSP</text><text class="t-tick" x="26.0" y="330.0" text-anchor="start">left-to-right LM, like GPT</text><text class="t-muted" x="139.0" y="360.0" text-anchor="middle"></text><text class="t-tick" x="79.0" y="378.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="109.0" y="378.0" text-anchor="middle">my</text><text class="t-tick" x="139.0" y="378.0" text-anchor="middle">dog</text><text class="t-tick" x="169.0" y="378.0" text-anchor="middle">is</text><text class="t-tick" x="199.0" y="378.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="56.0" y="405.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]</title><rect class="cell" x="65.5" y="387.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="95.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="103.6" y1="395.6" x2="114.4" y2="406.4"/><line class="edge" x1="114.4" y1="395.6" x2="103.6" y2="406.4"/><rect class="cell-masked" x="125.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="133.6" y1="395.6" x2="144.4" y2="406.4"/><line class="edge" x1="144.4" y1="395.6" x2="133.6" y2="406.4"/><rect class="cell-masked" x="155.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="163.6" y1="395.6" x2="174.4" y2="406.4"/><line class="edge" x1="174.4" y1="395.6" x2="163.6" y2="406.4"/><rect class="cell-masked" x="185.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="193.6" y1="395.6" x2="204.4" y2="406.4"/><line class="edge" x1="204.4" y1="395.6" x2="193.6" y2="406.4"/><text class="t-tick" x="56.0" y="435.0" text-anchor="end">my</text><g class="mark"><title>my attends to [CLS]</title><rect class="cell" x="65.5" y="417.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to my</title><rect class="cell" x="95.5" y="417.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="125.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="133.6" y1="425.6" x2="144.4" y2="436.4"/><line class="edge" x1="144.4" y1="425.6" x2="133.6" y2="436.4"/><rect class="cell-masked" x="155.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="163.6" y1="425.6" x2="174.4" y2="436.4"/><line class="edge" x1="174.4" y1="425.6" x2="163.6" y2="436.4"/><rect class="cell-masked" x="185.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="193.6" y1="425.6" x2="204.4" y2="436.4"/><line class="edge" x1="204.4" y1="425.6" x2="193.6" y2="436.4"/><text class="t-tick" x="56.0" y="465.0" text-anchor="end">dog</text><g class="mark"><title>dog attends to [CLS]</title><rect class="cell" x="65.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to my</title><rect class="cell" x="95.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to dog</title><rect class="cell" x="125.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="155.5" y="447.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="163.6" y1="455.6" x2="174.4" y2="466.4"/><line class="edge" x1="174.4" y1="455.6" x2="163.6" y2="466.4"/><rect class="cell-masked" x="185.5" y="447.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="193.6" y1="455.6" x2="204.4" y2="466.4"/><line class="edge" x1="204.4" y1="455.6" x2="193.6" y2="466.4"/><text class="t-tick" x="56.0" y="495.0" text-anchor="end">is</text><g class="mark"><title>is attends to [CLS]</title><rect class="cell" x="65.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to my</title><rect class="cell" x="95.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to dog</title><rect class="cell" x="125.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to is</title><rect class="cell" x="155.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="185.5" y="477.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="193.6" y1="485.6" x2="204.4" y2="496.4"/><line class="edge" x1="204.4" y1="485.6" x2="193.6" y2="496.4"/><text class="t-tick" x="56.0" y="525.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]</title><rect class="cell" x="65.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to my</title><rect class="cell" x="95.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to dog</title><rect class="cell" x="125.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to is</title><rect class="cell" x="155.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to [SEP]</title><rect class="cell" x="185.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="18.0" y="556.0" text-anchor="start"></text><text class="t-muted" x="26.0" y="354.0" text-anchor="start">who may look at whom</text><text class="t-muted" x="240.0" y="354.0" text-anchor="start">pre-training heads</text><rect class="box-on" x="240.0" y="372.0" width="120.0" height="26.0" rx="7"/><text class="t-note" x="300.0" y="390.0" text-anchor="middle">next word</text><text class="t-muted" x="240.0" y="496.0" text-anchor="start">left-only mask kept</text><text class="t-muted" x="240.0" y="512.0" text-anchor="start">during fine-tuning too</text><rect class="box-2" x="390.0" y="286.0" width="366.0" height="262.0" rx="14"/><text class="t-title" x="406.0" y="312.0" text-anchor="start">+ BiLSTM</text><text class="t-tick" x="406.0" y="330.0" text-anchor="start">the LTR model + a BiLSTM</text><text class="t-muted" x="519.0" y="360.0" text-anchor="middle"></text><text class="t-tick" x="459.0" y="378.0" text-anchor="middle">[CLS]</text><text class="t-tick" x="489.0" y="378.0" text-anchor="middle">my</text><text class="t-tick" x="519.0" y="378.0" text-anchor="middle">dog</text><text class="t-tick" x="549.0" y="378.0" text-anchor="middle">is</text><text class="t-tick" x="579.0" y="378.0" text-anchor="middle">[SEP]</text><text class="t-tick" x="436.0" y="405.0" text-anchor="end">[CLS]</text><g class="mark"><title>[CLS] attends to [CLS]</title><rect class="cell" x="445.5" y="387.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="475.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="483.6" y1="395.6" x2="494.4" y2="406.4"/><line class="edge" x1="494.4" y1="395.6" x2="483.6" y2="406.4"/><rect class="cell-masked" x="505.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="513.6" y1="395.6" x2="524.4" y2="406.4"/><line class="edge" x1="524.4" y1="395.6" x2="513.6" y2="406.4"/><rect class="cell-masked" x="535.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="543.6" y1="395.6" x2="554.4" y2="406.4"/><line class="edge" x1="554.4" y1="395.6" x2="543.6" y2="406.4"/><rect class="cell-masked" x="565.5" y="387.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="573.6" y1="395.6" x2="584.4" y2="406.4"/><line class="edge" x1="584.4" y1="395.6" x2="573.6" y2="406.4"/><text class="t-tick" x="436.0" y="435.0" text-anchor="end">my</text><g class="mark"><title>my attends to [CLS]</title><rect class="cell" x="445.5" y="417.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>my attends to my</title><rect class="cell" x="475.5" y="417.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="505.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="513.6" y1="425.6" x2="524.4" y2="436.4"/><line class="edge" x1="524.4" y1="425.6" x2="513.6" y2="436.4"/><rect class="cell-masked" x="535.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="543.6" y1="425.6" x2="554.4" y2="436.4"/><line class="edge" x1="554.4" y1="425.6" x2="543.6" y2="436.4"/><rect class="cell-masked" x="565.5" y="417.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="573.6" y1="425.6" x2="584.4" y2="436.4"/><line class="edge" x1="584.4" y1="425.6" x2="573.6" y2="436.4"/><text class="t-tick" x="436.0" y="465.0" text-anchor="end">dog</text><g class="mark"><title>dog attends to [CLS]</title><rect class="cell" x="445.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to my</title><rect class="cell" x="475.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>dog attends to dog</title><rect class="cell" x="505.5" y="447.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="535.5" y="447.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="543.6" y1="455.6" x2="554.4" y2="466.4"/><line class="edge" x1="554.4" y1="455.6" x2="543.6" y2="466.4"/><rect class="cell-masked" x="565.5" y="447.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="573.6" y1="455.6" x2="584.4" y2="466.4"/><line class="edge" x1="584.4" y1="455.6" x2="573.6" y2="466.4"/><text class="t-tick" x="436.0" y="495.0" text-anchor="end">is</text><g class="mark"><title>is attends to [CLS]</title><rect class="cell" x="445.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to my</title><rect class="cell" x="475.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to dog</title><rect class="cell" x="505.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>is attends to is</title><rect class="cell" x="535.5" y="477.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="565.5" y="477.5" width="27.0" height="27.0" rx="3"/><line class="edge" x1="573.6" y1="485.6" x2="584.4" y2="496.4"/><line class="edge" x1="584.4" y1="485.6" x2="573.6" y2="496.4"/><text class="t-tick" x="436.0" y="525.0" text-anchor="end">[SEP]</text><g class="mark"><title>[SEP] attends to [CLS]</title><rect class="cell" x="445.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to my</title><rect class="cell" x="475.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to dog</title><rect class="cell" x="505.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to is</title><rect class="cell" x="535.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>[SEP] attends to [SEP]</title><rect class="cell" x="565.5" y="507.5" width="27.0" height="27.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="398.0" y="556.0" text-anchor="start"></text><text class="t-muted" x="406.0" y="354.0" text-anchor="start">who may look at whom</text><text class="t-muted" x="620.0" y="354.0" text-anchor="start">pre-training heads</text><rect class="box-on" x="620.0" y="372.0" width="120.0" height="26.0" rx="7"/><text class="t-note" x="680.0" y="390.0" text-anchor="middle">next word</text><rect class="box-4" x="620.0" y="414.0" width="120.0" height="44.0" rx="7"/><text class="t-note" x="680.0" y="433.0" text-anchor="middle">BiLSTM</text><text class="t-muted" x="680.0" y="449.0" text-anchor="middle">new, random start</text><text class="t-muted" x="620.0" y="496.0" text-anchor="start">left-only mask kept</text><text class="t-muted" x="620.0" y="512.0" text-anchor="start">during fine-tuning too</text></svg><figcaption>The four models of Table 5, side by side. BERT-base: no mask, MLM and NSP heads. No NSP: no mask, MLM head only. LTR & No NSP: a left-to-right mask (the upper triangle blocked, like GPT), next-word head, and the mask is kept during fine-tuning too. + BiLSTM: the LTR model with a new, randomly initialised BiLSTM added for fine-tuning.</figcaption></figure>

> [!DEFINITION] BiLSTM
> A bidirectional LSTM: two LSTMs over the same sequence, one left to right and one right to left, with their outputs joined at each position. The same idea as ELMo, but here trained from zero during fine-tuning.

### Table 5, row by row

> [!PAPER] · §5.1, Table 5
> [![Table 5 of the BERT paper: Dev set results on MNLI-m, QNLI, MRPC, SST-2 (accuracy) and SQuAD (F1). BERT-base 84.4, 88.4, 86.7, 92.7, 88.5. No NSP 83.9, 84.9, 86.5, 92.6, 87.9. LTR and No NSP 82.1, 84.3, 77.5, 92.1, 77.8. Plus BiLSTM 82.1, 84.1, 75.7, 91.6, 84.9](https://ishwarj.com/img/papers/bert/p5-table5.png/bert/p5-table5.png)](/img/papers/bert/p5-table5.png)
>
> **What it says:** four rows (BERT-base, No NSP, LTR & No NSP, + BiLSTM) and five tasks. "+ BiLSTM" adds "a randomly initialized BiLSTM on top of the 'LTR + No NSP' model during fine-tuning".
>
> **What it means:**
>
> This table is the main evidence for the paper's central claim: bidirectionality is what makes BERT work. Read it as a ladder, from top to bottom. Each row removes or changes one thing, so the change in score between two rows is the price of that one thing.
>
> The last row is a rescue attempt. It keeps the left-to-right model and adds a BiLSTM on top. It asks whether a little right-side context, added late, can replace real bidirectionality. The differences below give the answer.

The five columns are tasks from Part 4:

| Column | Task | Score |
|---|---|---|
| MNLI-m | does sentence B follow from A, contradict it, or neither? ("m" = matched: test sentences from the same genres as training) | accuracy |
| QNLI | does this sentence contain the answer to this question? | accuracy |
| MRPC | do these two news sentences mean the same thing? | accuracy |
| SST-2 | is this movie-review sentence positive or negative? | accuracy |
| SQuAD | find the answer span in a paragraph (v1.1) | F1 |

> [!DEFINITION] BiLSTM
> A bidirectional LSTM: two LSTMs (Part 1), one reading the tokens left to right and one right to left, with their outputs joined for every token. Placed on top of another model, it adds some right-side context, but only in its own small layer.

Numbers in a table are easier to compare as differences. My script `bert_part5.py` subtracts each row from the row before (the Table 5 values are typed in from the paper):

```text
Table 5 (Dev set): change in points
                                                MNLI-m    QNLI    MRPC   SST-2   SQuAD
cost of removing NSP                              -0.5    -3.5    -0.2    -0.1    -0.6
cost of left-to-right instead of MLM              -1.8    -0.6    -9.0    -0.5   -10.1
effect of adding a BiLSTM to the LTR model        +0.0    -0.2    -1.8    -0.5    +7.1
```

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="Table 5 of the BERT paper as a bar chart: Dev set scores of BERT-base, No NSP, LTR and No NSP, and LTR plus BiLSTM on five tasks. The axis starts at 70."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="56" y1="290.0" x2="750" y2="290.0"/><text class="t-tick" x="48.0" y="294.0" text-anchor="end">70</text><line class="grid" x1="56" y1="242.0" x2="750" y2="242.0"/><text class="t-tick" x="48.0" y="246.0" text-anchor="end">75</text><line class="grid" x1="56" y1="194.0" x2="750" y2="194.0"/><text class="t-tick" x="48.0" y="198.0" text-anchor="end">80</text><line class="grid" x1="56" y1="146.0" x2="750" y2="146.0"/><text class="t-tick" x="48.0" y="150.0" text-anchor="end">85</text><line class="grid" x1="56" y1="98.0" x2="750" y2="98.0"/><text class="t-tick" x="48.0" y="102.0" text-anchor="end">90</text><line class="grid" x1="56" y1="50.0" x2="750" y2="50.0"/><text class="t-tick" x="48.0" y="54.0" text-anchor="end">95</text><g class="mark"><title>BERT-base, MNLI-m: 84.4</title><rect class="s1" x="67.4" y="151.8" width="26" height="138.2" rx="3"/></g><text class="t-cell" x="80.4" y="146.8" text-anchor="middle">84.4</text><g class="mark"><title>No NSP, MNLI-m: 83.9</title><rect class="s3" x="97.4" y="156.6" width="26" height="133.4" rx="3"/></g><text class="t-cell" x="110.4" y="151.6" text-anchor="middle">83.9</text><g class="mark"><title>LTR &amp; No NSP, MNLI-m: 82.1</title><rect class="s2" x="127.4" y="173.8" width="26" height="116.2" rx="3"/></g><text class="t-cell" x="140.4" y="168.8" text-anchor="middle">82.1</text><g class="mark"><title>+ BiLSTM, MNLI-m: 82.1</title><rect class="s4" x="157.4" y="173.8" width="26" height="116.2" rx="3"/></g><text class="t-cell" x="170.4" y="168.8" text-anchor="middle">82.1</text><text class="t-tick" x="125.4" y="316.0" text-anchor="middle">MNLI-m</text><g class="mark"><title>BERT-base, QNLI: 88.4</title><rect class="s1" x="206.2" y="113.4" width="26" height="176.6" rx="3"/></g><text class="t-cell" x="219.2" y="108.4" text-anchor="middle">88.4</text><g class="mark"><title>No NSP, QNLI: 84.9</title><rect class="s3" x="236.2" y="147.0" width="26" height="143.0" rx="3"/></g><text class="t-cell" x="249.2" y="142.0" text-anchor="middle">84.9</text><g class="mark"><title>LTR &amp; No NSP, QNLI: 84.3</title><rect class="s2" x="266.2" y="152.7" width="26" height="137.3" rx="3"/></g><text class="t-cell" x="279.2" y="147.7" text-anchor="middle">84.3</text><g class="mark"><title>+ BiLSTM, QNLI: 84.1</title><rect class="s4" x="296.2" y="154.6" width="26" height="135.4" rx="3"/></g><text class="t-cell" x="309.2" y="149.6" text-anchor="middle">84.1</text><text class="t-tick" x="264.2" y="316.0" text-anchor="middle">QNLI</text><g class="mark"><title>BERT-base, MRPC: 86.7</title><rect class="s1" x="345.0" y="129.7" width="26" height="160.3" rx="3"/></g><text class="t-cell" x="358.0" y="124.7" text-anchor="middle">86.7</text><g class="mark"><title>No NSP, MRPC: 86.5</title><rect class="s3" x="375.0" y="131.6" width="26" height="158.4" rx="3"/></g><text class="t-cell" x="388.0" y="126.6" text-anchor="middle">86.5</text><g class="mark"><title>LTR &amp; No NSP, MRPC: 77.5</title><rect class="s2" x="405.0" y="218.0" width="26" height="72.0" rx="3"/></g><text class="t-cell" x="418.0" y="213.0" text-anchor="middle">77.5</text><g class="mark"><title>+ BiLSTM, MRPC: 75.7</title><rect class="s4" x="435.0" y="235.3" width="26" height="54.7" rx="3"/></g><text class="t-cell" x="448.0" y="230.3" text-anchor="middle">75.7</text><text class="t-tick" x="403.0" y="316.0" text-anchor="middle">MRPC</text><g class="mark"><title>BERT-base, SST-2: 92.7</title><rect class="s1" x="483.8" y="72.1" width="26" height="217.9" rx="3"/></g><text class="t-cell" x="496.8" y="67.1" text-anchor="middle">92.7</text><g class="mark"><title>No NSP, SST-2: 92.6</title><rect class="s3" x="513.8" y="73.0" width="26" height="217.0" rx="3"/></g><text class="t-cell" x="526.8" y="68.0" text-anchor="middle">92.6</text><g class="mark"><title>LTR &amp; No NSP, SST-2: 92.1</title><rect class="s2" x="543.8" y="77.8" width="26" height="212.2" rx="3"/></g><text class="t-cell" x="556.8" y="72.8" text-anchor="middle">92.1</text><g class="mark"><title>+ BiLSTM, SST-2: 91.6</title><rect class="s4" x="573.8" y="82.6" width="26" height="207.4" rx="3"/></g><text class="t-cell" x="586.8" y="77.6" text-anchor="middle">91.6</text><text class="t-tick" x="541.8" y="316.0" text-anchor="middle">SST-2</text><g class="mark"><title>BERT-base, SQuAD F1: 88.5</title><rect class="s1" x="622.6" y="112.4" width="26" height="177.6" rx="3"/></g><text class="t-cell" x="635.6" y="107.4" text-anchor="middle">88.5</text><g class="mark"><title>No NSP, SQuAD F1: 87.9</title><rect class="s3" x="652.6" y="118.2" width="26" height="171.8" rx="3"/></g><text class="t-cell" x="665.6" y="113.2" text-anchor="middle">87.9</text><g class="mark"><title>LTR &amp; No NSP, SQuAD F1: 77.8</title><rect class="s2" x="682.6" y="215.1" width="26" height="74.9" rx="3"/></g><text class="t-cell" x="695.6" y="210.1" text-anchor="middle">77.8</text><g class="mark"><title>+ BiLSTM, SQuAD F1: 84.9</title><rect class="s4" x="712.6" y="147.0" width="26" height="143.0" rx="3"/></g><text class="t-cell" x="725.6" y="142.0" text-anchor="middle">84.9</text><text class="t-tick" x="680.6" y="316.0" text-anchor="middle">SQuAD F1</text><rect class="s1" x="56" y="14" width="12" height="12" rx="2"/><text class="t-note" x="74.0" y="24.0" text-anchor="start">BERT-base</text><rect class="s3" x="226" y="14" width="12" height="12" rx="2"/><text class="t-note" x="244.0" y="24.0" text-anchor="start">No NSP</text><rect class="s2" x="396" y="14" width="12" height="12" rx="2"/><text class="t-note" x="414.0" y="24.0" text-anchor="start">LTR &amp; No NSP</text><rect class="s4" x="566" y="14" width="12" height="12" rx="2"/><text class="t-note" x="584.0" y="24.0" text-anchor="start">+ BiLSTM</text><line class="axis" x1="56" y1="290.0" x2="750" y2="290.0"/></svg><figcaption>Table 5 as a chart (Dev set; the axis starts at 70 so the gaps are visible). Removing NSP mostly hurts QNLI. Switching to left-to-right crashes MRPC and SQuAD. The BiLSTM rescues part of SQuAD but nothing else.</figcaption></figure>

**Row 1 to row 2: removing NSP.**

> [!PAPER] · §5.1
> [![We first examine the impact brought by the NSP task. In Table 5, we show that removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1. The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD](https://ishwarj.com/img/papers/bert/p5-nsp-ltr.pngbert/p5-nsp-ltr.png)](/img/papers/bert/p5-nsp-ltr.png)
>
> **What it says:** "removing NSP hurts performance significantly on QNLI, MNLI, and SQuAD 1.1." And comparing No NSP with LTR & No NSP: "The LTR model performs worse than the MLM model on all tasks, with large drops on MRPC and SQuAD."
>
> **What it means:**
>
> The paragraph makes two separate claims, and they rest on different rows. The first compares BERT-base with No NSP: next sentence prediction helps. The second compares No NSP with LTR & No NSP: reading in both directions helps, and it helps a lot.
>
> Keep the two apart, because they do not have the same strength. The differences below show that the second claim is much stronger than the first.

Looking at the computed differences, the NSP claim is weaker than the word "significantly" suggests. QNLI really drops (**3.5** points), but MNLI drops **0.5** and SQuAD **0.6**, and MRPC and SST-2 barely move (0.2 and 0.1). The paper does not report how much these Dev scores vary between runs, so we cannot tell how much of a 0.5-point gap is noise. Keep this in mind: in Part 6 we will see that later work (RoBERTa) questioned whether NSP is needed at all.

> [!NOTE] Why QNLI might care about NSP
> QNLI asks whether a sentence answers a question: a judgement about how two pieces of text relate, which is exactly what NSP practises (Part 3). That fits the drop, but the paper does not test this explanation directly.

**Row 2 to row 3: one direction instead of two.** This is the big one. With everything else equal, going from the masked LM to a left-to-right LM costs **9.0** points on MRPC and **10.1** points of F1 on SQuAD, and loses something on every task. This is the cleanest evidence in the paper that the bidirectional design, not just more data or a bigger batch, explains much of BERT's gain.

> [!PAPER] · §5.1
> [![For SQuAD it is intuitively clear that a LTR model will perform poorly at token predictions, since the token-level hidden states have no right-side context. In order to make a good faith attempt at strengthening the LTR system, we added a randomly initialized BiLSTM on top](https://ishwarj.com/img/papers/bert/p5-bilstm.png/bert/p5-bilstm.png)](/img/papers/bert/p5-bilstm.png)
> [![The paragraph ends at the top of the next column: results are still far worse than those of the pre-trained bidirectional models. The BiLSTM hurts performance on the GLUE tasks](https://ishwarj.com/img/papers/bert/p5-bilstm2.pngbert/p5-bilstm2.png)](/img/papers/bert/p5-bilstm2.png)
>
> **What it says:** an LTR model does badly on SQuAD "since the token-level hidden states have no right-side context". As "a good faith attempt" to help it, they add "a randomly initialized BiLSTM on top". It "does significantly improve results on SQuAD", but they stay "far worse" than the bidirectional models, and "the BiLSTM hurts performance on the GLUE tasks".
>
> **What it means:**
>
> A reader might object: "just add some right context on top of the left-to-right model". The authors try exactly that, in "good faith". The BiLSTM reads the finished vectors in both directions, so each token gets some right-side context at the end.
>
> It is like asking a student who read a text only forwards to look back once at the end. That helps on SQuAD, where the end of an answer depends on the words after it. But it is not the same as reading both ways in every layer, and on GLUE it even hurts.

Why does SQuAD suffer most? SQuAD needs a start and an end position for the answer (Part 4). Whether a word is the *end* of an answer depends heavily on the words after it. In an LTR model, the vector for each token was built without ever seeing those words. The BiLSTM adds right-side context, which is why SQuAD jumps by **7.1** points (77.8 to 84.9). But it is one small layer trained only on the task data, while BERT mixes both sides in all 12 layers during pre-training. The result is still **3.6** points below BERT-base (84.9 versus 88.5).

<figure class="fig"><svg viewBox="0 0 760 338" role="img" aria-label="What the vector of the answer word leonardo can see. In a left-to-right model it sees only the tokens up to itself, so it cannot know that da vinci follows. In BERT it sees the whole question and passage, in every layer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">Question: who painted it?   Passage: the mona lisa was painted by leonardo da vinci</text><text class="t-note" x="14.0" y="60.0" text-anchor="start">Left-to-right model</text><rect class="box" x="14.0" y="72.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="36.5" y="91.0" text-anchor="middle">[CLS]</text><path class="edge" d="M36.5,104.0 Q293.2,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="62.0" y="72.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="77.5" y="91.0" text-anchor="middle">who</text><path class="edge" d="M77.5,104.0 Q313.8,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="96.0" y="72.0" width="59.0" height="30.0" rx="5"/><text class="t-tick" x="125.5" y="91.0" text-anchor="middle">painted</text><path class="edge" d="M125.5,104.0 Q337.8,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="158.0" y="72.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="173.0" y="91.0" text-anchor="middle">it</text><path class="edge" d="M173.0,104.0 Q361.5,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="191.0" y="72.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="206.0" y="91.0" text-anchor="middle">?</text><path class="edge" d="M206.0,104.0 Q378.0,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="224.0" y="72.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="246.5" y="91.0" text-anchor="middle">[SEP]</text><path class="edge" d="M246.5,104.0 Q398.2,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="272.0" y="72.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="287.5" y="91.0" text-anchor="middle">the</text><path class="edge" d="M287.5,104.0 Q418.8,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="306.0" y="72.0" width="38.0" height="30.0" rx="5"/><text class="t-tick" x="325.0" y="91.0" text-anchor="middle">mona</text><path class="edge" d="M325.0,104.0 Q437.5,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="347.0" y="72.0" width="38.0" height="30.0" rx="5"/><text class="t-tick" x="366.0" y="91.0" text-anchor="middle">lisa</text><path class="edge" d="M366.0,104.0 Q458.0,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="388.0" y="72.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="403.5" y="91.0" text-anchor="middle">was</text><path class="edge" d="M403.5,104.0 Q476.8,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="422.0" y="72.0" width="59.0" height="30.0" rx="5"/><text class="t-tick" x="451.5" y="91.0" text-anchor="middle">painted</text><path class="edge" d="M451.5,104.0 Q500.8,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box" x="484.0" y="72.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="499.0" y="91.0" text-anchor="middle">by</text><path class="edge" d="M499.0,104.0 Q524.5,144.0 550.0,106.0" style="stroke-opacity:0.7"/><rect class="box-on" x="517.0" y="72.0" width="66.0" height="30.0" rx="5"/><text class="t-tick" x="550.0" y="91.0" text-anchor="middle">leonardo</text><rect class="box-ghost" x="586.0" y="72.0" width="30.0" height="30.0" rx="5"/><text class="t-muted" x="601.0" y="91.0" text-anchor="middle">da</text><rect class="box-ghost" x="619.0" y="72.0" width="45.0" height="30.0" rx="5"/><text class="t-muted" x="641.5" y="91.0" text-anchor="middle">vinci</text><rect class="box-ghost" x="667.0" y="72.0" width="45.0" height="30.0" rx="5"/><text class="t-muted" x="689.5" y="91.0" text-anchor="middle">[SEP]</text><path class="edge" d="M586.0,108.0 L586.0,116.0 L712.0,116.0 L712.0,108.0"/><text class="t-tick" x="649.0" y="134.0" text-anchor="middle">hidden from "leonardo"</text><text class="t-note" x="14.0" y="180.0" text-anchor="start">Bidirectional model (BERT)</text><rect class="box" x="14.0" y="192.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="36.5" y="211.0" text-anchor="middle">[CLS]</text><path class="edge" d="M36.5,224.0 Q293.2,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="62.0" y="192.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="77.5" y="211.0" text-anchor="middle">who</text><path class="edge" d="M77.5,224.0 Q313.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="96.0" y="192.0" width="59.0" height="30.0" rx="5"/><text class="t-tick" x="125.5" y="211.0" text-anchor="middle">painted</text><path class="edge" d="M125.5,224.0 Q337.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="158.0" y="192.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="173.0" y="211.0" text-anchor="middle">it</text><path class="edge" d="M173.0,224.0 Q361.5,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="191.0" y="192.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="206.0" y="211.0" text-anchor="middle">?</text><path class="edge" d="M206.0,224.0 Q378.0,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="224.0" y="192.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="246.5" y="211.0" text-anchor="middle">[SEP]</text><path class="edge" d="M246.5,224.0 Q398.2,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="272.0" y="192.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="287.5" y="211.0" text-anchor="middle">the</text><path class="edge" d="M287.5,224.0 Q418.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="306.0" y="192.0" width="38.0" height="30.0" rx="5"/><text class="t-tick" x="325.0" y="211.0" text-anchor="middle">mona</text><path class="edge" d="M325.0,224.0 Q437.5,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="347.0" y="192.0" width="38.0" height="30.0" rx="5"/><text class="t-tick" x="366.0" y="211.0" text-anchor="middle">lisa</text><path class="edge" d="M366.0,224.0 Q458.0,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="388.0" y="192.0" width="31.0" height="30.0" rx="5"/><text class="t-tick" x="403.5" y="211.0" text-anchor="middle">was</text><path class="edge" d="M403.5,224.0 Q476.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="422.0" y="192.0" width="59.0" height="30.0" rx="5"/><text class="t-tick" x="451.5" y="211.0" text-anchor="middle">painted</text><path class="edge" d="M451.5,224.0 Q500.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="484.0" y="192.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="499.0" y="211.0" text-anchor="middle">by</text><path class="edge" d="M499.0,224.0 Q524.5,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box-on" x="517.0" y="192.0" width="66.0" height="30.0" rx="5"/><text class="t-tick" x="550.0" y="211.0" text-anchor="middle">leonardo</text><rect class="box-3" x="586.0" y="192.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="601.0" y="211.0" text-anchor="middle">da</text><path class="edge-on" d="M601.0,224.0 Q575.5,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box-3" x="619.0" y="192.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="641.5" y="211.0" text-anchor="middle">vinci</text><path class="edge-on" d="M641.5,224.0 Q595.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><rect class="box" x="667.0" y="192.0" width="45.0" height="30.0" rx="5"/><text class="t-tick" x="689.5" y="211.0" text-anchor="middle">[SEP]</text><path class="edge-on" d="M689.5,224.0 Q619.8,264.0 550.0,226.0" style="stroke-opacity:0.7"/><text class="t-muted" x="14.0" y="280.0" text-anchor="start">Teal arrows: right-side words. Only BERT can use them.</text><text class="t-muted" x="14.0" y="306.0" text-anchor="start">Without "da vinci", the vector of "leonardo" cannot tell whether a longer name follows, or where the answer should end.</text><text class="t-muted" x="14.0" y="326.0" text-anchor="start">The highlighted box is the token whose vector must say "the answer starts here". Green: the true answer span.</text></svg><figcaption>What the vector of the answer's first word can see. Question "who painted it?", passage "the mona lisa was painted by leonardo da vinci". In a left-to-right model, "leonardo" sees only what came before it; "da vinci" is hidden, so its vector cannot say where the answer ends. In BERT it sees both sides, in every layer.</figcaption></figure>

On the GLUE tasks the BiLSTM does not help at all (MNLI unchanged, the others down by 0.2 to 1.8 points). A plausible reason, not tested in the paper: a randomly initialized layer has to be learned from small task datasets, and MRPC has only a few thousand training pairs (3,600, according to Section 5.2).

### Why not just do what ELMo does?

> [!PAPER] · §5.1
> [![We recognize that it would also be possible to train separate LTR and RTL models and represent each token as the concatenation of the two models, as ELMo does. However: (a) this is twice as expensive as a single bidirectional model; (b) this is non-intuitive for tasks like QA, since the RTL model would not be able to condition the answer on the question; (c) this it is strictly less powerful than a deep bidirectional model, since it can use both left and right context at every layer](https://ishwarj.com/img/papers/bert/p5-elmo-args.pngrt/p5-elmo-args.png)](/img/papers/bert/p5-elmo-args.png)
>
> **What it says:** you could train a separate LTR and RTL model and glue their vectors together, "as ELMo does". Three reasons against: (a) "twice as expensive"; (b) "non-intuitive for tasks like QA, since the RTL model would not be able to condition the answer on the question"; (c) "strictly less powerful than a deep bidirectional model, since it can use both left and right context at every layer".
>
> **What it means:**
>
> ELMo already used both directions, so why not just build a bigger ELMo? Here the paper does not run an experiment; it argues instead. The three reasons are cost, question answering and depth.
>
> The key word is "strictly" in reason (c). ELMo is like two people who each read the story in one direction and compare notes only at the end. In BERT, the two directions meet in every layer. So the glued model can do nothing that the deep model cannot also do.

Each argument in turn:

- **(a) Cost.** Two full models to pre-train and to run, instead of one.
- **(b) Question answering.** In BERT's input the question comes first and the paragraph second (Part 4). A right-to-left model reading the paragraph has not reached the question yet (the question is to its left), so its vectors for the paragraph words know nothing about what is being asked.
- **(c) Depth.** In ELMo the two directions meet only at the very end. In BERT, every layer mixes both sides, so later layers can build on information that already combines left and right. A small grammar slip in the paper can confuse readers here: in "this it is strictly less powerful ... since **it** can use both left and right context at every layer", the second "it" means the deep bidirectional model, not the concatenation.

<figure class="fig"><svg viewBox="0 0 760 236" role="img" aria-label="Attention from passage tokens to question tokens. A left-to-right model reads the question before the passage, so passage tokens can use it. A right-to-left model reads the passage first and never reaches the question to its left. BERT sees both."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="20.0" text-anchor="start">Rows: passage tokens. Columns: question tokens. The question comes first in the input.</text><text class="t-title" x="152.0" y="56.0" text-anchor="middle">left-to-right</text><text class="t-muted" x="152.0" y="74.0" text-anchor="middle">key</text><text class="t-tick" x="101.0" y="92.0" text-anchor="middle">who</text><text class="t-tick" x="135.0" y="92.0" text-anchor="middle">painted</text><text class="t-tick" x="169.0" y="92.0" text-anchor="middle">it</text><text class="t-tick" x="203.0" y="92.0" text-anchor="middle">?</text><text class="t-tick" x="76.0" y="121.0" text-anchor="end">painted</text><g class="mark"><title>painted attends to who</title><rect class="cell" x="85.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to painted</title><rect class="cell" x="119.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to it</title><rect class="cell" x="153.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to ?</title><rect class="cell" x="187.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="76.0" y="155.0" text-anchor="end">by</text><g class="mark"><title>by attends to who</title><rect class="cell" x="85.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to painted</title><rect class="cell" x="119.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to it</title><rect class="cell" x="153.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to ?</title><rect class="cell" x="187.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="76.0" y="189.0" text-anchor="end">leonardo</text><g class="mark"><title>leonardo attends to who</title><rect class="cell" x="85.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to painted</title><rect class="cell" x="119.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to it</title><rect class="cell" x="153.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to ?</title><rect class="cell" x="187.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="14.0" y="222.0" text-anchor="start"></text><text class="t-tick" x="14.0" y="226.0" text-anchor="start">question is on the left: visible</text><text class="t-title" x="402.0" y="56.0" text-anchor="middle">right-to-left</text><text class="t-muted" x="402.0" y="74.0" text-anchor="middle">key</text><text class="t-tick" x="351.0" y="92.0" text-anchor="middle">who</text><text class="t-tick" x="385.0" y="92.0" text-anchor="middle">painted</text><text class="t-tick" x="419.0" y="92.0" text-anchor="middle">it</text><text class="t-tick" x="453.0" y="92.0" text-anchor="middle">?</text><text class="t-tick" x="326.0" y="121.0" text-anchor="end">painted</text><rect class="cell-masked" x="335.5" y="101.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="344.9" y1="110.9" x2="357.1" y2="123.1"/><line class="edge" x1="357.1" y1="110.9" x2="344.9" y2="123.1"/><rect class="cell-masked" x="369.5" y="101.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="378.9" y1="110.9" x2="391.1" y2="123.1"/><line class="edge" x1="391.1" y1="110.9" x2="378.9" y2="123.1"/><rect class="cell-masked" x="403.5" y="101.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="412.9" y1="110.9" x2="425.1" y2="123.1"/><line class="edge" x1="425.1" y1="110.9" x2="412.9" y2="123.1"/><rect class="cell-masked" x="437.5" y="101.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="446.9" y1="110.9" x2="459.1" y2="123.1"/><line class="edge" x1="459.1" y1="110.9" x2="446.9" y2="123.1"/><text class="t-tick" x="326.0" y="155.0" text-anchor="end">by</text><rect class="cell-masked" x="335.5" y="135.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="344.9" y1="144.9" x2="357.1" y2="157.1"/><line class="edge" x1="357.1" y1="144.9" x2="344.9" y2="157.1"/><rect class="cell-masked" x="369.5" y="135.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="378.9" y1="144.9" x2="391.1" y2="157.1"/><line class="edge" x1="391.1" y1="144.9" x2="378.9" y2="157.1"/><rect class="cell-masked" x="403.5" y="135.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="412.9" y1="144.9" x2="425.1" y2="157.1"/><line class="edge" x1="425.1" y1="144.9" x2="412.9" y2="157.1"/><rect class="cell-masked" x="437.5" y="135.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="446.9" y1="144.9" x2="459.1" y2="157.1"/><line class="edge" x1="459.1" y1="144.9" x2="446.9" y2="157.1"/><text class="t-tick" x="326.0" y="189.0" text-anchor="end">leonardo</text><rect class="cell-masked" x="335.5" y="169.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="344.9" y1="178.9" x2="357.1" y2="191.1"/><line class="edge" x1="357.1" y1="178.9" x2="344.9" y2="191.1"/><rect class="cell-masked" x="369.5" y="169.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="378.9" y1="178.9" x2="391.1" y2="191.1"/><line class="edge" x1="391.1" y1="178.9" x2="378.9" y2="191.1"/><rect class="cell-masked" x="403.5" y="169.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="412.9" y1="178.9" x2="425.1" y2="191.1"/><line class="edge" x1="425.1" y1="178.9" x2="412.9" y2="191.1"/><rect class="cell-masked" x="437.5" y="169.5" width="31.0" height="31.0" rx="3"/><line class="edge" x1="446.9" y1="178.9" x2="459.1" y2="191.1"/><line class="edge" x1="459.1" y1="178.9" x2="446.9" y2="191.1"/><text class="t-muted" x="264.0" y="222.0" text-anchor="start"></text><text class="t-bad" x="264.0" y="226.0" text-anchor="start">question is on the left: never visible</text><text class="t-title" x="652.0" y="56.0" text-anchor="middle">BERT</text><text class="t-muted" x="652.0" y="74.0" text-anchor="middle">key</text><text class="t-tick" x="601.0" y="92.0" text-anchor="middle">who</text><text class="t-tick" x="635.0" y="92.0" text-anchor="middle">painted</text><text class="t-tick" x="669.0" y="92.0" text-anchor="middle">it</text><text class="t-tick" x="703.0" y="92.0" text-anchor="middle">?</text><text class="t-tick" x="576.0" y="121.0" text-anchor="end">painted</text><g class="mark"><title>painted attends to who</title><rect class="cell" x="585.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to painted</title><rect class="cell" x="619.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to it</title><rect class="cell" x="653.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>painted attends to ?</title><rect class="cell" x="687.5" y="101.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="576.0" y="155.0" text-anchor="end">by</text><g class="mark"><title>by attends to who</title><rect class="cell" x="585.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to painted</title><rect class="cell" x="619.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to it</title><rect class="cell" x="653.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>by attends to ?</title><rect class="cell" x="687.5" y="135.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="576.0" y="189.0" text-anchor="end">leonardo</text><g class="mark"><title>leonardo attends to who</title><rect class="cell" x="585.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to painted</title><rect class="cell" x="619.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to it</title><rect class="cell" x="653.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>leonardo attends to ?</title><rect class="cell" x="687.5" y="169.5" width="31.0" height="31.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="514.0" y="222.0" text-anchor="start"></text><text class="t-tick" x="514.0" y="226.0" text-anchor="start">every token sees every token</text></svg><figcaption>Argument (b), as attention tables. Rows are passage tokens, columns question tokens. A left-to-right model lets passage tokens see the question (it comes first). A right-to-left model blocks all of it: the passage never sees the question. BERT allows everything.</figcaption></figure>

Argument (a) is easy to measure. I built an LTR stack and an RTL stack of BERT-base size and timed them against one bidirectional model ([`bert_part5_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5_math.py)):

```text
parameters: one bidirectional model 109,482,240; LTR + RTL = 218,964,480
vector per token: bidirectional (512, 768) ; LTR (512, 768) + RTL (512, 768) -> concat (512, 1536)
time for a batch of 8 x 512 tokens on mps (median of 5): one model 133 ms, LTR + RTL 261 ms -> 1.96x
in a sequence of 6 tokens, which positions can token 3 use (1-based)?
  bidirectional, any layer: [1, 2, 3, 4, 5, 6]
  LTR stack, any layer:     [1, 2, 3]    RTL stack, any layer: [3, 4, 5, 6]
  concatenation: both lists, but only side by side at the very top; no layer ever mixes them
```

Twice the parameters and 1.96 times the time, as the paper says. And the last lines are argument (c) in miniature: in the glued model, no single layer ever combines token 1 with token 6 when building token 3.

<figure class="fig"><svg viewBox="0 0 760 346" role="img" aria-label="Left: ELMo-style, a left-to-right stack and a right-to-left stack whose top vectors are concatenated; inside each stack information flows one way only. Right: BERT, one stack where every layer mixes both sides. Measured: two stacks need twice the parameters and twice the time."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">ELMo-style: two one-way models, glued at the top</text><text class="t-title" x="420.0" y="22.0" text-anchor="start">BERT: one model, both sides in every layer</text><rect class="box-1" x="22.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="186.0" x2="44.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-1" x="76.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="186.0" x2="98.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="98.0" y1="186.0" x2="98.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-1" x="130.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="186.0" x2="152.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="98.0" y1="186.0" x2="152.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="152.0" y1="186.0" x2="152.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-1" x="22.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="142.0" x2="44.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-1" x="76.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="142.0" x2="98.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="98.0" y1="142.0" x2="98.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-1" x="130.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-1" x1="44.0" y1="142.0" x2="152.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="98.0" y1="142.0" x2="152.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-1" x1="152.0" y1="142.0" x2="152.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-1" x="22.0" y="98.0" width="44.0" height="26.0" rx="5"/><rect class="box-1" x="76.0" y="98.0" width="44.0" height="26.0" rx="5"/><rect class="box-1" x="130.0" y="98.0" width="44.0" height="26.0" rx="5"/><text class="t-tick" x="44.0" y="236.0" text-anchor="middle">the</text><text class="t-tick" x="98.0" y="236.0" text-anchor="middle">kid</text><text class="t-tick" x="152.0" y="236.0" text-anchor="middle">smiles</text><text class="t-note" x="98.0" y="84.0" text-anchor="middle">LTR stack</text><rect class="box-2" x="202.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="224.0" y1="186.0" x2="224.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="278.0" y1="186.0" x2="224.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="332.0" y1="186.0" x2="224.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-2" x="256.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="278.0" y1="186.0" x2="278.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="332.0" y1="186.0" x2="278.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-2" x="310.0" y="186.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="332.0" y1="186.0" x2="332.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-2" x="202.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="224.0" y1="142.0" x2="224.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="278.0" y1="142.0" x2="224.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="332.0" y1="142.0" x2="224.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-2" x="256.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="278.0" y1="142.0" x2="278.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-2" x1="332.0" y1="142.0" x2="278.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-2" x="310.0" y="142.0" width="44.0" height="26.0" rx="5"/><line class="edge-2" x1="332.0" y1="142.0" x2="332.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-2" x="202.0" y="98.0" width="44.0" height="26.0" rx="5"/><rect class="box-2" x="256.0" y="98.0" width="44.0" height="26.0" rx="5"/><rect class="box-2" x="310.0" y="98.0" width="44.0" height="26.0" rx="5"/><text class="t-tick" x="224.0" y="236.0" text-anchor="middle">the</text><text class="t-tick" x="278.0" y="236.0" text-anchor="middle">kid</text><text class="t-tick" x="332.0" y="236.0" text-anchor="middle">smiles</text><text class="t-note" x="278.0" y="84.0" text-anchor="middle">RTL stack</text><rect class="box-ghost" x="37.0" y="49.0" width="82.0" height="17.0" rx="4"/><g class="mark"><rect class="s1" x="40.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="53.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="66.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="79.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="92.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="105.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="117.0" y="49.0" width="82.0" height="17.0" rx="4"/><g class="mark"><rect class="s2" x="120.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="133.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="146.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="159.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="172.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="185.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><text class="t-muted" x="40.0" y="44.0" text-anchor="start">smiles = [LTR ; RTL] = 1,536 numbers</text><line class="edge" x1="76.0" y1="98.0" x2="76.0" y2="70.0" marker-end="url(#ah)"/><line class="edge" x1="256.0" y1="98.0" x2="160.0" y2="70.0" marker-end="url(#ah)"/><rect class="box-3" x="470.0" y="186.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="186.0" x2="498.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="186.0" x2="498.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="186.0" x2="498.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-3" x="540.0" y="186.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="186.0" x2="568.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="186.0" x2="568.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="186.0" x2="568.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-3" x="610.0" y="186.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="186.0" x2="638.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="186.0" x2="638.0" y2="168.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="186.0" x2="638.0" y2="168.0" style="stroke-opacity:0.55"/><rect class="box-3" x="470.0" y="142.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="142.0" x2="498.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="142.0" x2="498.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="142.0" x2="498.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-3" x="540.0" y="142.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="142.0" x2="568.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="142.0" x2="568.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="142.0" x2="568.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-3" x="610.0" y="142.0" width="56.0" height="26.0" rx="5"/><line class="edge-3" x1="498.0" y1="142.0" x2="638.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="568.0" y1="142.0" x2="638.0" y2="124.0" style="stroke-opacity:0.55"/><line class="edge-3" x1="638.0" y1="142.0" x2="638.0" y2="124.0" style="stroke-opacity:0.55"/><rect class="box-3" x="470.0" y="98.0" width="56.0" height="26.0" rx="5"/><rect class="box-3" x="540.0" y="98.0" width="56.0" height="26.0" rx="5"/><rect class="box-3" x="610.0" y="98.0" width="56.0" height="26.0" rx="5"/><text class="t-tick" x="498.0" y="236.0" text-anchor="middle">the</text><text class="t-tick" x="568.0" y="236.0" text-anchor="middle">kid</text><text class="t-tick" x="638.0" y="236.0" text-anchor="middle">smiles</text><rect class="box-ghost" x="497.0" y="49.0" width="82.0" height="17.0" rx="4"/><g class="mark"><rect class="s3" x="500.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="513.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="526.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="539.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="552.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="565.0" y="52.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><text class="t-muted" x="500.0" y="44.0" text-anchor="start">smiles = 768 numbers</text><line class="edge" x1="638.0" y1="98.0" x2="540.0" y2="70.0" marker-end="url(#ah)"/><text class="t-tick" x="14.0" y="270.0" text-anchor="start">Measured, BERT-base size, batch of 8 x 512 tokens on an Apple GPU (bert_part5_math.py):</text><text class="t-tick" x="150.0" y="299.0" text-anchor="end">LTR + RTL</text><rect class="s2" x="160" y="287" width="300.0" height="16" rx="3"/><text class="t-val" x="466.0" y="300.0" text-anchor="start">261 ms, 219.0M parameters</text><text class="t-tick" x="150.0" y="329.0" text-anchor="end">one bidirectional</text><rect class="s3" x="160" y="317" width="152.9" height="16" rx="3"/><text class="t-val" x="318.9" y="330.0" text-anchor="start">133 ms, 109.5M parameters</text></svg><figcaption>ELMo-style versus BERT, with the measured cost. Left: an LTR stack and an RTL stack, each one-directional, glued at the top into 1,536 numbers per token. Right: one stack where every layer mixes both sides, 768 numbers per token. Glued: 219.0M parameters and 261 ms per batch; bidirectional: 109.5M and 133 ms.</figcaption></figure>

How ELMo actually combines its layers, in its own paper:

> [!PAPER] Peters et al. (2018a), ELMo · §3.2, Equation (1)
> [![Equation 1 of the ELMo paper: the ELMo vector for token k is gamma task times the sum over layers j of s j task times h k j, a task specific weighting of all biLM layers with softmax-normalized weights s and a scalar gamma](https://ishwarj.com/img/papers/bert/p5-elmo-eq1.pngert/p5-elmo-eq1.png)](/img/papers/bert/p5-elmo-eq1.png)
>
> **What it says:** "we compute a task specific weighting of all biLM layers": $$\text{ELMo}_k = \gamma \sum_{j=0}^{L} s_j\, h_{k,j}$$, where the $$s_j$$ are "softmax-normalized weights" and $$\gamma$$ scales the whole vector.
>
> **What it means:**
>
> Think of a sound mixer with one slider for each layer of ELMo. The task model learns where to set each slider ($$s_j$$) and one master volume ($$\gamma$$). The LSTMs themselves stay frozen.
>
> Here $$h_{k,j}$$ is the vector of token $$k$$ at layer $$j$$, with its forward and backward halves joined. The task model learns only the $$L + 1$$ weights $$s_j$$ and the single number $$\gamma$$.
>
> This "weighted sum" of layers is the idea that Section 5.3 tests on BERT's layers (Table 7, below).

> [!NOTE] An argument, not a measurement
> Point (c) says "strictly less powerful". That is a statement about what the model can represent in principle. The paper gives no experiment comparing BERT with an LTR+RTL concatenation of the same size.

## The other differences between BERT and GPT {§A.4}

Table 5's LTR & No NSP row matters for one more reason, which the paper explains in Appendix A.4.

> [!PAPER] · §A.4
> [![Appendix A.4: OpenAI GPT is the most comparable method. Many design decisions in BERT were made to be as close to GPT as possible. Other differences: GPT is trained on the BooksCorpus (800M words), BERT on BooksCorpus and Wikipedia (2,500M words); GPT uses [SEP] and [CLS] only at fine-tuning time, BERT learns them and the sentence A and B embeddings during pre-training; GPT was trained for 1M steps with a batch size of 32,000 words, BERT for 1M steps with 128,000 words; GPT used the same learning rate of 5e-5 for all fine-tuning experiments, BERT chooses a task-specific fine-tuning learning rate](https://ishwarj.com/img/papers/bert/p5-a4.pngpers/bert/p5-a4.png)](/img/papers/bert/p5-a4.png)
>
> **What it says:** BERT was designed to be as close to GPT as possible, "so that the two methods could be minimally compared". But four other differences remain: training data, when [SEP], [CLS] and the A/B embeddings are learned, batch size, and the fine-tuning learning rate.
>
> **What it means:**
>
> A fair race needs the same track for both runners. BERT was built to be as close to GPT as possible, but four differences remain, and the table below lists them.
>
> Any of these four could explain part of BERT's advantage over GPT. More data, bigger batches or a tuned learning rate might give a few points on their own. A fair comparison has to rule them out before it gives the credit to bidirectionality.

| | OpenAI GPT | BERT |
|---|---|---|
| Pre-training data | BooksCorpus, 800M words | BooksCorpus 800M + Wikipedia 2,500M words |
| [SEP], [CLS], segment A/B | added only at fine-tuning | learned during pre-training |
| Batch size | 32,000 words, 1M steps | 128,000 words, 1M steps |
| Fine-tuning learning rate | always 5e-5 | best of several, chosen per task on Dev |

> [!PAPER] · §A.4
> [![The opening of Appendix A.4: BERT and OpenAI GPT are fine-tuning approaches, while ELMo is a feature-based approach; many design decisions in BERT were intentionally made to make it as close to GPT as possible; the bi-directionality and the two pre-training tasks account for the majority of the empirical improvements](https://ishwarj.com/img/papers/bert/p5-a4-open.pngbert/p5-a4-open.png)](/img/papers/bert/p5-a4-open.png)
>
> **What it says:** design decisions were "intentionally made to make it as close to GPT as possible", and "the bi-directionality and the two pre-training tasks presented in Section 3.1 account for the majority of the empirical improvements".
>
> **What it means:**
>
> This is the paper's thesis in one sentence. BERT and GPT are both fine-tuning approaches, and BERT copied GPT's choices on purpose wherever it could. So the differences that remain are few, and they are the ones that count.
>
> The claim is that two of them, bidirectionality and the two pre-training tasks, give "the majority" of the gain. Section 5.1 is the evidence for it.

<figure class="fig"><svg viewBox="0 0 760 384" role="img" aria-label="The four differences between GPT and BERT besides the architecture: BERT used about four times more text, a four times larger batch, learned its special tokens during pre-training, and tuned the fine-tuning learning rate per task. The LTR and No NSP ablation keeps all four and still loses."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">Appendix A.4: the four non-architecture differences between OpenAI GPT and BERT</text><rect class="s2" x="14" y="36" width="12" height="12" rx="2"/><text class="t-note" x="32.0" y="46.0" text-anchor="start">OpenAI GPT</text><rect class="s1" x="130" y="36" width="12" height="12" rx="2"/><text class="t-note" x="148.0" y="46.0" text-anchor="start">BERT</text><text class="t-note" x="14.0" y="78.0" text-anchor="start">pre-training text</text><rect class="s2" x="190" y="64" width="80.0" height="18" rx="3"/><text class="t-tick" x="276.0" y="78.0" text-anchor="start">BooksCorpus, 800M words</text><rect class="s1" x="190" y="88" width="330.0" height="18" rx="3"/><text class="t-tick" x="526.0" y="102.0" text-anchor="start">BooksCorpus + Wikipedia, 3,300M words</text><text class="t-note" x="14.0" y="142.0" text-anchor="start">batch size</text><rect class="s2" x="190" y="128" width="82.5" height="18" rx="3"/><text class="t-tick" x="278.5" y="142.0" text-anchor="start">32,000 words per step, 1M steps</text><rect class="s1" x="190" y="152" width="330.0" height="18" rx="3"/><text class="t-tick" x="526.0" y="166.0" text-anchor="start">128,000 words per step, 1M steps</text><text class="t-note" x="14.0" y="206.0" text-anchor="start">[CLS], [SEP], A/B</text><rect class="box-2" x="190.0" y="192.0" width="250.0" height="22.0" rx="5"/><text class="t-tick" x="200.0" y="207.0" text-anchor="start">GPT: added only at fine-tuning</text><rect class="box-1" x="190.0" y="220.0" width="250.0" height="22.0" rx="5"/><text class="t-tick" x="200.0" y="235.0" text-anchor="start">BERT: learned during pre-training</text><text class="t-note" x="14.0" y="270.0" text-anchor="start">fine-tuning rate</text><rect class="box-2" x="190.0" y="256.0" width="250.0" height="22.0" rx="5"/><text class="t-tick" x="200.0" y="271.0" text-anchor="start">GPT: always 5e-5</text><rect class="box-1" x="190.0" y="284.0" width="250.0" height="22.0" rx="5"/><text class="t-tick" x="200.0" y="299.0" text-anchor="start">BERT: best of 5e-5, 3e-5, 2e-5 on Dev</text><rect class="box-on" x="14.0" y="326.0" width="732.0" height="46.0" rx="10"/><text class="t-note" x="26.0" y="345.0" text-anchor="start">The "LTR &amp; No NSP" row of Table 5 = GPT's objective + all four BERT advantages above.</text><text class="t-tick" x="26.0" y="363.0" text-anchor="start">So its gap to BERT-base (MRPC 77.5 vs 86.7, SQuAD 77.8 vs 88.5) comes from the masked LM and NSP.</text></svg><figcaption>The four non-architecture differences between OpenAI GPT and BERT, as bars and boxes: training text (800M against 3,300M words), batch size (32,000 against 128,000 words per step), when [CLS], [SEP] and A/B are learned, and the fine-tuning learning rate. The "LTR & No NSP" row of Table 5 is GPT's objective with all four BERT advantages.</figcaption></figure>

> [!PAPER] · §A.4
> [![To isolate the effect of these differences, we perform ablation experiments in Section 5.1 which demonstrate that the majority of the improvements are in fact coming from the two pre-training tasks and the bidirectionality they enable](https://ishwarj.com/img/papers/bert/p5-a4-end.png/bert/p5-a4-end.png)](/img/papers/bert/p5-a4-end.png)
>
> **What it says:** the Section 5.1 ablations show "that the majority of the improvements are in fact coming from the two pre-training tasks and the bidirectionality they enable".
>
> **What it means:**
>
> The logic is a clever control. The LTR & No NSP model is GPT's recipe: left to right, with no NSP. But it is trained with **all four of BERT's advantages**: BERT's data, input format, batch size and fine-tuning.
>
> So any gap between LTR & No NSP and BERT-base cannot come from those four things. It must come from the masked LM, which gives bidirectionality, and from NSP. That gap is large: up to 9.2 points on MRPC (77.5 versus 86.7) and 10.7 on SQuAD (77.8 versus 88.5).

## Effect of model size {§5.2}

> [!PAPER] · §5.2
> [![Section 5.2: we trained BERT models with a differing number of layers, hidden units and attention heads, otherwise using the same hyperparameters and training procedure. Table 6 reports the average Dev Set accuracy from 5 random restarts of fine-tuning. Larger models lead to a strict accuracy improvement across all four datasets, even for MRPC which only has 3,600 labeled training examples](https://ishwarj.com/img/papers/bert/p5-size-intro.pngt/p5-size-intro.png)](/img/papers/bert/p5-size-intro.png)
>
> **What it says:** several BERT models with different numbers of layers, hidden sizes and heads, all trained the same way. Table 6 reports "the average Dev Set accuracy from 5 random restarts of fine-tuning". "Larger models lead to a strict accuracy improvement", "even for MRPC which only has 3,600 labeled training examples".
>
> **What it means:**
>
> In 2018, many people expected a huge model to overfit a tiny dataset. Think of a student who memorises the answers to 3,600 practice questions and then fails the real exam. The yellow box below defines overfitting.
>
> Here bigger was better even on MRPC, the smallest task. The average over 5 restarts matters: small datasets give noisy scores, and the average makes the trend easier to trust.

> [!DEFINITION] Random restarts
> Fine-tuning the same pre-trained model several times with different random choices (the order of the training examples and the starting values of the new output layer). Results vary a little between restarts, so averaging 5 of them gives a steadier number.

> [!DEFINITION] Overfitting
> When a model memorises its training examples instead of learning the general pattern, so it does well on training data but badly on new data. Big models with few examples are the classic risk.

A small inconsistency: the text says "all four datasets", but Table 6 shows three tasks (MNLI-m, MRPC, SST-2). The paper does not say what the fourth one is.

> [!PAPER] · §5.2, Table 6
> [![Table 6: ablation over BERT model size. Columns: number of layers, hidden size, number of attention heads, masked LM perplexity of held-out training data, and Dev accuracy on MNLI-m, MRPC and SST-2. Rows: 3, 768, 12, 5.84, 77.9, 79.8, 88.4. 6, 768, 3, 5.24, 80.6, 82.2, 90.7. 6, 768, 12, 4.68, 81.9, 84.8, 91.3. 12, 768, 12, 3.99, 84.4, 86.7, 92.9. 12, 1024, 16, 3.54, 85.7, 86.9, 93.3. 24, 1024, 16, 3.23, 86.6, 87.8, 93.7](https://ishwarj.com/img/papers/bert/p5-table6.png/bert/p5-table6.png)](/img/papers/bert/p5-table6.png)
>
> **What it says:** six model sizes. For each: #L (layers), #H (hidden size), #A (attention heads), "LM (ppl)", "the masked LM perplexity of held-out training data", and Dev accuracy on three tasks.
>
> **What it means:**
>
> Read the table from top to bottom: the models grow in layers, width or heads. Every column improves as you go down. Perplexity falls, and all three accuracies rise.
>
> The fourth row (12, 768, 12) is BERT-base and the last (24, 1024, 16) is BERT-large. The perplexity column measures how well each model fills in the blanks; it is explained below.

You met L, H and A in Part 2. The new column is perplexity.

> [!DEFINITION] Perplexity (ppl)
> A measure of how surprised a language model is by real text. A perplexity of 4 means the model is, on average, as unsure as if it were choosing between 4 equally likely words. Lower is better; a perfect model would have perplexity 1.

For a masked LM, the perplexity is computed over the positions the model has to predict:

$$
\text{ppl} = \exp\!\left( -\frac{1}{N} \sum_{i=1}^{N} \log p(x_i \mid \text{context}) \right)
$$

where:

- $$N$$ is the number of predicted (masked) positions;
- $$x_i$$ is the true token at the $$i$$-th predicted position;
- $$p(x_i \mid \text{context})$$ is the probability the model gave to that true token, from its softmax over the vocabulary;
- $$\log$$ is the natural logarithm, and the minus sign makes the sum positive: the inside of the bracket is the average **cross-entropy** loss (Part 3);
- $$\exp$$ ("e to the power of") undoes the logarithm, turning an average loss into a number of "equally likely choices".

So BERT-base's 3.99 means: on its held-out training text, when it predicts a masked token, it is on average about as unsure as a choice between 4 words.

### How many parameters is each row?

Where the per-layer term $$12H^2 + 13H$$ comes from, piece by piece (all for one layer with hidden size $$H$$ and feed-forward size $$4H$$):

| Part | Matrices | Biases and LayerNorms | Count |
|---|---|---|---|
| Q, K, V, O projections | $$4 \times H \times H$$ | $$4H$$ | $$4H^2 + 4H$$ |
| Feed-forward | $$H \times 4H + 4H \times H$$ | $$4H + H$$ | $$8H^2 + 5H$$ |
| Two LayerNorms | | $$2 \times 2H$$ | $$4H$$ |
| **One layer** | | | $$12H^2 + 13H$$ |

For $$H = 768$$: $$12 \times 768^2 + 13 \times 768 = 7{,}077{,}888 + 9{,}984 = 7{,}087{,}872$$ parameters per layer.

<figure class="fig"><svg viewBox="0 0 760 288" role="img" aria-label="Stacked bars for the six model sizes of Table 6: embeddings, attention, feed-forward, and the small LayerNorm and pooler terms. The feed-forward networks are always two thirds of the layer parameters; the embeddings are a large share only in small models."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">Where the parameters of each Table 6 model live (counted, term by term)</text><rect class="s1" x="14" y="34" width="12" height="12" rx="2"/><text class="t-note" x="32.0" y="44.0" text-anchor="start">embeddings</text><rect class="s2" x="174" y="34" width="12" height="12" rx="2"/><text class="t-note" x="192.0" y="44.0" text-anchor="start">attention</text><rect class="s3" x="334" y="34" width="12" height="12" rx="2"/><text class="t-note" x="352.0" y="44.0" text-anchor="start">feed-forward</text><rect class="s4" x="494" y="34" width="12" height="12" rx="2"/><text class="t-note" x="512.0" y="44.0" text-anchor="start">LayerNorm + pooler</text><text class="t-tick" x="140.0" y="75.0" text-anchor="end">L=3, H=768, A=12</text><g class="mark"><title>embeddings: 23,837,184</title><rect class="s1" x="150.0" y="62" width="33.4" height="20"/></g><g class="mark"><title>attention: 7,087,104</title><rect class="s2" x="183.4" y="62" width="9.9" height="20"/></g><g class="mark"><title>feed-forward: 14,167,296</title><rect class="s3" x="193.4" y="62" width="19.9" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 599,808</title><rect class="s4" x="213.2" y="62" width="1.5" height="20"/></g><text class="t-val" x="222.1" y="76.0" text-anchor="start">45.7M</text><text class="t-tick" x="140.0" y="109.0" text-anchor="end">L=6, H=768, A=3</text><g class="mark"><title>embeddings: 23,837,184</title><rect class="s1" x="150.0" y="96" width="33.4" height="20"/></g><g class="mark"><title>attention: 14,174,208</title><rect class="s2" x="183.4" y="96" width="19.9" height="20"/></g><g class="mark"><title>feed-forward: 28,334,592</title><rect class="s3" x="203.3" y="96" width="39.7" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 609,024</title><rect class="s4" x="243.0" y="96" width="1.5" height="20"/></g><text class="t-val" x="251.9" y="110.0" text-anchor="start">67.0M</text><text class="t-tick" x="140.0" y="143.0" text-anchor="end">L=6, H=768, A=12</text><g class="mark"><title>embeddings: 23,837,184</title><rect class="s1" x="150.0" y="130" width="33.4" height="20"/></g><g class="mark"><title>attention: 14,174,208</title><rect class="s2" x="183.4" y="130" width="19.9" height="20"/></g><g class="mark"><title>feed-forward: 28,334,592</title><rect class="s3" x="203.3" y="130" width="39.7" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 609,024</title><rect class="s4" x="243.0" y="130" width="1.5" height="20"/></g><text class="t-val" x="251.9" y="144.0" text-anchor="start">67.0M</text><text class="t-tick" x="140.0" y="177.0" text-anchor="end">L=12, H=768, A=12</text><g class="mark"><title>embeddings: 23,837,184</title><rect class="s1" x="150.0" y="164" width="33.4" height="20"/></g><g class="mark"><title>attention: 28,348,416</title><rect class="s2" x="183.4" y="164" width="39.8" height="20"/></g><g class="mark"><title>feed-forward: 56,669,184</title><rect class="s3" x="223.2" y="164" width="79.5" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 627,456</title><rect class="s4" x="302.7" y="164" width="1.5" height="20"/></g><text class="t-val" x="311.5" y="178.0" text-anchor="start">109.5M</text><text class="t-tick" x="140.0" y="211.0" text-anchor="end">L=12, H=1024, A=16</text><g class="mark"><title>embeddings: 31,782,912</title><rect class="s1" x="150.0" y="198" width="44.6" height="20"/></g><g class="mark"><title>attention: 50,380,800</title><rect class="s2" x="194.6" y="198" width="70.7" height="20"/></g><g class="mark"><title>feed-forward: 100,724,736</title><rect class="s3" x="265.2" y="198" width="141.3" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 1,098,752</title><rect class="s4" x="406.5" y="198" width="1.5" height="20"/></g><text class="t-val" x="416.0" y="212.0" text-anchor="start">184.0M</text><text class="t-tick" x="140.0" y="245.0" text-anchor="end">L=24, H=1024, A=16</text><g class="mark"><title>embeddings: 31,782,912</title><rect class="s1" x="150.0" y="232" width="44.6" height="20"/></g><g class="mark"><title>attention: 100,761,600</title><rect class="s2" x="194.6" y="232" width="141.3" height="20"/></g><g class="mark"><title>feed-forward: 201,449,472</title><rect class="s3" x="335.9" y="232" width="282.5" height="20"/></g><g class="mark"><title>LayerNorm + pooler: 1,147,904</title><rect class="s4" x="618.4" y="232" width="1.6" height="20"/></g><text class="t-val" x="628.0" y="246.0" text-anchor="start">335.1M</text><text class="t-muted" x="14.0" y="276.0" text-anchor="start">Each layer adds 12H² + 13H numbers: 7.09M at H = 768, 12.60M at H = 1024. The embedding tables do not grow with L.</text></svg><figcaption>Where the parameters of each Table 6 model live. Blue: embeddings (23.8M or 31.8M, fixed by H, not by L). Orange: attention. Green: feed-forward. The embedding tables do not grow with L, so in the 3-layer model they are half of everything.</figcaption></figure>

The paper gives parameter counts only for BERT-base and BERT-large. I built every model of Table 6 with the Hugging Face `BertConfig` (30,522-token vocabulary, 512 positions, feed-forward size 4H, as in Part 2) on PyTorch's `meta` device, which creates the shapes without allocating any memory, and counted:

```python
import torch
from transformers import BertConfig, BertModel

for L, H, A in [(3, 768, 12), (6, 768, 3), (6, 768, 12), (12, 768, 12), (12, 1024, 16), (24, 1024, 16)]:
    cfg = BertConfig(vocab_size=30522, hidden_size=H, num_hidden_layers=L, num_attention_heads=A,
                     intermediate_size=4 * H, max_position_embeddings=512, type_vocab_size=2)
    with torch.device("meta"):              # shapes only, no memory
        model = BertModel(cfg)
    print(L, H, A, sum(p.numel() for p in model.parameters()))
```

```text
 #L    #H  #A        params       formula  embeddings   ppl  MNLI-m  MRPC  SST-2
  3   768  12    45,691,392    45,691,392  23,837,184  5.84    77.9  79.8   88.4
  6   768   3    66,955,008    66,955,008  23,837,184  5.24    80.6  82.2   90.7
  6   768  12    66,955,008    66,955,008  23,837,184  4.68    81.9  84.8   91.3
 12   768  12   109,482,240   109,482,240  23,837,184  3.99    84.4  86.7   92.9
 12  1024  16   183,987,200   183,987,200  31,782,912  3.54    85.7  86.9   93.3
 24  1024  16   335,141,888   335,141,888  31,782,912  3.23    86.6  87.8   93.7
```

The "formula" column is the count from the equation of Part 2 (embeddings, plus $$12H^2 + 13H$$ per layer, plus the $$H \times H$$ pooler), and it matches the real model exactly in every row. Three things stand out:

- **BERT-base is 109.5M** and **BERT-large is 335.1M** parameters, counted this way. The paper rounds them to "110M" and "340M". (Part 2 discusses the 340M.)
- **The two 6-layer models have exactly the same number of parameters** (66,955,008), yet 12 heads beat 3 heads on all three tasks (MNLI-m 81.9 versus 80.6). The number of heads only changes how the same $$H \times H$$ matrices are split up, not how many numbers they hold.
- **The embeddings are a big share of small models.** In the 3-layer model, 23.8M of its 45.7M parameters (52%) are embeddings, the word-lookup table alone.

<figure class="fig"><svg viewBox="0 0 760 350" role="img" aria-label="Table 6 as a chart: Dev accuracy on MNLI-m, MRPC and SST-2 rises with every bigger model, from 3 layers (46M parameters) to 24 layers (335M). Parameter counts are measured; accuracies are from the paper."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="56" y1="260.0" x2="640" y2="260.0"/><text class="t-tick" x="48.0" y="264.0" text-anchor="end">76</text><line class="grid" x1="56" y1="216.0" x2="640" y2="216.0"/><text class="t-tick" x="48.0" y="220.0" text-anchor="end">80</text><line class="grid" x1="56" y1="172.0" x2="640" y2="172.0"/><text class="t-tick" x="48.0" y="176.0" text-anchor="end">84</text><line class="grid" x1="56" y1="128.0" x2="640" y2="128.0"/><text class="t-tick" x="48.0" y="132.0" text-anchor="end">88</text><line class="grid" x1="56" y1="84.0" x2="640" y2="84.0"/><text class="t-tick" x="48.0" y="88.0" text-anchor="end">92</text><line class="grid" x1="56" y1="40.0" x2="640" y2="40.0"/><text class="t-tick" x="48.0" y="44.0" text-anchor="end">96</text><polyline class="l3" points="86.0,123.6 190.8,98.3 295.6,91.7 400.4,74.1 505.2,69.7 610.0,65.3"/><g class="mark"><title>SST-2, L=3 H=768 A=12: 88.4</title><circle class="s3 ring" cx="86.0" cy="123.6" r="5"/></g><g class="mark"><title>SST-2, L=6 H=768 A=3: 90.7</title><circle class="s3 ring" cx="190.8" cy="98.3" r="5"/></g><g class="mark"><title>SST-2, L=6 H=768 A=12: 91.3</title><circle class="s3 ring" cx="295.6" cy="91.7" r="5"/></g><g class="mark"><title>SST-2, L=12 H=768 A=12: 92.9</title><circle class="s3 ring" cx="400.4" cy="74.1" r="5"/></g><g class="mark"><title>SST-2, L=12 H=1024 A=16: 93.3</title><circle class="s3 ring" cx="505.2" cy="69.7" r="5"/></g><g class="mark"><title>SST-2, L=24 H=1024 A=16: 93.7</title><circle class="s3 ring" cx="610.0" cy="65.3" r="5"/></g><rect class="s3" x="626.0" y="60.3" width="10" height="10" rx="2"/><text class="t-note" x="642.0" y="69.3" text-anchor="start">SST-2 93.7</text><polyline class="l2" points="86.0,218.2 190.8,191.8 295.6,163.2 400.4,142.3 505.2,140.1 610.0,130.2"/><g class="mark"><title>MRPC, L=3 H=768 A=12: 79.8</title><circle class="s2 ring" cx="86.0" cy="218.2" r="5"/></g><g class="mark"><title>MRPC, L=6 H=768 A=3: 82.2</title><circle class="s2 ring" cx="190.8" cy="191.8" r="5"/></g><g class="mark"><title>MRPC, L=6 H=768 A=12: 84.8</title><circle class="s2 ring" cx="295.6" cy="163.2" r="5"/></g><g class="mark"><title>MRPC, L=12 H=768 A=12: 86.7</title><circle class="s2 ring" cx="400.4" cy="142.3" r="5"/></g><g class="mark"><title>MRPC, L=12 H=1024 A=16: 86.9</title><circle class="s2 ring" cx="505.2" cy="140.1" r="5"/></g><g class="mark"><title>MRPC, L=24 H=1024 A=16: 87.8</title><circle class="s2 ring" cx="610.0" cy="130.2" r="5"/></g><rect class="s2" x="626.0" y="125.2" width="10" height="10" rx="2"/><text class="t-note" x="642.0" y="134.2" text-anchor="start">MRPC 87.8</text><polyline class="l1" points="86.0,239.1 190.8,209.4 295.6,195.1 400.4,167.6 505.2,153.3 610.0,143.4"/><g class="mark"><title>MNLI-m, L=3 H=768 A=12: 77.9</title><circle class="s1 ring" cx="86.0" cy="239.1" r="5"/></g><g class="mark"><title>MNLI-m, L=6 H=768 A=3: 80.6</title><circle class="s1 ring" cx="190.8" cy="209.4" r="5"/></g><g class="mark"><title>MNLI-m, L=6 H=768 A=12: 81.9</title><circle class="s1 ring" cx="295.6" cy="195.1" r="5"/></g><g class="mark"><title>MNLI-m, L=12 H=768 A=12: 84.4</title><circle class="s1 ring" cx="400.4" cy="167.6" r="5"/></g><g class="mark"><title>MNLI-m, L=12 H=1024 A=16: 85.7</title><circle class="s1 ring" cx="505.2" cy="153.3" r="5"/></g><g class="mark"><title>MNLI-m, L=24 H=1024 A=16: 86.6</title><circle class="s1 ring" cx="610.0" cy="143.4" r="5"/></g><rect class="s1" x="626.0" y="138.4" width="10" height="10" rx="2"/><text class="t-note" x="642.0" y="147.4" text-anchor="start">MNLI-m 86.6</text><text class="t-tick" x="86.0" y="280.0" text-anchor="middle">L=3, H=768</text><text class="t-tick" x="86.0" y="296.0" text-anchor="middle">A=12</text><text class="t-muted" x="86.0" y="314.0" text-anchor="middle">46M params</text><text class="t-muted" x="86.0" y="330.0" text-anchor="middle">LM ppl 5.84</text><text class="t-tick" x="190.8" y="280.0" text-anchor="middle">L=6, H=768</text><text class="t-tick" x="190.8" y="296.0" text-anchor="middle">A=3</text><text class="t-muted" x="190.8" y="314.0" text-anchor="middle">67M params</text><text class="t-muted" x="190.8" y="330.0" text-anchor="middle">LM ppl 5.24</text><text class="t-tick" x="295.6" y="280.0" text-anchor="middle">L=6, H=768</text><text class="t-tick" x="295.6" y="296.0" text-anchor="middle">A=12</text><text class="t-muted" x="295.6" y="314.0" text-anchor="middle">67M params</text><text class="t-muted" x="295.6" y="330.0" text-anchor="middle">LM ppl 4.68</text><text class="t-tick" x="400.4" y="280.0" text-anchor="middle">L=12, H=768</text><text class="t-tick" x="400.4" y="296.0" text-anchor="middle">A=12</text><text class="t-muted" x="400.4" y="314.0" text-anchor="middle">109M params</text><text class="t-muted" x="400.4" y="330.0" text-anchor="middle">LM ppl 3.99</text><text class="t-tick" x="505.2" y="280.0" text-anchor="middle">L=12, H=1024</text><text class="t-tick" x="505.2" y="296.0" text-anchor="middle">A=16</text><text class="t-muted" x="505.2" y="314.0" text-anchor="middle">184M params</text><text class="t-muted" x="505.2" y="330.0" text-anchor="middle">LM ppl 3.54</text><text class="t-tick" x="610.0" y="280.0" text-anchor="middle">L=24, H=1024</text><text class="t-tick" x="610.0" y="296.0" text-anchor="middle">A=16</text><text class="t-muted" x="610.0" y="314.0" text-anchor="middle">335M params</text><text class="t-muted" x="610.0" y="330.0" text-anchor="middle">LM ppl 3.23</text><line class="axis" x1="56" y1="260" x2="640" y2="260"/><text class="t-tick" x="12.0" y="24.0" text-anchor="start">Dev accuracy (Table 6, average of 5 fine-tuning runs)</text></svg><figcaption>Table 6 as a chart. Accuracy rises with every step up in size, for all three tasks, including MRPC with its few thousand training examples. Parameter counts are my measurements; everything else is from the paper.</figcaption></figure>

### Was this big for 2018?

The two comparison models, in their own papers:

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · Table 3
> [![Table 3 of Attention Is All You Need: variations on the Transformer, with the big model row highlighted: N 6, d model 1024, d ff 4096, h 16, and 213 million parameters](https://ishwarj.com/img/papers/bert/p5-vaswani-table3.png-vaswani-table3.png)](/img/papers/bert/p5-vaswani-table3.png)
>
> **What it says:** the "big" Transformer has $$N = 6$$ layers, $$d_{\text{model}} = 1024$$, $$d_{ff} = 4096$$, $$h = 16$$ heads, and **213 million** parameters in total (encoder and decoder together).
>
> **What it means:**
>
> This is the "L=6, H=1024, A=16" of the BERT sentence below. The "big" Transformer was built for translation, so it has an encoder and a decoder.
>
> The BERT paper counts only the encoder side, "100M parameters for the encoder". That is roughly its share of the 213M in total.

> [!PAPER] Al-Rfou et al. (2018), Character-Level Language Modeling with Deeper Self-Attention · Abstract and Table 1
> [![From the Al-Rfou et al. abstract: a deep 64-layer transformer model with fixed context outperforms RNN variants by a large margin](https://ishwarj.com/img/papers/bert/p5-alrfou-abstract.pngalrfou-abstract.png)](/img/papers/bert/p5-alrfou-abstract.png)
> [![Table 1 of Al-Rfou et al.: parameters in millions, with the T64 model at 235 million for training and 219 million for inference, bits per character 1.13 on text8](https://ishwarj.com/img/papers/bert/p5-alrfou-table1.png5-alrfou-table1.png)](/img/papers/bert/p5-alrfou-table1.png)
>
> **What it says:** "a deep (64-layer) transformer model" reaches 1.13 bits per character on text8. Table 1: T64 has **235M** parameters for training (219M at inference; the difference is extra training-only outputs).
>
> **What it means:**
>
> This model reads text one character at a time, not one word at a time, and it is very deep: 64 layers. With 235M parameters, it was the "largest Transformer" that the BERT authors had found.
>
> BERT-large, with 335M parameters by our count, was larger than both models. So the Table 6 gains came on top of sizes that were already at the edge of 2018 practice.

Can we check these two numbers with our formula? Only roughly, because both models differ from BERT in their embeddings and outputs:

```text
Vaswani et al. 2017, big encoder: L=6, H=1024: the layers alone, if shaped like BERT's (12H^2 + 13H each) = 75,577,344   (paper says 100M for the encoder)
Al-Rfou et al. 2018: L=64, H=512: the layers alone, if shaped like BERT's (12H^2 + 13H each) = 201,752,576   (paper says 235M)
```

The layers alone give 75.6M and 201.8M. The rest of the published totals is embeddings and other parts (the big Transformer's shared vocabulary table of about 37,000 tokens × 1,024 adds roughly 38M). So the orders of magnitude agree; the exact numbers depend on details these papers do not spell out in the same way.

> [!PAPER] · §5.2
> [![For example, the largest Transformer explored in Vaswani et al. (2017) is (L=6, H=1024, A=16) with 100M parameters for the encoder, and the largest Transformer we have found in the literature is (L=64, H=512, A=2) with 235M parameters (Al-Rfou et al., 2018). By contrast, BERT-base contains 110M parameters and BERT-large contains 340M parameters](https://ishwarj.com/img/papers/bert/p5-size-prior.pngt/p5-size-prior.png)](/img/papers/bert/p5-size-prior.png)
>
> **What it says:** the largest Transformer in *Attention Is All You Need* is (L=6, H=1024, A=16) with "100M parameters for the encoder"; the largest they found anywhere is (L=64, H=512, A=2) with 235M (Al-Rfou et al., 2018). BERT-base has 110M and BERT-large 340M.
>
> **What it means:**
>
> This sentence places BERT among the models of its time. Before BERT, a Transformer with a few hundred million parameters was rare. BERT-large was bigger than any Transformer the authors knew of.
>
> That makes Table 6 more striking. The gains did not come from growing a tiny model to a normal size. They came on top of models that were "already quite large".

> [!PAPER] · §5.2
> [![It has long been known that increasing the model size will lead to continual improvements on large-scale tasks such as machine translation and language modeling. However, we believe that this is the first work to demonstrate convincingly that scaling to extreme model sizes also leads to large improvements on very small scale tasks, provided that the model has been sufficiently pre-trained](https://ishwarj.com/img/papers/bert/p5-size-claim.pngt/p5-size-claim.png)](/img/papers/bert/p5-size-claim.png)
> [![Peters et al. presented mixed results on increasing the pre-trained bi-LM size from two to four layers, and Melamud et al. found 200 to 600 hidden units helped but 1,000 did not. Both used a feature-based approach. We hypothesize that when the model is fine-tuned directly and uses only a very small number of randomly initialized additional parameters, the task-specific models can benefit from the larger, more expressive pre-trained representations even when downstream task data is very small](https://ishwarj.com/img/papers/bert/p5-size-claim2.png/p5-size-claim2.png)](/img/papers/bert/p5-size-claim2.png)
>
> **What it says:** bigger models were already known to help large tasks like translation. The new claim: "scaling to extreme model sizes also leads to large improvements on very small scale tasks, provided that the model has been sufficiently pre-trained". Earlier feature-based work found mixed results from making models bigger. The authors "hypothesize" the difference is fine-tuning: with only a few new random parameters, even a small task can use a large pre-trained model.
>
> **What it means:**
>
> The new claim is about small tasks. Bigger models were known to help when there is a lot of task data, as in translation. The surprise is that they also help when a task has only a few thousand examples, "provided that the model has been sufficiently pre-trained".
>
> Why did earlier feature-based work see mixed results? The authors "hypothesize" that the answer is fine-tuning. With only a few new random weights, almost everything the task uses was learned in pre-training. So a small task does not have to train a big model from zero.
>
> Note the careful word: the paper offers this explanation but does not test it directly. The measured fact is Table 6 itself. Still, this is the seed of the idea that shaped the following years: pre-train a bigger model, and every task benefits.

### A real perplexity, for scale

First, the definition with a toy you can follow. For $$N$$ predicted words with probabilities $$p_1, \dots, p_N$$ given to the true words:

$$
\text{NLL} = -\frac{1}{N}\sum_{n=1}^{N} \log p_n, \qquad \text{perplexity} = e^{\text{NLL}}
$$

where NLL is the mean negative log-likelihood in **nats** (natural-log units). Sanity check: a model that is uniform over 4 words gives every word $$p = 1/4$$, so NLL $$= \log 4 = 1.386$$ and perplexity $$= 4$$: "as unsure as a 4-way guess".

A real sentence, three masked words, `bert-base-uncased`:

```text
input: she opened the [MASK] with her [MASK] and walked into the [MASK] .
door     p = 0.9586   -log p = 0.0423   (top 3 guesses: door, gate, box)
key      p = 0.1324   -log p = 2.0222   (top 3 guesses: key, hand, keys)
kitchen  p = 0.1823   -log p = 1.7022   (top 3 guesses: room, kitchen, house)
mean NLL = (0.0423 + 2.0222 + 1.7022) / 3 = 1.2554 nats -> perplexity = exp(1.2554) = 3.51
GPT-2 mean NLL = 3.7125 -> perplexity 40.96  (same three words, left context only)
```

<figure class="fig"><svg viewBox="0 0 760 224" role="img" aria-label="Perplexity of bert-base-uncased on one sentence with three masked words. The model gives the true words probabilities 0.9586, 0.1324 and 0.1823; their minus logs average 1.2554; e to that power is a perplexity of 3.51."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">she opened the [MASK] with her [MASK] and walked into the [MASK] .</text><text class="t-note" x="14.0" y="52.0" text-anchor="start">1. probability of the true word</text><text class="t-note" x="276.0" y="52.0" text-anchor="start">2. surprise = minus log p</text><text class="t-note" x="520.0" y="52.0" text-anchor="start">3. average, then e to that</text><text class="t-tick" x="14.0" y="87.0" text-anchor="start">door</text><rect class="s1" x="80" y="74" width="143.8" height="18" rx="3"/><text class="t-val" x="229.8" y="88.0" text-anchor="start">0.9586</text><rect class="s2" x="276" y="74" width="3.1" height="18" rx="3"/><text class="t-val" x="285.1" y="88.0" text-anchor="start">0.0423</text><text class="t-tick" x="14.0" y="127.0" text-anchor="start">key</text><rect class="s1" x="80" y="114" width="19.9" height="18" rx="3"/><text class="t-val" x="105.9" y="128.0" text-anchor="start">0.1324</text><rect class="s2" x="276" y="114" width="150.0" height="18" rx="3"/><text class="t-val" x="432.0" y="128.0" text-anchor="start">2.0222</text><text class="t-tick" x="14.0" y="167.0" text-anchor="start">kitchen</text><rect class="s1" x="80" y="154" width="27.3" height="18" rx="3"/><text class="t-val" x="113.3" y="168.0" text-anchor="start">0.1823</text><rect class="s2" x="276" y="154" width="126.3" height="18" rx="3"/><text class="t-val" x="408.3" y="168.0" text-anchor="start">1.7022</text><rect class="box-on" x="520.0" y="74.0" width="226.0" height="104.0" rx="10"/><text class="t-note" x="633.0" y="100.0" text-anchor="middle">mean = 1.2554 nats</text><text class="t-big" x="633.0" y="128.0" text-anchor="middle">ppl = e^1.2554 = 3.51</text><text class="t-muted" x="633.0" y="152.0" text-anchor="middle">about as unsure as a pick</text><text class="t-muted" x="633.0" y="168.0" text-anchor="middle">among 3.5 equal words</text><text class="t-tick" x="14.0" y="210.0" text-anchor="start">Same three words for GPT-2, which sees only the left side: perplexity 40.96. "key" alone gets p = 0.0006.</text></svg><figcaption>Perplexity in three steps on one sentence. 1: the probability of each true word. 2: the surprise, minus the log of each. 3: average the surprise and raise e to it: 3.51, "about as unsure as a pick among 3.5 equal words".</figcaption></figure>

BERT's 3.51 here is close to Table 6's numbers (3.23 to 5.84), but it is one sentence. (GPT-2's 40.96 is not a fair comparison: it predicts each word from the left side only, a harder task.)


What does a masked-LM perplexity of about 4 look like on real text? I measured the released `bert-base-uncased` on the test split of WikiText-2, a public set of Wikipedia articles, using the paper's masking recipe (15% of tokens chosen; of those 80% `[MASK]`, 10% random, 10% unchanged), sequences of 512 tokens, and a fixed random seed:

```python
chosen = torch.rand(x.shape) < 0.15                    # pick 15% of positions
chosen[:, 0] = chosen[:, -1] = False                   # never [CLS] or [SEP]
r = torch.rand(x.shape)
inp = x.clone()
inp[chosen & (r < 0.8)] = tok.mask_token_id            # 80%: [MASK]
rnd = chosen & (r >= 0.8) & (r < 0.9)                  # 10%: a random token
inp[rnd] = torch.randint(len(tok), x.shape)[rnd]       # the other 10% stay unchanged
logp = torch.log_softmax(model(input_ids=inp).logits[chosen], -1)
nll += -logp.gather(1, x[chosen][:, None]).sum()       # cross-entropy on the chosen positions only
```

```text
WikiText-2 test: 261,428 WordPiece tokens -> 512 sequences of 512 ([CLS] + 510 + [SEP])
predicted positions: 38,984 of 261,120 (14.93%): 31,282 [MASK], 3,947 random, 3,755 unchanged
mean cross-entropy = 1.9967 nats -> masked-LM perplexity = 7.36   (top-1 accuracy 63.6%)
```

So on this text the released model has a perplexity of **7.36**, and its first guess is right **63.6%** of the time. That is higher (worse) than the **3.99** in Table 6. I cannot reproduce the paper's number, and the comparison is not like for like:

- The paper measured "held-out training data": text from the same BooksCorpus and Wikipedia mix BERT was trained on. WikiText-2 is a different sample, with its own formatting (I undid its `@-@` style marks and dropped headings, but other differences remain).
- Table 6's models are the ablation models, trained with the same procedure; the paper does not say the 3.99 was measured on the released checkpoint.
- My random replacements draw from the whole vocabulary, including rarely used tokens.

What the measurement does show: the 15% / 80-10-10 recipe behaves as described (14.93% of positions chosen, split 80.2% / 10.1% / 9.6%), and a single-digit perplexity on unseen text is in the same range as Table 6's numbers.

## Feature-based approach with BERT {§5.3}

Part 1 introduced two ways to reuse a pre-trained model: feature-based (freeze it, use its vectors) and fine-tuning (train everything). Every BERT result so far was fine-tuned. Section 5.3 asks whether frozen BERT features are good too.

> [!PAPER] · §5.3
> [![Section 5.3: all results so far used fine-tuning. However, the feature-based approach, where fixed features are extracted from the pre-trained model, has certain advantages. First, not all tasks can be easily represented by a Transformer encoder architecture. Second, there are major computational benefits to pre-compute an expensive representation of the training data once and then run many experiments with cheaper models on top](https://ishwarj.com/img/papers/bert/p5-feature-why.png/p5-feature-why.png)](/img/papers/bert/p5-feature-why.png)
>
> **What it says:** the feature-based approach "has certain advantages". First, "not all tasks can be easily represented by a Transformer encoder architecture". Second, "major computational benefits": compute the expensive BERT vectors once, then run many experiments "with cheaper models on top".
>
> **What it means:**
>
> Think of a cook who makes one big pot of stock on Sunday and uses it for different soups all week. The feature-based approach runs the expensive BERT once and saves its vectors. Many cheaper models then reuse the saved vectors.
>
> The first advantage matters too. Some tasks do not fit the shape of Figure 4, one encoder plus one small layer, and need their own model on top. In practice, frozen features are often what people use: for search indexes, for clustering, or when many small models reuse the same text.

**Use case.** A team wants to try twenty different classifiers on a million documents. Fine-tuning means twenty full BERT training runs. Feature-based means running BERT once over the million documents, saving the vectors, and training twenty small models on the saved vectors, each in minutes.

> [!PAPER] · §5.3
> [![The two approaches are compared on the CoNLL-2003 Named Entity Recognition task. The input uses a case-preserving WordPiece model and includes the maximal document context provided by the data. Following standard practice it is a tagging task, but no CRF layer is used](https://ishwarj.com/img/papers/bert/p5-feature-setup.png5-feature-setup.png)](/img/papers/bert/p5-feature-setup.png)
> [![We use the representation of the first sub-token as the input to the token-level classifier over the NER label set](https://ishwarj.com/img/papers/bert/p5-feature-setup-b.pngfeature-setup-b.png)](/img/papers/bert/p5-feature-setup-b.png)
> [![To ablate the fine-tuning approach, we apply the feature-based approach by extracting the activations from one or more layers without fine-tuning any parameters of BERT. These contextual embeddings are used as input to a randomly initialized two-layer 768-dimensional BiLSTM before the classification layer](https://ishwarj.com/img/papers/bert/p5-feature-setup2.png-feature-setup2.png)](/img/papers/bert/p5-feature-setup2.png)
>
> **What it says:** the task is CoNLL-2003 named entity recognition. BERT reads text with a "case-preserving WordPiece model" and "the maximal document context". There is no CRF layer, and each word is represented by its **first sub-token**. For the feature-based runs, activations are extracted "without fine-tuning any parameters of BERT" and fed to "a randomly initialized two-layer 768-dimensional BiLSTM".
>
> **What it means:**
>
> These details are what you would need to copy to reproduce the result. Each one is a small, deliberate choice. Names start with capital letters, so the model keeps case. The model also sees the whole document around each sentence, because more context helps to spot a name.
>
> Two more choices keep the comparison simple. There is no CRF, so each word gets its tag on its own (the yellow boxes below explain these terms). And in the feature-based runs, no weight of BERT changes: only the new BiLSTM and the output layer learn.

> [!DEFINITION] CoNLL-2003
> A standard named entity recognition dataset of English news stories (Reuters), from a 2003 shared task. Every word is tagged as part of a person, organisation, location, miscellaneous name, or none.

> [!DEFINITION] Case-preserving (cased)
> A tokenizer that keeps capital letters, so "Apple" and "apple" are different tokens. The `bert-base-uncased` model of earlier parts lowercases everything; for names, capitals are a strong clue, so NER uses the cased model.

> [!DEFINITION] CRF (conditional random field)
> An output layer often put on top of taggers. It scores whole sequences of tags, so it can learn rules like "I-PER cannot follow B-ORG". BERT leaves it out and predicts every tag on its own.

The tags come from the CoNLL-2003 data format, described in its own paper:

> [!PAPER] Tjong Kim Sang and De Meulder (2003), Introduction to the CoNLL-2003 Shared Task · §2.3
> [![The CoNLL-2003 data format: one word per line with its part-of-speech tag, chunk tag and named entity tag; the I-XXX tag is used for words inside a named entity of type XXX, B-XXX marks the first word of a second entity right after another of the same type, following the IOB scheme of Ramshaw and Marcus 1995](https://ishwarj.com/img/papers/bert/p5-conll-format.pngp5-conll-format.png)](/img/papers/bert/p5-conll-format.png)
>
> **What it says:** each line holds a word, its part of speech, its chunk tag and its **named entity tag**. "The I-XXX tag is used for words inside a named entity of type XXX"; when two entities of the same type are next to each other, the first word of the second one gets B-XXX. Four types: persons (PER), organisations (ORG), locations (LOC) and miscellaneous (MISC).
>
> **What it means:**
>
> The tags are a small grammar. "I" means inside a name of a given type, and "B" marks where a new name starts right after another of the same type. So the tags of neighbouring words depend on each other.
>
> That is the kind of rule a CRF can enforce, and BERT's per-token classifier cannot: each word picks its tag alone. The figure below uses the common modern variant, in which every name starts with a B-tag.

<figure class="fig"><svg viewBox="0 0 760 244" role="img" aria-label="A real CoNLL-2003 sentence: Lithuania - Danius Gleveckas ( 13rd ). Each word has a BIO tag (B-LOC, O, B-PER, I-PER, O, O, O). WordPiece splits Danius into two pieces and Gleveckas into four; the tagger uses only the first piece of each word."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">A real CoNLL-2003 dev sentence (#99): one tag per word, one vector per word</text><rect class="box-3" x="14.0" y="40.0" width="95.0" height="28.0" rx="6"/><text class="t-note" x="61.5" y="59.0" text-anchor="middle">B-LOC</text><text class="t-mono" x="61.5" y="92.0" text-anchor="middle">Lithuania</text><rect class="box-on" x="15.0" y="112.0" width="93.0" height="28.0" rx="5"/><text class="t-tick" x="61.5" y="131.0" text-anchor="middle">Lithuania</text><line class="edge-on" x1="61.5" y1="142.0" x2="61.5" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="40.5" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="43.5" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="55.5" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="67.5" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="117.0" y="40.0" width="62.0" height="28.0" rx="6"/><text class="t-muted" x="148.0" y="59.0" text-anchor="middle">O</text><text class="t-mono" x="148.0" y="92.0" text-anchor="middle">-</text><rect class="box-on" x="131.0" y="112.0" width="34.0" height="28.0" rx="5"/><text class="t-tick" x="148.0" y="131.0" text-anchor="middle">-</text><line class="edge-on" x1="148.0" y1="142.0" x2="148.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="127.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="130.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="142.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="154.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box-2" x="187.0" y="40.0" width="100.0" height="28.0" rx="6"/><text class="t-note" x="237.0" y="59.0" text-anchor="middle">B-PER</text><text class="t-mono" x="237.0" y="92.0" text-anchor="middle">Danius</text><rect class="box-on" x="187.0" y="112.0" width="48.0" height="28.0" rx="5"/><text class="t-tick" x="211.0" y="131.0" text-anchor="middle">Dani</text><line class="edge-on" x1="211.0" y1="142.0" x2="211.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="190.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="193.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="205.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="217.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="239.0" y="112.0" width="48.0" height="28.0" rx="5"/><text class="t-muted" x="263.0" y="131.0" text-anchor="middle">##us</text><rect class="box-2" x="295.0" y="40.0" width="208.0" height="28.0" rx="6"/><text class="t-note" x="399.0" y="59.0" text-anchor="middle">I-PER</text><text class="t-mono" x="399.0" y="92.0" text-anchor="middle">Gleveckas</text><rect class="box-on" x="295.0" y="112.0" width="34.0" height="28.0" rx="5"/><text class="t-tick" x="312.0" y="131.0" text-anchor="middle">G</text><line class="edge-on" x1="312.0" y1="142.0" x2="312.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="291.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="294.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="306.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="318.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="333.0" y="112.0" width="57.0" height="28.0" rx="5"/><text class="t-muted" x="361.5" y="131.0" text-anchor="middle">##lev</text><rect class="box-ghost" x="394.0" y="112.0" width="57.0" height="28.0" rx="5"/><text class="t-muted" x="422.5" y="131.0" text-anchor="middle">##eck</text><rect class="box-ghost" x="455.0" y="112.0" width="48.0" height="28.0" rx="5"/><text class="t-muted" x="479.0" y="131.0" text-anchor="middle">##as</text><rect class="box" x="511.0" y="40.0" width="62.0" height="28.0" rx="6"/><text class="t-muted" x="542.0" y="59.0" text-anchor="middle">O</text><text class="t-mono" x="542.0" y="92.0" text-anchor="middle">(</text><rect class="box-on" x="525.0" y="112.0" width="34.0" height="28.0" rx="5"/><text class="t-tick" x="542.0" y="131.0" text-anchor="middle">(</text><line class="edge-on" x1="542.0" y1="142.0" x2="542.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="521.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="524.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="536.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="548.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box" x="581.0" y="40.0" width="86.0" height="28.0" rx="6"/><text class="t-muted" x="624.0" y="59.0" text-anchor="middle">O</text><text class="t-mono" x="624.0" y="92.0" text-anchor="middle">13rd</text><rect class="box-on" x="581.0" y="112.0" width="34.0" height="28.0" rx="5"/><text class="t-tick" x="598.0" y="131.0" text-anchor="middle">13</text><line class="edge-on" x1="598.0" y1="142.0" x2="598.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="577.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="580.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="592.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="604.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="619.0" y="112.0" width="48.0" height="28.0" rx="5"/><text class="t-muted" x="643.0" y="131.0" text-anchor="middle">##rd</text><rect class="box" x="675.0" y="40.0" width="62.0" height="28.0" rx="6"/><text class="t-muted" x="706.0" y="59.0" text-anchor="middle">O</text><text class="t-mono" x="706.0" y="92.0" text-anchor="middle">)</text><rect class="box-on" x="689.0" y="112.0" width="34.0" height="28.0" rx="5"/><text class="t-tick" x="706.0" y="131.0" text-anchor="middle">)</text><line class="edge-on" x1="706.0" y1="142.0" x2="706.0" y2="168.0" marker-end="url(#ah-on)"/><rect class="box-ghost" x="685.0" y="171.0" width="40.0" height="16.0" rx="4"/><g class="mark"><rect class="s1" x="688.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="700.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="712.0" y="174.0" width="10" height="10" rx="2" style="fill-opacity:0.95"/></g><text class="t-muted" x="14.0" y="214.0" text-anchor="start">top: BIO tags (B = beginning of a name, I = inside it, O = outside). middle: WordPiece pieces (bert-base-cased).</text><text class="t-muted" x="14.0" y="232.0" text-anchor="start">bottom: the tagger reads only the vector of the first piece of each word (highlighted); the other pieces are ignored.</text></svg><figcaption>A real CoNLL-2003 dev sentence (#99), "Lithuania - Danius Gleveckas ( 13rd )": one tag per word (B-LOC, O, B-PER, I-PER, O, O, O). WordPiece splits "Danius" into Dani ##us and "Gleveckas" into G ##lev ##eck ##as; the tagger reads only the first piece of each word.</figcaption></figure>

Why would a CRF help? A per-token classifier picks each tag on its own, and can produce impossible sequences. A toy with hand-made scores for three words (rows) and three tags (columns):

```text
scores (rows = words ['met', 'Ada', 'Lovelace'], columns = ['O', 'B-PER', 'I-PER']): 3.0, 0.5, 0.2; 1.2, 1.0, 0.4; 0.3, 0.6, 2.5
per-token argmax: ['O', 'O', 'I-PER']  (sum 6.7; O followed by I-PER is invalid)
best valid sequence: ['O', 'B-PER', 'I-PER']  (sum 6.5)
```

<figure class="fig"><svg viewBox="0 0 760 244" role="img" aria-label="Left: BIO transition rules; O followed by I-PER is not allowed. Right: made-up scores for the words met, Ada, Lovelace. Choosing each word's best tag alone gives O, O, I-PER, which breaks the rule; the best valid sequence is O, B-PER, I-PER."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">Which tag may follow which (BIO rules)</text><text class="t-title" x="330.0" y="22.0" text-anchor="start">Hand-made tag scores for three words</text><text class="t-muted" x="136.0" y="68.0" text-anchor="middle">next tag (column)</text><text class="t-tick" x="92.0" y="86.0" text-anchor="middle">O</text><text class="t-tick" x="136.0" y="86.0" text-anchor="middle">B-PER</text><text class="t-tick" x="180.0" y="86.0" text-anchor="middle">I-PER</text><text class="t-tick" x="62.0" y="120.0" text-anchor="end">O</text><g class="mark"><title>O attends to O</title><rect class="cell" x="71.5" y="95.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>O attends to B-PER</title><rect class="cell" x="115.5" y="95.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><rect class="cell-masked" x="159.5" y="95.5" width="41.0" height="41.0" rx="3"/><line class="edge" x1="172.1" y1="108.1" x2="187.9" y2="123.9"/><line class="edge" x1="187.9" y1="108.1" x2="172.1" y2="123.9"/><text class="t-tick" x="62.0" y="164.0" text-anchor="end">B-PER</text><g class="mark"><title>B-PER attends to O</title><rect class="cell" x="71.5" y="139.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>B-PER attends to B-PER</title><rect class="cell" x="115.5" y="139.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>B-PER attends to I-PER</title><rect class="cell" x="159.5" y="139.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-tick" x="62.0" y="208.0" text-anchor="end">I-PER</text><g class="mark"><title>I-PER attends to O</title><rect class="cell" x="71.5" y="183.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>I-PER attends to B-PER</title><rect class="cell" x="115.5" y="183.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><g class="mark"><title>I-PER attends to I-PER</title><rect class="cell" x="159.5" y="183.5" width="41.0" height="41.0" rx="3" style="fill-opacity:0.62"/></g><text class="t-muted" x="14.0" y="246.0" text-anchor="start">previous tag (row)</text><text class="t-tick" x="440.0" y="56.0" text-anchor="middle">O</text><text class="t-tick" x="530.0" y="56.0" text-anchor="middle">B-PER</text><text class="t-tick" x="620.0" y="56.0" text-anchor="middle">I-PER</text><text class="t-mono" x="390.0" y="88.0" text-anchor="end">met</text><rect class="box-on" x="400.0" y="70.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="440.0" y="89.0" text-anchor="middle">3.0</text><rect class="box" x="490.0" y="70.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="530.0" y="89.0" text-anchor="middle">0.5</text><rect class="box" x="580.0" y="70.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="620.0" y="89.0" text-anchor="middle">0.2</text><text class="t-mono" x="390.0" y="128.0" text-anchor="end">Ada</text><rect class="box-2" x="400.0" y="110.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="440.0" y="129.0" text-anchor="middle">1.2</text><rect class="box-on" x="490.0" y="110.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="530.0" y="129.0" text-anchor="middle">1.0</text><rect class="box" x="580.0" y="110.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="620.0" y="129.0" text-anchor="middle">0.4</text><text class="t-mono" x="390.0" y="168.0" text-anchor="end">Lovelace</text><rect class="box" x="400.0" y="150.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="440.0" y="169.0" text-anchor="middle">0.3</text><rect class="box" x="490.0" y="150.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="530.0" y="169.0" text-anchor="middle">0.6</text><rect class="box-on" x="580.0" y="150.0" width="80.0" height="28.0" rx="6"/><text class="t-val" x="620.0" y="169.0" text-anchor="middle">2.5</text><rect class="box-2" x="400.0" y="196.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="418.0" y="206.0" text-anchor="start">each word on its own: O O I-PER (sum 6.7): invalid</text><rect class="box-on" x="400.0" y="216.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="418.0" y="226.0" text-anchor="start">best valid sequence: O B-PER I-PER (sum 6.5)</text></svg><figcaption>Left: which tag may follow which (an I-PER cannot follow O). Right: picking the best tag per word gives O O I-PER (score 6.7), which breaks the rule; the best valid sequence is O B-PER I-PER (score 6.5). A CRF finds the best valid sequence; BERT's tagger in Section 5.3 does not use one.</figcaption></figure>

> [!DEFINITION] Viterbi decoding
> The algorithm a CRF uses to find the highest-scoring valid tag sequence, by keeping the best partial sequence ending in each tag at each word. It is exact and fast: work proportional to (words) × (tags)².

> [!DEFINITION] First sub-token
> WordPiece can split one word into several pieces (Part 2): "Lovelace" might become `Love` `##lace`. A tagger needs one label per word, so BERT uses only the vector of the first piece and ignores the rest.

> [!DEFINITION] Hidden state (activation)
> The vector a layer outputs for each token. BERT-base has 13 of them per token: the embedding output, then one after each of the 12 layers. "Extracting the activations" means saving some of these vectors.

The six feature choices in Table 7 read different hidden states:

<figure class="fig"><svg viewBox="0 0 760 290" role="img" aria-label="The six feature choices of Table 7, drawn as which hidden states they read: the embeddings alone, one layer, a weighted sum, or the last four layers glued together."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="409.0" y="22.0" text-anchor="middle">which of the 13 hidden states each feature choice uses</text><text class="t-tick" x="229.0" y="44.0" text-anchor="middle">E</text><text class="t-tick" x="259.0" y="44.0" text-anchor="middle">1</text><text class="t-tick" x="289.0" y="44.0" text-anchor="middle">2</text><text class="t-tick" x="319.0" y="44.0" text-anchor="middle">3</text><text class="t-tick" x="349.0" y="44.0" text-anchor="middle">4</text><text class="t-tick" x="379.0" y="44.0" text-anchor="middle">5</text><text class="t-tick" x="409.0" y="44.0" text-anchor="middle">6</text><text class="t-tick" x="439.0" y="44.0" text-anchor="middle">7</text><text class="t-tick" x="469.0" y="44.0" text-anchor="middle">8</text><text class="t-tick" x="499.0" y="44.0" text-anchor="middle">9</text><text class="t-tick" x="529.0" y="44.0" text-anchor="middle">10</text><text class="t-tick" x="559.0" y="44.0" text-anchor="middle">11</text><text class="t-tick" x="589.0" y="44.0" text-anchor="middle">12</text><text class="t-tick" x="204.0" y="72.0" text-anchor="end">Embeddings</text><rect class="box-1" x="216.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="246.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="276.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="306.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="336.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="366.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="396.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="426.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="456.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="486.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="516.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="546.0" y="56.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="576.0" y="56.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="68.0" text-anchor="start">one layer</text><text class="t-muted" x="616.0" y="82.0" text-anchor="start">paper Dev F1 91.0</text><text class="t-tick" x="204.0" y="106.0" text-anchor="end">Second-to-last hidden</text><rect class="box-ghost" x="216.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="246.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="276.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="306.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="336.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="366.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="396.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="426.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="456.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="486.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="516.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-1" x="546.0" y="90.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="576.0" y="90.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="102.0" text-anchor="start">one layer</text><text class="t-muted" x="616.0" y="116.0" text-anchor="start">paper Dev F1 95.6</text><text class="t-tick" x="204.0" y="140.0" text-anchor="end">Last hidden</text><rect class="box-ghost" x="216.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="246.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="276.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="306.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="336.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="366.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="396.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="426.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="456.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="486.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="516.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="546.0" y="124.0" width="26.0" height="26.0" rx="4"/><rect class="box-1" x="576.0" y="124.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="136.0" text-anchor="start">one layer</text><text class="t-muted" x="616.0" y="150.0" text-anchor="start">paper Dev F1 94.9</text><text class="t-tick" x="204.0" y="174.0" text-anchor="end">Weighted sum last four</text><rect class="box-ghost" x="216.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="246.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="276.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="306.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="336.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="366.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="396.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="426.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="456.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="486.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="516.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="546.0" y="158.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="576.0" y="158.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="170.0" text-anchor="start">weighted sum of 4</text><text class="t-muted" x="616.0" y="184.0" text-anchor="start">paper Dev F1 95.9</text><text class="t-tick" x="204.0" y="208.0" text-anchor="end">Concat last four</text><rect class="box-ghost" x="216.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="246.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="276.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="306.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="336.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="366.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="396.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="426.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-ghost" x="456.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-2" x="486.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-2" x="516.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-2" x="546.0" y="192.0" width="26.0" height="26.0" rx="4"/><rect class="box-2" x="576.0" y="192.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="204.0" text-anchor="start">4 x 768 = 3,072 numbers</text><text class="t-muted" x="616.0" y="218.0" text-anchor="start">paper Dev F1 96.1</text><text class="t-tick" x="204.0" y="242.0" text-anchor="end">Weighted sum all 12 layers</text><rect class="box-ghost" x="216.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="246.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="276.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="306.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="336.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="366.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="396.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="426.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="456.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="486.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="516.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="546.0" y="226.0" width="26.0" height="26.0" rx="4"/><rect class="box-3" x="576.0" y="226.0" width="26.0" height="26.0" rx="4"/><text class="t-tick" x="616.0" y="238.0" text-anchor="start">weighted sum of 12</text><text class="t-muted" x="616.0" y="252.0" text-anchor="start">paper Dev F1 95.5</text><text class="t-muted" x="214.0" y="278.0" text-anchor="start">E = embedding output (before any layer); 1 to 12 = outputs of the 12 layers</text></svg><figcaption>The six feature choices of Table 7. A weighted sum learns one weight per layer and adds the layers up; concatenation glues the last four 768-number vectors into one of 3,072 numbers.</figcaption></figure>

> [!PAPER] · §5.3, Table 7
> [![Table 7: CoNLL-2003 NER results. ELMo Dev F1 95.7, Test F1 92.2. CVT Test 92.6. CSE Test 93.1. Fine-tuning: BERT-large Dev 96.6, Test 92.8; BERT-base Dev 96.4, Test 92.4. Feature-based BERT-base, Dev F1: Embeddings 91.0, Second-to-Last Hidden 95.6, Last Hidden 94.9, Weighted Sum Last Four Hidden 95.9, Concat Last Four Hidden 96.1, Weighted Sum All 12 Layers 95.5. Scores are averaged over 5 random restarts](https://ishwarj.com/img/papers/bert/p5-table7.png/bert/p5-table7.png)](/img/papers/bert/p5-table7.png)
>
> **What it says:** three earlier systems (ELMo, CVT, CSE), BERT fine-tuned, and six feature-based BERT-base variants. Scores are entity-level F1, "averaged over 5 random restarts".
>
> **What it means:**
>
> The table has three blocks: earlier systems, BERT fine-tuned, and frozen BERT-base with six ways to pick its vectors. The question is how close the frozen rows get to the fine-tuned ones.
>
> The scores are entity-level F1: a name counts as found only if its type and its whole span are right. Each score is an average over 5 restarts, so small gaps are a little easier to trust than single runs.

> [!PAPER] · §5.3
> [![Results are presented in Table 7. BERT-large performs competitively with state-of-the-art methods. The best performing method concatenates the token representations from the top four hidden layers of the pre-trained Transformer, which is only 0.3 F1 behind fine-tuning the entire model. This demonstrates that BERT is effective for both fine-tuning and feature-based approaches](https://ishwarj.com/img/papers/bert/p5-feature-result.png-feature-result.png)](/img/papers/bert/p5-feature-result.png)
>
> **What it says:** BERT-large "performs competitively with state-of-the-art methods". The best feature-based variant, concatenating the top four layers, is "only 0.3 F1 behind fine-tuning the entire model".
>
> **What it means:**
>
> The best frozen choice is "only 0.3 F1 behind fine-tuning the entire model". So you can freeze BERT and lose very little, at least on this task.
>
> That is a practical result. A frozen BERT can run once and serve many small models, as in the use case above. Note also the careful word "competitively" for BERT-large; the list below shows why the paper chose it.

Reading Table 7 carefully:

- **Fine-tuning:** BERT-large 96.6 Dev / 92.8 Test, BERT-base 96.4 / 92.4. Note the word "competitively": on the **Test** set, CSE (93.1) is higher than BERT-large (92.8). NER is not one of BERT's eleven new records.
- **Embeddings alone: 91.0.** The embedding output is the word-lookup vector plus position and segment, before any attention. It knows nothing about the sentence. Context is worth more than 5 points here.
- **Last hidden 94.9 is worse than second-to-last 95.6.** One common explanation (not tested in the paper): the last layer is shaped most strongly by the pre-training tasks, predicting masked words, so the layer before it holds more general information.
- **Combining layers helps:** weighted sum of the last four 95.9, concatenating them **96.1**. Summing all 12 layers (95.5) is worse than using only the top four.
- **96.1 versus 96.4:** the best frozen choice is 0.3 points behind fine-tuned BERT-base on Dev.

> [!DEFINITION] Weighted sum of layers
> Multiply each layer's vector by a learned number (its weight) and add them up. The result has the same size as one layer (768 numbers). ELMo combined its layers this way. In code: `(softmax(w)[:, None] * layers).sum(0)`.

The frozen-feature tagger of Section 5.3, shape by shape, on "Ada Lovelace met Charles Babbage in London .":

```text
WordPiece tokens               [13]
all hidden states              [13, 13, 768]
first sub-token of each word   [8, 13, 768]
concat last four layers        [8, 3072]
BiLSTM output                  [8, 768]
tag scores                     [8, 9]
BiLSTM layer 1 (input 3072): 2 directions x 4 x (384*3072 + 384*384 + 2*384) = 10,622,976
BiLSTM layer 2 (input 768):  2 directions x 4 x (384*768 + 384*384 + 2*384)  = 3,545,088
classifier 9 x 768 + 9 = 6,921;  tagger total 14,174,985 trainable, BERT-base-cased frozen (108,310,272 parameters never change)
```

The LSTM count: each direction has 4 gates, each gate a matrix from the input (3,072 numbers) and one from its own previous state (384 numbers), plus two bias vectors in the PyTorch layout: $$4 \times (384 \times 3072 + 384 \times 384 + 2 \times 384)$$ per direction.

<figure class="fig"><svg viewBox="0 0 760 228" role="img" aria-label="The feature-based NER tagger. The frozen BERT gives 13 hidden states per word; the last four are concatenated into 3,072 numbers, a two-layer BiLSTM with 384 units per direction turns them into 768, and a 9 by 768 matrix gives one score per tag."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">The frozen-feature tagger of Section 5.3, shape by shape (n = number of words)</text><rect class="box" x="14.0" y="44.0" width="132.0" height="50.0" rx="10"/><text class="t-note" x="80.0" y="66.0" text-anchor="middle">BERT hidden states</text><text class="t-muted" x="80.0" y="84.0" text-anchor="middle">frozen</text><text class="t-mono" x="80.0" y="116.0" text-anchor="middle">n x 13 x 768</text><line class="edge" x1="148.0" y1="69.0" x2="162.0" y2="69.0" marker-end="url(#ah)"/><rect class="box" x="164.0" y="44.0" width="132.0" height="50.0" rx="10"/><text class="t-note" x="230.0" y="66.0" text-anchor="middle">concat last four</text><text class="t-muted" x="230.0" y="84.0" text-anchor="middle">no weights</text><text class="t-mono" x="230.0" y="116.0" text-anchor="middle">n x 3,072</text><line class="edge" x1="298.0" y1="69.0" x2="312.0" y2="69.0" marker-end="url(#ah)"/><rect class="box-4" x="314.0" y="44.0" width="132.0" height="50.0" rx="10"/><text class="t-note" x="380.0" y="66.0" text-anchor="middle">BiLSTM, 2 layers</text><text class="t-muted" x="380.0" y="84.0" text-anchor="middle">14.17M weights</text><text class="t-mono" x="380.0" y="116.0" text-anchor="middle">n x 768</text><line class="edge" x1="448.0" y1="69.0" x2="462.0" y2="69.0" marker-end="url(#ah)"/><rect class="box-3" x="464.0" y="44.0" width="132.0" height="50.0" rx="10"/><text class="t-note" x="530.0" y="66.0" text-anchor="middle">linear W, 9 x 768</text><text class="t-muted" x="530.0" y="84.0" text-anchor="middle">6,921 weights</text><text class="t-mono" x="530.0" y="116.0" text-anchor="middle">n x 9</text><line class="edge" x1="598.0" y1="69.0" x2="612.0" y2="69.0" marker-end="url(#ah)"/><rect class="box" x="614.0" y="44.0" width="132.0" height="50.0" rx="10"/><text class="t-note" x="680.0" y="66.0" text-anchor="middle">softmax</text><text class="t-muted" x="680.0" y="84.0" text-anchor="middle">per word</text><text class="t-mono" x="680.0" y="116.0" text-anchor="middle">9 tag probabilities</text><g class="mark"><rect class="s1" x="164.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="178.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="195.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="209.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="226.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s3" x="240.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="257.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="271.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><text class="t-tick" x="164.0" y="172.0" text-anchor="start">[h9 ; h10 ; h11 ; h12]</text><rect class="box-ghost" x="313.0" y="137.0" width="74.0" height="18.0" rx="4"/><g class="mark"><rect class="s4" x="316.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.52"/></g><g class="mark"><rect class="s4" x="330.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><rect class="s4" x="344.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s4" x="358.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="372.0" y="140.0" width="12" height="12" rx="2" style="fill-opacity:0.61"/></g><text class="t-tick" x="316.0" y="172.0" text-anchor="start">384 forward + 384 backward</text><rect class="box-ghost" x="466.0" y="134.0" width="105.0" height="39.0" rx="5"/><rect class="s3" x="470.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="481.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="492.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="503.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="514.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="525.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="536.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="547.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="558.0" y="138.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="470.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="481.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="492.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="503.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="514.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="525.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="536.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="547.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="558.0" y="149.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="470.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="481.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="492.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="503.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="514.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="525.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="536.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="547.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="558.0" y="160.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><text class="t-tick" x="518.5" y="189.0" text-anchor="middle">9 rows, one per tag</text><text class="t-tick" x="14.0" y="214.0" text-anchor="start">Trainable: 14,174,985 numbers (BiLSTM + classifier). BERT-base-cased: 108,310,272 numbers, never updated.</text></svg><figcaption>The Section 5.3 tagger, shape by shape. BERT's 13 hidden states (n × 13 × 768, frozen) → concatenate the last four (n × 3,072) → a 2-layer BiLSTM with 768 outputs (14.17M weights, trained) → a linear layer to 9 tags (6,921 weights) → softmax per word.</figcaption></figure>
### A smaller re-run of the feature-based experiment

Can we see the same pattern ourselves? I ran a smaller version of the experiment in `bert_part5_ner.py`:

- **Model:** `bert-base-cased`, frozen (no weight of BERT changes).
- **Data:** CoNLL-2003 from the Hugging Face hub (`eriktks/conll2003`). Training: the first **5,000** of the 14,041 training sentences. Evaluation: the whole Dev set, **3,250** sentences (51,362 words). Each sentence is read on its own: this copy of the dataset has no document boundaries, so there is no "maximal document context".
- **Features:** all 13 hidden states, first sub-token of every word, saved once.
- **Classifier:** a 2-layer BiLSTM with 384 units per direction (768 in total, my reading of "768-dimensional"), then a linear layer over the 9 tags; Adam, learning rate 0.001, batch 32, 4 epochs. Each choice trained with **3 seeds**, averaged (the paper used 5).
- **Score:** entity-level F1 on Dev, computed by my own code (an entity counts as correct only if its type and its exact span match).

The heart of the feature extraction:

```python
enc = tok(words, is_split_into_words=True, return_tensors="pt")
with torch.no_grad():                                                  # BERT is frozen
    hs = torch.stack(bert(**enc, output_hidden_states=True).hidden_states, 2)[0]   # (tokens, 13, 768)
wid = enc.word_ids(0)                                                  # which word each token belongs to
first = [t for t, w in enumerate(wid) if w is not None and (t == 0 or wid[t - 1] != w)]
features = hs[first]                                                   # (words, 13, 768): first sub-token only

concat_last_four = features[:, 9:13].flatten(1)                        # (words, 3072)
weighted_last_four = (torch.softmax(w, 0)[None, :, None] * features[:, 9:13]).sum(1)   # w: 4 learned weights
```

The results:

```text
CoNLL-2003: 5000 training sentences used, 3250 dev sentences; labels: ['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']
features: 67634 train words, 51362 dev words, 13 layers x 768 numbers each, 34 s
Embeddings                   dev F1 = 82.66   (seeds: 82.73, 82.42, 82.82; 113 s)
Second-to-last hidden        dev F1 = 89.94   (seeds: 88.72, 90.25, 90.84; 127 s)
Last hidden                  dev F1 = 88.64   (seeds: 88.13, 89.03, 88.77; 161 s)
Weighted sum last four       dev F1 = 90.37   (seeds: 90.08, 90.00, 91.05; 311 s)
Concat last four             dev F1 = 91.17   (seeds: 90.94, 91.41, 91.16; 484 s)
Weighted sum all 12 layers   dev F1 = 90.49   (seeds: 91.28, 89.60, 90.59; 450 s)
```

<figure class="fig"><svg viewBox="0 0 760 342" role="img" aria-label="Dev F1 for the six feature choices: the paper (Table 7) and our smaller re-run. Absolute numbers differ, but in both the embeddings alone are clearly worst and the deep layers are far better."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="s1" x="210" y="10" width="12" height="12" rx="2"/><text class="t-note" x="228.0" y="20.0" text-anchor="start">paper, Table 7 (BiLSTM, document context, 5 runs)</text><rect class="s2" x="210" y="30" width="12" height="12" rx="2"/><text class="t-note" x="228.0" y="40.0" text-anchor="start">our smaller re-run (5,000 sentences, 3 runs)</text><line class="grid" x1="210.0" y1="58" x2="210.0" y2="298"/><text class="t-tick" x="210.0" y="314.0" text-anchor="middle">75</text><line class="grid" x1="298.0" y1="58" x2="298.0" y2="298"/><text class="t-tick" x="298.0" y="314.0" text-anchor="middle">80</text><line class="grid" x1="386.0" y1="58" x2="386.0" y2="298"/><text class="t-tick" x="386.0" y="314.0" text-anchor="middle">85</text><line class="grid" x1="474.0" y1="58" x2="474.0" y2="298"/><text class="t-tick" x="474.0" y="314.0" text-anchor="middle">90</text><line class="grid" x1="562.0" y1="58" x2="562.0" y2="298"/><text class="t-tick" x="562.0" y="314.0" text-anchor="middle">95</text><line class="grid" x1="650.0" y1="58" x2="650.0" y2="298"/><text class="t-tick" x="650.0" y="314.0" text-anchor="middle">100</text><text class="t-tick" x="200.0" y="80.0" text-anchor="end">Embeddings</text><g class="mark"><title>Embeddings: 91.0</title><rect class="s1" x="210" y="63" width="281.6" height="13" rx="3"/></g><text class="t-val" x="497.6" y="74.0" text-anchor="start">91.0</text><g class="mark"><title>Embeddings: 82.66</title><rect class="s2" x="210" y="78" width="134.8" height="13" rx="3"/></g><text class="t-val" x="350.8" y="89.0" text-anchor="start">82.66</text><text class="t-tick" x="200.0" y="120.0" text-anchor="end">Second-to-last hidden</text><g class="mark"><title>Second-to-last hidden: 95.6</title><rect class="s1" x="210" y="103" width="362.6" height="13" rx="3"/></g><text class="t-val" x="578.6" y="114.0" text-anchor="start">95.6</text><g class="mark"><title>Second-to-last hidden: 89.94</title><rect class="s2" x="210" y="118" width="262.9" height="13" rx="3"/></g><text class="t-val" x="478.9" y="129.0" text-anchor="start">89.94</text><text class="t-tick" x="200.0" y="160.0" text-anchor="end">Last hidden</text><g class="mark"><title>Last hidden: 94.9</title><rect class="s1" x="210" y="143" width="350.2" height="13" rx="3"/></g><text class="t-val" x="566.2" y="154.0" text-anchor="start">94.9</text><g class="mark"><title>Last hidden: 88.64</title><rect class="s2" x="210" y="158" width="240.1" height="13" rx="3"/></g><text class="t-val" x="456.1" y="169.0" text-anchor="start">88.64</text><text class="t-tick" x="200.0" y="200.0" text-anchor="end">Weighted sum last four</text><g class="mark"><title>Weighted sum last four: 95.9</title><rect class="s1" x="210" y="183" width="367.8" height="13" rx="3"/></g><text class="t-val" x="583.8" y="194.0" text-anchor="start">95.9</text><g class="mark"><title>Weighted sum last four: 90.37</title><rect class="s2" x="210" y="198" width="270.5" height="13" rx="3"/></g><text class="t-val" x="486.5" y="209.0" text-anchor="start">90.37</text><text class="t-tick" x="200.0" y="240.0" text-anchor="end">Concat last four</text><g class="mark"><title>Concat last four: 96.1</title><rect class="s1" x="210" y="223" width="371.4" height="13" rx="3"/></g><text class="t-val" x="587.4" y="234.0" text-anchor="start">96.1</text><g class="mark"><title>Concat last four: 91.17</title><rect class="s2" x="210" y="238" width="284.6" height="13" rx="3"/></g><text class="t-val" x="500.6" y="249.0" text-anchor="start">91.17</text><text class="t-tick" x="200.0" y="280.0" text-anchor="end">Weighted sum all 12 layers</text><g class="mark"><title>Weighted sum all 12 layers: 95.5</title><rect class="s1" x="210" y="263" width="360.8" height="13" rx="3"/></g><text class="t-val" x="576.8" y="274.0" text-anchor="start">95.5</text><g class="mark"><title>Weighted sum all 12 layers: 90.49</title><rect class="s2" x="210" y="278" width="272.6" height="13" rx="3"/></g><text class="t-val" x="488.6" y="289.0" text-anchor="start">90.49</text><text class="t-tick" x="430.0" y="332.0" text-anchor="middle">entity-level F1 on the CoNLL-2003 Dev set (axis starts at 75)</text></svg><figcaption>Dev F1 for each feature choice: the paper's Table 7 and my smaller re-run. The absolute numbers are lower in my run (fewer training sentences, no document context, a smaller training budget), but the shape agrees.</figcaption></figure>

Lined up against Table 7:

| Feature choice | Paper, Dev F1 (Table 7) | Our re-run, Dev F1 |
|---|---|---|
| Embeddings | 91.0 | 82.66 |
| Second-to-last hidden | 95.6 | 89.94 |
| Last hidden | 94.9 | 88.64 |
| Weighted sum last four | 95.9 | 90.37 |
| Concat last four | **96.1** | **91.17** |
| Weighted sum all 12 layers | 95.5 | 90.49 |

What agrees with the paper:

- **The embeddings alone are clearly worst** (82.66), about 6 to 8.5 points below every choice that uses BERT's layers. Context is what BERT adds.
- **Concatenating the last four layers is best** (91.17), in both the paper and our run.
- **The second-to-last layer beats the last layer** (89.94 versus 88.64), as in the paper (95.6 versus 94.9).
- **Combining the top four layers beats any single layer**, whether summed or concatenated.

What does not agree: in our run the weighted sum of all 12 layers (90.49) edges out the weighted sum of the last four (90.37), the opposite of the paper's order. The gap is 0.12 points, while the three seeds of the 12-layer choice alone range from 89.60 to 91.28, so this run cannot separate those two.

Our absolute numbers are about 5 to 8.5 points lower than the paper's. That is expected, not a contradiction: we trained on 5,000 of the 14,041 sentences, for 4 epochs, with no hyperparameter search, and without the document context the paper adds to every sentence. We also did not fine-tune BERT, so this experiment says nothing about the 96.4 of fine-tuning; it only checks the shape of the feature-based rows.

## How long to pre-train {§C.1}

The last two ablations are in Appendix C. The first asks whether BERT's long pre-training is necessary.

> [!PAPER] · §C.1
> [![Appendix C.1: Figure 5 presents MNLI Dev accuracy after fine-tuning from a checkpoint pre-trained for k steps. Question 1: does BERT really need such a large amount of pre-training (128,000 words per batch times 1,000,000 steps)? Answer: yes, BERT-base achieves almost 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps. Question 2: does MLM pre-training converge slower than LTR pre-training, since only 15% of words are predicted in each batch? Answer: the MLM model does converge slightly slower than the LTR model. However, in terms of absolute accuracy the MLM model begins to outperform the LTR model almost immediately](https://ishwarj.com/img/papers/bert/p5-c1.pngpers/bert/p5-c1.png)](/img/papers/bert/p5-c1.png)
>
> **What it says:** two questions. Does BERT need all that pre-training? "Yes": "almost 1.0% additional accuracy on MNLI" at 1M steps compared to 500k. Does the masked LM learn more slowly, since it only predicts 15% of the words? It does "converge slightly slower", but "begins to outperform the LTR model almost immediately".
>
> **What it means:**
>
> The masked LM looks wasteful. In each sentence, only 15% of the words get a prediction, so 85% give no training signal of their own. A left-to-right model learns from every word.
>
> So two worries are natural: does BERT need such long training, and does it learn more slowly? The answer to both is yes, with an important addition: the masked LM "begins to outperform the LTR model almost immediately". And the long training pays: from 500k to 1M steps, MNLI rises by almost 1 point.

> [!DEFINITION] Checkpoint
> A saved copy of a model's weights at some point during training. Fine-tuning from the checkpoint at step k shows how good the model was after k steps of pre-training.

> [!DEFINITION] Converge
> To settle: a training run converges when more steps no longer change it much. "Converges slower" means it needs more steps to reach its best.

The total pre-training in Question 1 is the paper's own product: 128,000 words per batch × 1,000,000 steps = **128 billion** words processed (with the paper's rounding of 256 × 512 = 131,072 to 128,000, as in Part 3).

> [!PAPER] · §C.1, Figure 5
> [![Figure 5: MNLI Dev accuracy after fine-tuning, starting from model parameters pre-trained for k steps, for k from near 0 to 1,000 thousand. Two curves: BERT-base with the masked LM, rising to about 84, and BERT-base left-to-right, flattening near 82](https://ishwarj.com/img/papers/bert/p5-figure5.pngbert/p5-figure5.png)](/img/papers/bert/p5-figure5.png)
>
> **What it says:** two curves, BERT-base with the masked LM and BERT-base trained left to right, measured at several checkpoints.
>
> **What it means:**
>
> The x-axis is k, the pre-training steps in thousands, and the y-axis is MNLI dev accuracy after fine-tuning. Each point is a separate fine-tuning run, started from the checkpoint saved after k steps.
>
> The figure shows both answers at once. The MLM curve is still rising at 1M steps, and it sits above the LTR curve almost the whole way.

How to read it:

- **Both curves rise**, then flatten. More pre-training always helped, but with smaller gains later on.
- **The MLM curve is still climbing between 500k and 1M steps**: that is the "almost 1.0%" of Question 1.
- **At the very first checkpoint the left-to-right model is ahead**, which fits "converges slightly slower": it gets a training signal from every word, the MLM from only 15%. But the MLM curve passes it straight away and the gap then widens to about 2 points by the end (reading the figure, not a number printed in the paper).

The trade-off is worth it. The MLM sees fewer training signals per batch, but each one uses context from both sides.

How many predictions does each objective get from one batch?

```text
one batch = 256 x 512 = 131,072 tokens; masked LM predicts 15% = 19,661; left-to-right LM predicts every next token, about 131,072
over 1,000,000 steps: 19,661,000,000 masked-LM predictions versus about 131,072,000,000 for LTR (6.67x more)
```

<figure class="fig"><svg viewBox="0 0 760 204" role="img" aria-label="Each square is one token of a batch. The masked LM is trained to predict only the 15 percent that were chosen; a left-to-right model predicts every token."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="14.0" y="22.0" text-anchor="start">Appendix C.1: how many predictions one batch gives</text><text class="t-note" x="14.0" y="50.0" text-anchor="start">masked LM</text><rect class="s1" x="14.0" y="62.0" width="14" height="14" rx="2"/><rect class="s1" x="31.0" y="62.0" width="14" height="14" rx="2"/><rect class="s1" x="48.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="65.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="82.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="99.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="116.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="133.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="150.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="167.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="184.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="201.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="218.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="235.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="252.0" y="62.0" width="14" height="14" rx="2"/><rect class="s1" x="269.0" y="62.0" width="14" height="14" rx="2"/><rect class="s1" x="286.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="303.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="320.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="337.0" y="62.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="14.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="31.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="48.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="65.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="82.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="99.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="116.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="133.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="150.0" y="79.0" width="14" height="14" rx="2"/><rect class="s1" x="167.0" y="79.0" width="14" height="14" rx="2"/><rect class="s1" x="184.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="201.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="218.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="235.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="252.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="269.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="286.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="303.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="320.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="337.0" y="79.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="14.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="31.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="48.0" y="96.0" width="14" height="14" rx="2"/><rect class="s1" x="65.0" y="96.0" width="14" height="14" rx="2"/><rect class="s1" x="82.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="99.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="116.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="133.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="150.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="167.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="184.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="201.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="218.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="235.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="252.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="269.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="286.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="303.0" y="96.0" width="14" height="14" rx="2"/><rect class="s1" x="320.0" y="96.0" width="14" height="14" rx="2"/><rect class="s1" x="337.0" y="96.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="14.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="31.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="48.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="65.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="82.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="99.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="116.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="133.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="150.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="167.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="184.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="201.0" y="113.0" width="14" height="14" rx="2"/><rect class="s1" x="218.0" y="113.0" width="14" height="14" rx="2"/><rect class="s1" x="235.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="252.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="269.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="286.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="303.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="320.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="337.0" y="113.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="14.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="31.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="48.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="65.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="82.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="99.0" y="130.0" width="14" height="14" rx="2"/><rect class="s1" x="116.0" y="130.0" width="14" height="14" rx="2"/><rect class="s1" x="133.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="150.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="167.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="184.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="201.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="218.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="235.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="252.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="269.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="286.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="303.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="320.0" y="130.0" width="14" height="14" rx="2"/><rect class="cell-masked" x="337.0" y="130.0" width="14" height="14" rx="2"/><text class="t-tick" x="14.0" y="164.0" text-anchor="start">15% of tokens: 19,661 predictions per batch</text><text class="t-note" x="394.0" y="50.0" text-anchor="start">left-to-right LM</text><rect class="s2" x="394.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="411.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="428.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="445.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="462.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="479.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="496.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="513.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="530.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="547.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="564.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="581.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="598.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="615.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="632.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="649.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="666.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="683.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="700.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="717.0" y="62.0" width="14" height="14" rx="2"/><rect class="s2" x="394.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="411.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="428.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="445.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="462.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="479.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="496.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="513.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="530.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="547.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="564.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="581.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="598.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="615.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="632.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="649.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="666.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="683.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="700.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="717.0" y="79.0" width="14" height="14" rx="2"/><rect class="s2" x="394.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="411.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="428.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="445.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="462.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="479.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="496.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="513.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="530.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="547.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="564.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="581.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="598.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="615.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="632.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="649.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="666.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="683.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="700.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="717.0" y="96.0" width="14" height="14" rx="2"/><rect class="s2" x="394.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="411.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="428.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="445.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="462.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="479.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="496.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="513.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="530.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="547.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="564.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="581.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="598.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="615.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="632.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="649.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="666.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="683.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="700.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="717.0" y="113.0" width="14" height="14" rx="2"/><rect class="s2" x="394.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="411.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="428.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="445.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="462.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="479.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="496.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="513.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="530.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="547.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="564.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="581.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="598.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="615.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="632.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="649.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="666.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="683.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="700.0" y="130.0" width="14" height="14" rx="2"/><rect class="s2" x="717.0" y="130.0" width="14" height="14" rx="2"/><text class="t-tick" x="394.0" y="164.0" text-anchor="start">every token: about 131,072 per batch</text><text class="t-muted" x="14.0" y="192.0" text-anchor="start">Same batch of 131,072 tokens. The masked LM gets 6.67x fewer training signals, yet overtakes LTR almost at once (Figure 5).</text></svg><figcaption>Training signal per batch. The masked LM predicts 15% of tokens (19,661 per batch); a left-to-right LM predicts almost every token (about 131,072). The masked LM gets 6.67 times fewer training signals, yet Figure 5 shows it overtakes the left-to-right model almost at once.</figcaption></figure>

That is the "converges slightly slower" of Question 2, made concrete: per step the masked LM learns from 6.67 times fewer predictions. Figure 5 shows it is still the better choice.

## Masking strategies {§C.2}

Part 3 explained the 80/10/10 masking recipe and *why* the paper uses it. Appendix C.2 tests whether it was a good choice.

> [!PAPER] · §C.2
> [![Appendix C.2: In Section 3.1, we mention that BERT uses a mixed strategy for masking the target tokens when pre-training with the masked language model objective. The following is an ablation study to evaluate the effect of different masking strategies](https://ishwarj.com/img/papers/bert/p5-c2.pngpers/bert/p5-c2.png)](/img/papers/bert/p5-c2.png)
> [![Note that the purpose of the masking strategies is to reduce the mismatch between pre-training and fine-tuning, as the [MASK] symbol never appears during the fine-tuning stage. We report the Dev results for both MNLI and NER. For NER, we report both fine-tuning and feature-based approaches, as we expect the mismatch will be amplified for the feature-based approach as the model will not have the chance to adjust the representations](https://ishwarj.com/img/papers/bert/p5-c2-why.png/bert/p5-c2-why.png)](/img/papers/bert/p5-c2-why.png)
>
> **What it says:** the mixed strategy exists "to reduce the mismatch between pre-training and fine-tuning, as the [MASK] symbol never appears during the fine-tuning stage". They test MNLI and NER, and for NER also the feature-based approach, because "the mismatch will be amplified for the feature-based approach as the model will not have the chance to adjust the representations".
>
> **What it means:**
>
> Part 3 explained the 80/10/10 recipe, but the main paper only argues for it. Here it is tested.
>
> Why would the mismatch be worse for frozen features? With fine-tuning, every weight of BERT is adjusted on the task's real sentences, so the model can unlearn any habit tied to `[MASK]`. With frozen features, BERT is used exactly as pre-trained.
>
> Suppose its vectors for ordinary, unmasked words are not very informative, because it was never asked to predict an unmasked word. Then nothing can fix that afterwards. So the feature-based NER column is the most sensitive test of the recipe.

> [!DEFINITION] Pre-train / fine-tune mismatch
> A difference between what the model sees during pre-training and what it sees later. During pre-training, inputs contain `[MASK]` tokens; real sentences in a task never do. A model that learned to do its "real work" only at `[MASK]` positions might handle normal text less well.

> [!PAPER] · §C.2, Table 8
> [![Table 8: ablation over different masking strategies. Masking rates MASK, SAME, RND and Dev set results MNLI fine-tune, NER fine-tune, NER feature-based. 80, 10, 10: 84.2, 95.4, 94.9. 100, 0, 0: 84.3, 94.9, 94.0. 80, 0, 20: 84.1, 95.2, 94.6. 80, 20, 0: 84.4, 95.2, 94.7. 0, 20, 80: 83.7, 94.8, 94.6. 0, 0, 100: 83.6, 94.9, 94.6](https://ishwarj.com/img/papers/bert/p5-table8.png/bert/p5-table8.png)](/img/papers/bert/p5-table8.png)
> [![The results are presented in Table 8. MASK means that we replace the target token with the [MASK] symbol for MLM; SAME means that we keep the target token as is; RND means that we replace the target token with another random token. The left part of the table shows the probabilities used during MLM pre-training (BERT uses 80%, 10%, 10%). For the feature-based approach, we concatenate the last 4 layers of BERT as the features](https://ishwarj.com/img/papers/bert/p5-c2-legend.pngrt/p5-c2-legend.png)](/img/papers/bert/p5-c2-legend.png)
>
> **What it says:** for the 15% of tokens chosen for prediction, MASK = replace with `[MASK]`, SAME = keep the token, RND = replace with a random token. Each row is a different split. "BERT uses 80%, 10%, 10%." The feature-based column uses the concatenation of the last four layers, the winner of Table 7.
>
> **What it means:**
>
> Read each row as a recipe for the 15% of tokens chosen for prediction. It says how many become `[MASK]`, how many stay the same, and how many become a random token. BERT's own recipe is the first row.
>
> Each recipe is measured in three ways: MNLI fine-tuned, NER fine-tuned, and NER with frozen features. Six recipes and three measurements give 18 numbers, and the list below reads them row by row.

<figure class="fig"><svg viewBox="0 0 760 320" role="img" aria-label="Table 8 as a picture: six masking recipes (share of chosen tokens replaced by [MASK], kept the same, or replaced by a random token) and their Dev scores."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="20.0" text-anchor="start">what happens to a chosen token</text><text class="t-title" x="450.0" y="20.0" text-anchor="start">Dev results (Table 8)</text><rect class="s1" x="20" y="32" width="12" height="12" rx="2"/><text class="t-note" x="38.0" y="42.0" text-anchor="start">MASK</text><rect class="s3" x="100" y="32" width="12" height="12" rx="2"/><text class="t-note" x="118.0" y="42.0" text-anchor="start">SAME</text><rect class="s2" x="180" y="32" width="12" height="12" rx="2"/><text class="t-note" x="198.0" y="42.0" text-anchor="start">RND</text><text class="t-tick" x="470.0" y="52.0" text-anchor="middle">MNLI</text><text class="t-tick" x="570.0" y="52.0" text-anchor="middle">NER fine-tune</text><text class="t-tick" x="670.0" y="52.0" text-anchor="middle">NER feature-based</text><rect class="s1" x="20.0" y="66" width="240.0" height="24" rx="3"/><text class="t-cell on" x="140.0" y="82.0" text-anchor="middle">80%</text><rect class="s3" x="260.0" y="66" width="30.0" height="24" rx="3"/><text class="t-cell on" x="275.0" y="82.0" text-anchor="middle">10%</text><rect class="s2" x="290.0" y="66" width="30.0" height="24" rx="3"/><text class="t-cell on" x="305.0" y="82.0" text-anchor="middle">10%</text><text class="t-tick t-strong" x="330.0" y="82.0" text-anchor="start">BERT</text><text class="t-val" x="470.0" y="82.0" text-anchor="middle">84.2</text><text class="t-val t-on" x="570.0" y="82.0" text-anchor="middle">95.4</text><text class="t-val t-on" x="670.0" y="82.0" text-anchor="middle">94.9</text><rect class="s1" x="20.0" y="104" width="300.0" height="24" rx="3"/><text class="t-cell on" x="170.0" y="120.0" text-anchor="middle">100%</text><text class="t-val" x="470.0" y="120.0" text-anchor="middle">84.3</text><text class="t-val" x="570.0" y="120.0" text-anchor="middle">94.9</text><text class="t-val t-bad" x="670.0" y="120.0" text-anchor="middle">94.0</text><rect class="s1" x="20.0" y="142" width="240.0" height="24" rx="3"/><text class="t-cell on" x="140.0" y="158.0" text-anchor="middle">80%</text><rect class="s2" x="260.0" y="142" width="60.0" height="24" rx="3"/><text class="t-cell on" x="290.0" y="158.0" text-anchor="middle">20%</text><text class="t-val" x="470.0" y="158.0" text-anchor="middle">84.1</text><text class="t-val" x="570.0" y="158.0" text-anchor="middle">95.2</text><text class="t-val" x="670.0" y="158.0" text-anchor="middle">94.6</text><rect class="s1" x="20.0" y="180" width="240.0" height="24" rx="3"/><text class="t-cell on" x="140.0" y="196.0" text-anchor="middle">80%</text><rect class="s3" x="260.0" y="180" width="60.0" height="24" rx="3"/><text class="t-cell on" x="290.0" y="196.0" text-anchor="middle">20%</text><text class="t-val t-on" x="470.0" y="196.0" text-anchor="middle">84.4</text><text class="t-val" x="570.0" y="196.0" text-anchor="middle">95.2</text><text class="t-val" x="670.0" y="196.0" text-anchor="middle">94.7</text><rect class="s3" x="20.0" y="218" width="60.0" height="24" rx="3"/><text class="t-cell on" x="50.0" y="234.0" text-anchor="middle">20%</text><rect class="s2" x="80.0" y="218" width="240.0" height="24" rx="3"/><text class="t-cell on" x="200.0" y="234.0" text-anchor="middle">80%</text><text class="t-val" x="470.0" y="234.0" text-anchor="middle">83.7</text><text class="t-val t-bad" x="570.0" y="234.0" text-anchor="middle">94.8</text><text class="t-val" x="670.0" y="234.0" text-anchor="middle">94.6</text><rect class="s2" x="20.0" y="256" width="300.0" height="24" rx="3"/><text class="t-cell on" x="170.0" y="272.0" text-anchor="middle">100%</text><text class="t-val t-bad" x="470.0" y="272.0" text-anchor="middle">83.6</text><text class="t-val" x="570.0" y="272.0" text-anchor="middle">94.9</text><text class="t-val" x="670.0" y="272.0" text-anchor="middle">94.6</text><text class="t-muted" x="20.0" y="308.0" text-anchor="start">highlighted: best and worst value in each column</text></svg><figcaption>Table 8 as a picture. Each bar shows what happens to the tokens chosen for prediction. In each results column, the best value is drawn in the accent colour and the worst in red.</figcaption></figure>

Row by row:

1. **80 / 10 / 10 (BERT's recipe):** 84.2, 95.4, 94.9. The best NER scores in both columns.
2. **100 / 0 / 0 (always `[MASK]`):** MNLI 84.3 is fine, but feature-based NER drops to **94.0**, the worst value in that column and 0.9 below BERT's recipe. This is the predicted mismatch: the model only ever had to predict at `[MASK]`, and its frozen vectors for normal words are less useful.
3. **80 / 0 / 20:** 84.1, 95.2, 94.6. Random replacement without any "keep" cases.
4. **80 / 20 / 0:** 84.4, 95.2, 94.7. Interestingly the best MNLI score in the table, 0.2 above BERT's recipe.
5. **0 / 20 / 80 (no `[MASK]` at all):** 83.7, 94.8, 94.6.
6. **0 / 0 / 100 (always random):** **83.6** on MNLI, the worst. Here the model is trained to find and fix wrong words, and 15% of every training input is noise.

> [!PAPER] · §C.2
> [![From the table it can be seen that fine-tuning is surprisingly robust to different masking strategies. However, as expected, using only the MASK strategy was problematic when applying the feature-based approach to NER. Interestingly, using only the RND strategy performs much worse than our strategy as well](https://ishwarj.com/img/papers/bert/p5-c2-result.pngrt/p5-c2-result.png)](/img/papers/bert/p5-c2-result.png)
>
> **What it says:** "fine-tuning is surprisingly robust to different masking strategies." But "using only the MASK strategy was problematic when applying the feature-based approach to NER", and "using only the RND strategy performs much worse than our strategy as well".
>
> **What it means:**
>
> The 80/10/10 recipe is a safe middle ground. Its main value is to protect frozen-feature use: always `[MASK]` gives the worst score there. Always a random token gives the worst MNLI score.
>
> "Surprisingly robust" is an honest phrase. The authors seem to have expected the recipe to matter more when they fine-tune, and it did not. The numbers below show how small the differences are.

Putting numbers on "robust": across all six recipes, MNLI moves within **0.8** points (83.6 to 84.4) and fine-tuned NER within **0.6** (94.8 to 95.4). Feature-based NER moves within **0.9** (94.0 to 94.9), with always-`[MASK]` at the bottom. These are small differences, single numbers without error bars, so the honest summary is: the masking recipe matters little when you fine-tune, and a bit more when you freeze.

One more detail, which the paper does not explain: BERT's own row here scores 84.2 on MNLI, while the BERT-base row of Table 5 scores 84.4. They are probably different training runs, but the paper does not say.

> [!TAKEAWAYS] Key takeaways
> - An **ablation** removes one part, keeps everything else equal, and measures the loss. Section 5 and Appendix C do this for BERT's main design choices.
> - **Bidirectionality is the big one.** Going from the masked LM to left-to-right (Table 5) costs **9.0** points on MRPC and **10.1** F1 on SQuAD. A BiLSTM on top wins back 7.1 on SQuAD but stays 3.6 below BERT-base, and hurts GLUE.
> - **NSP's measured effect is mixed:** 3.5 points on QNLI, but only 0.1 to 0.6 on the other four tasks.
> - The **LTR & No NSP** model has all of BERT's other advantages over GPT (data, batch size, input format, tuned fine-tuning), so its gap to BERT isolates the effect of the pre-training tasks (Appendix A.4).
> - **Bigger is better**, even on small tasks (Table 6). Measured sizes: 45.7M to 335.1M parameters; BERT-base is 109.5M. The two 6-layer models have identical parameter counts, yet 12 heads beat 3.
> - **Frozen BERT features work almost as well as fine-tuning** for NER: concatenating the last four layers gives 96.1 Dev F1 versus 96.4 fine-tuned (Table 7). Our smaller frozen-feature re-run agreed on the shape: embeddings alone worst (82.66), concat last four best (91.17).
> - **Longer pre-training helps** (almost 1.0 MNLI point from 500k to 1M steps), and the masked LM overtakes left-to-right almost immediately despite learning from only 15% of tokens (Figure 5).
> - **Masking recipes barely matter for fine-tuning** (within 0.8 points), but always-`[MASK]` hurts frozen features most (Table 8). 80/10/10 is a safe compromise.

**Next, in Part 6:** what happened after BERT. Later work revisited exactly these ablations (RoBERTa dropped NSP; ALBERT, DistilBERT and ELECTRA changed size and the masked LM itself), BERT's limits, how it is used today, and a summary of the whole paper.

<details>
<summary>Run it yourself</summary>

The scripts behind every measured number in this part:

- [`code/papers/bert/bert_part5.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5.py): the Table 5 differences, the Table 6 parameter counts, and the WikiText-2 perplexity. About 6 minutes on a CPU.
- [`code/papers/bert/bert_part5_ner.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5_ner.py): the frozen-feature NER experiment. It took about 28 minutes on an Apple M5 Pro (64 GB).

```bash
pip install torch transformers datasets
python bert_part5.py        # writes results/part5.json
python bert_part5_ner.py    # writes results/part5_ner.json
```

<figure><img src="/img/papers/bert/part5-run.png" alt="Terminal output of bert_part5.py: Table 5 differences, Table 6 parameter counts matching the formula, and the WikiText-2 masked-LM perplexity of 7.36" loading="lazy" /><figcaption>The real output of bert_part5.py.</figcaption></figure>

<figure><img src="/img/papers/bert/part5-ner-run.png" alt="Terminal output of bert_part5_ner.py: dataset sizes, feature extraction, and Dev F1 for six frozen-feature choices, each averaged over three seeds" loading="lazy" /><figcaption>The real output of bert_part5_ner.py.</figcaption></figure>

</details>

## References

**The BERT paper**

1. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805). NAACL 2019 ([ACL Anthology](https://aclanthology.org/N19-1423/)).
2. Google Research. [BERT code and models](https://github.com/google-research/bert), including `extract_features.py` and the cased models.

**Papers the BERT paper cites in this part**

3. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever. [*Improving Language Understanding by Generative Pre-Training*](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) (OpenAI GPT). OpenAI, 2018.
4. M. E. Peters et al. [*Deep contextualized word representations*](https://arxiv.org/abs/1802.05365) (ELMo, "2018a"). NAACL 2018.
5. M. E. Peters, M. Neumann, L. Zettlemoyer, W. Yih. [*Dissecting Contextual Word Embeddings: Architecture and Representation*](https://arxiv.org/abs/1808.08949) ("2018b", the mixed results on bi-LM size). EMNLP 2018.
6. O. Melamud, J. Goldberger, I. Dagan. [*context2vec: Learning Generic Context Embedding with Bidirectional LSTM*](https://aclanthology.org/K16-1006/). CoNLL 2016.
7. A. Vaswani et al. [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762). NeurIPS 2017.
8. R. Al-Rfou, D. Choe, N. Constant, M. Guo, L. Jones. [*Character-Level Language Modeling with Deeper Self-Attention*](https://arxiv.org/abs/1808.04444). AAAI 2019.
9. E. F. Tjong Kim Sang, F. De Meulder. [*Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition*](https://aclanthology.org/W03-0419/). CoNLL 2003.
10. K. Clark, M.-T. Luong, C. D. Manning, Q. V. Le. [*Semi-Supervised Sequence Modeling with Cross-View Training*](https://arxiv.org/abs/1809.08370) (CVT, Table 7). EMNLP 2018.
11. A. Akbik, D. Blythe, R. Vollgraf. [*Contextual String Embeddings for Sequence Labeling*](https://aclanthology.org/C18-1139/) (CSE, Table 7). COLING 2018.

**Other sources used in this part**

12. L. A. Ramshaw, M. P. Marcus. [*Text Chunking using Transformation-Based Learning*](https://aclanthology.org/W95-0107/) (the IOB tagging scheme). Workshop on Very Large Corpora, 1995.
13. S. Merity, C. Xiong, J. Bradbury, R. Socher. [*Pointer Sentinel Mixture Models*](https://arxiv.org/abs/1609.07843) (WikiText-2). ICLR 2017.
14. Code for this part: [`bert_part5.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5.py), [`bert_part5_ner.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5_ner.py), [`bert_part5_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/bert/bert_part5_math.py).

