# BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/bert/
Updated: 2026-10-21
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/bert/index.md


## Why this paper

Before BERT, a model that understood language had to be built again for almost every task. BERT showed a simpler way: teach one model how language works by letting it fill in hidden words in billions of words of ordinary text, then adapt that same model to each task by adding one small layer. It set new best results on eleven tasks at once, won the Best Long Paper award at NAACL 2019, and within a year Google wrote that it was using BERT to understand searches. Most of the ideas you meet in modern language models (pre-training, fine-tuning, `[CLS]` vectors, sentence embeddings) are easiest to understand here, where they first came together.

The paper is short (nine pages plus an appendix), but dense. This breakdown reads it slowly, in its own order, and checks its claims with real code on the model the authors released.

## How the six parts are organised

**Before Part 1 (optional): [Before and After BERT](https://ishwarj.com/papers/bert/background-before-and-after-bert/).** The road to BERT, from counting word pairs to word vectors, recurrent networks, attention and pre-training, and what came after it: the benchmark curves, RoBERTa, T5, GPT-3, scaling laws and ModernBERT.

Each part follows the paper's own sections:

1. **[The Big Idea: Reading in Both Directions](https://ishwarj.com/papers/bert/part-1-the-big-idea/)**: the title, the Abstract and §1. Why one-directional language models are a limit, and BERT's fix.
2. **[Inside BERT: Architecture and Input](https://ishwarj.com/papers/bert/part-2-architecture-and-input/)**: §2 and §3. The related work, the model's size and shape, and how text becomes the numbers BERT reads.
3. **[Pre-training: Masked LM and Next Sentence Prediction](https://ishwarj.com/papers/bert/part-3-pre-training/)**: §3.1, Appendix A.1 and A.2. The two pre-training tasks, the data and the training recipe.
4. **[Fine-tuning and Results: GLUE, SQuAD and SWAG](https://ishwarj.com/papers/bert/part-4-fine-tuning-and-results/)**: §3.2, §4, Appendix A.3, A.5 and B.1. How one model is adapted to each task, and every result table.
5. **[Ablations: What Really Matters](https://ishwarj.com/papers/bert/part-5-ablations/)**: §5, Appendix A.4, C.1 and C.2. The experiments that take BERT apart to see which pieces matter.
6. **[After BERT: Impact, Limits and Summary](https://ishwarj.com/papers/bert/part-6-impact-and-summary/)**: §6 and later work. What came next, what BERT cannot do, and the whole paper on one page.

## How to read the boxes

- A **teal "From the paper" box** shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, **What it says** gives the main words of the paper. **What it means** then explains those lines, and what the authors mean but do not write. When the box shows a different paper, the box gives the name of that paper.
- The small **Paper §3.1** tag above a heading tells you which section of the paper you are reading, so you can follow along in the PDF.
- A **green "Key takeaways" box** ends every part with what to remember.

Code blocks show real code, and the output shown under them is what that code actually printed. The scripts are in [`code/papers/bert`](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/bert).

## What you need to know first

Nothing. Every term is explained when it first appears. BERT is built on **attention**, the mechanism that lets each word look at the other words; the paper only needs the idea, but if you want to see how attention works inside, the [attention series](https://ishwarj.com/writings/attention-1-self-attention/) explains it from zero.

