Transformer, explained

The 2017 paper that replaced recurrence with attention and gave us the Transformer, the design behind every large language model today. Seven parts, with the paper's own text, worked equations, the whole model written from scratch in Python, and real code on a released translation model.

Attention Is All You Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. NeurIPS 2017, 2017. arXiv:1706.03762

What you will learn

  • Why recurrent networks were slow to train, and how attention removes the step-by-step loop
  • The encoder and the decoder, with every shape of the base model
  • Scaled dot-product attention and multi-head attention, worked by hand and on real weights
  • Why the paper divides by the square root of the key size, and what the mask in the decoder does
  • Sine and cosine position encodings, and why they let the model read word order
  • The training recipe: the learning-rate warm-up, dropout and label smoothing
  • What the results and the model variations show, and what the paper started
  • How to write the whole Transformer yourself in Python, check it against a released model, and train a small one

Why this paper

Before 2017, the best translation systems read a sentence one word at a time. Each step waited for the step before it, so training could not use the full power of a GPU. This paper removed that loop. Its model, the Transformer, lets every word look at every other word in one step, through a mechanism called attention.

The Transformer translated better than the systems before it and trained in a fraction of the time. The same design now runs every large language model, many image models and speech models.

How the seven parts are organised

Before Part 1 (optional): The Road to the Transformer. Recurrent networks, encoder-decoder translation, and the first attention mechanism.

Each part follows the paper's own sections:

  1. The Big Idea: the title, the abstract, §1 and §2. What was wrong with recurrence, and the claim that attention alone is enough.
  2. The Architecture: §3, Figure 1, §3.1, §3.3, §3.4 and §3.5. The encoder and decoder stacks, residual connections, layer normalisation, the feed-forward network, embeddings and position encodings.
  3. Attention: §3.2 and Figure 2. Scaled dot-product attention, multi-head attention, masking, and the three ways the model uses attention.
  4. Why Self-Attention: §4 and Table 1. Cost per layer, steps in sequence and path length, against recurrent and convolutional layers.
  5. Training: §5. The data, batching, hardware, the learning-rate schedule, dropout and label smoothing.
  6. Results, Variations and Impact: §6, §7, Tables 2 to 4 and the attention pictures of the appendix. What the paper showed, and what came after.
  7. Build It from Scratch: the whole Transformer in about 300 lines of PyTorch, one block at a time. We load the released model's weights into our own code, get the same translation, and train a small Transformer on your laptop.

How to read the boxes

  • A teal "From the paper" box shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, What it says gives the main words of the paper. What it means then explains those lines, and what the authors mean but do not write. When the box shows a different paper, the box gives the name of that paper.
  • A yellow definition box explains a technical word the first time it occurs. If you know the word, skip the box.
  • The small Paper §3 tag above a heading tells you which section of the paper you read.
  • A green "Key takeaways" box ends each part with the facts to remember.

Code blocks show real code, and the output under a code block is what that code printed. The scripts are in code/papers/transformer.

What you need to know first

You need to know what a vector and a matrix are, and how to multiply them. Every other term is explained when it first occurs. The attention series explains self-attention from zero, and is a good companion to Part 3.

Parts

  • Background: The Road to the Transformer

    How machines translated sentences before 2017: recurrent networks, LSTMs, sequence to sequence and the first attention. Each idea fixed a problem, and each one left the problem that the Transformer solves. With small runs you can check.

  • Part 1: The Big Idea: Attention Is All You Need

    Why the best translation models of 2016 were slow to train, and the claim that attention alone can replace them. We read the title, the abstract and the first two sections, and we run a released Transformer that translates English into German.

  • Part 2: The Architecture: Stacks, Feed-Forward Layers and Positions

    How the Transformer is built, block by block: the encoder and decoder stacks, the add and norm step, the feed-forward layer, the embeddings and the sine waves that give each word its place. We open a released English to German model, rebuild each piece in our own Python code and check it against the model.

  • Part 3: Attention: Queries, Keys and Values

    How one attention step works, why the scores are divided by the square root of the key size, and why the model runs eight heads side by side. We write attention and multi-head attention from scratch in Python, match them with the PyTorch library and with a released translation model, and show three common bugs.

  • Part 4: Why Self-Attention: Cost, Steps and Paths

    Section 4 compares self-attention with recurrent and convolutional layers on three measures: work per layer, steps that must wait for each other, and the distance a signal must travel. We compute every column of Table 1 in Python, check the counts with a FLOP counter, and build the restricted attention window in code.

  • Part 5: Training: Data, Schedule, Adam and Regularization

    How the Transformer was trained: the data, the batches, eight GPUs for twelve hours, Adam with a warm-up learning rate, dropout and label smoothing. We write each rule in Python, check it against PyTorch, and train a tiny Transformer step by step.

  • Part 6: Results, Variations and Impact

    What the Transformer scored, what it cost to train, which parts of it matter, and how it did on a second task. We write BLEU, beam search, checkpoint averaging and the size count in Python, and check each one against a library.

  • Part 7: Build It from Scratch: The Whole Transformer in PyTorch

    We write the whole Transformer of the paper in readable PyTorch, one small class per block, and check each block against PyTorch's own layers. Then we load a released translation model into our code, get the same translations, and train a small model on a laptop.