Transformer, read beside the paper

The paper and its explanation side by side.

• Part 1: The Big Idea: Attention Is All You Need

  Why the best translation models of 2016 were slow to train, and the claim that attention alone can replace them. We read the title, the abstract and the first two sections, and we run a released Transformer that translates English into German.

• Part 2: The Architecture: Stacks, Feed-Forward Layers and Positions

  How the Transformer is built, block by block: the encoder and decoder stacks, the add and norm step, the feed-forward layer, the embeddings and the sine waves that give each word its place. We open a released English to German model, rebuild each piece in our own Python code and check it against the model.

• Part 3: Attention: Queries, Keys and Values

  How one attention step works, why the scores are divided by the square root of the key size, and why the model runs eight heads side by side. We write attention and multi-head attention from scratch in Python, match them with the PyTorch library and with a released translation model, and show three common bugs.

• Part 4: Why Self-Attention: Cost, Steps and Paths

  Section 4 compares self-attention with recurrent and convolutional layers on three measures: work per layer, steps that must wait for each other, and the distance a signal must travel. We compute every column of Table 1 in Python, check the counts with a FLOP counter, and build the restricted attention window in code.

• Part 5: Training: Data, Schedule, Adam and Regularization

  How the Transformer was trained: the data, the batches, eight GPUs for twelve hours, Adam with a warm-up learning rate, dropout and label smoothing. We write each rule in Python, check it against PyTorch, and train a tiny Transformer step by step.

• Part 6: Results, Variations and Impact

  What the Transformer scored, what it cost to train, which parts of it matter, and how it did on a second task. We write BLEU, beam search, checkpoint averaging and the size count in Python, and check each one against a library.

• Part 7: Build It from Scratch: The Whole Transformer in PyTorch

  We write the whole Transformer of the paper in readable PyTorch, one small class per block, and check each block against PyTorch's own layers. Then we load a released translation model into our code, get the same translations, and train a small model on a laptop.