ViT, explained · Part 1 of 2 · Covers Title, Abstract, §1

The Big Idea: An Image Is Worth 16×16 Words

The title, the abstract and the introduction of the Vision Transformer paper, line by line: what it means to cut a picture into 16×16 patches and read them like words, why convolutional networks ruled computer vision, what inductive bias, locality and translation equivariance are, and the claim that enough data beats built-in assumptions. With a real run of ViT-B/16 and ResNet-50 on the same picture and the quadratic cost of attention worked out by hand.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. ICLR 2021, 2020. arXiv:2010.11929

On 22 October 2020, twelve researchers at Google posted a paper with an odd title: An Image is Worth 16×16 Words. It took the Transformer, the network design that had taken over language processing, cut photographs into small squares, and fed the squares to the Transformer as if they were words. On the standard photo benchmark, ImageNet, the model matched or beat the best convolutional networks of the day, once it had been trained on enough pictures. The paper was accepted at ICLR 2021, and the model it introduced, the Vision Transformer (ViT), is today the base of most image models and of the image half of models that read both text and pictures.

This series reads the paper slowly, in the paper's own order. Each piece follows the same pattern:

1. The exact lines from the paper, as a highlighted screenshot in a teal box like the one below.
2. A plain-English explanation, with a yellow box for every new word.
3. A picture, and real code when it helps, with its real output.
4. Why it matters, and only then the next piece.

The small Paper §1 tag above each heading tells you which section of the paper you are reading. The screenshots come from the paper's second arXiv version (3 June 2021), the one most people read today. We run every experiment on the model weights Google released, through the Hugging Face transformers library, on an ordinary laptop.

The title#

Let us take the title apart, one piece at a time.

• An image. A digital picture. For this paper, a photograph of one main thing: a cat, a car, a bird.
• 16×16 words. The picture is cut into squares of 16 pixels by 16 pixels. Each square is treated as one "word". A 224×224 picture gives 14×14 = 196 such squares. The paper calls them patches.
• Transformers. The kind of neural network used. It was introduced in 2017 for translating sentences, and by 2020 it was the standard network for text. The paper uses it with almost no changes.
• Image recognition. The task: look at a picture and say what is in it. The paper's version of the task is image classification: choose one label out of a fixed list (for ImageNet, one of 1,000 labels).
• At scale. The key words. The method only works well when the model is trained on very many pictures: 14 million to 300 million. On the "ordinary" 1.3 million pictures of ImageNet, it loses to convolutional networks. This is the paper's main finding, and most of the paper is about it.

The title's analogy is worth drawing out, because the whole paper rests on it. A language model reads a sentence as a sequence of word tokens, each turned into a vector of numbers. ViT reads a picture as a sequence of patch tokens, each turned into a vector of numbers of the same size. After that step, the Transformer cannot tell whether it is reading a sentence or a photo.

Here is the paper's own drawing of the model, which Part 2 explains block by block. We show it now because it makes the title concrete.

The authors and the footnotes#

The order of the names matters less than usual: six of the twelve are marked as equal contributors. When this series says "the authors" it means the team.

There is one footnote on page 1, attached to the last sentence of the abstract.

The abstract, sentence by sentence#

The abstract is one paragraph of four sentences. We read it in two halves.

Sentence 1 says that in language the question was settled. "De-facto standard" means "the standard in practice, even if nobody voted on it". By 2020 every strong language model was a Transformer. The second half of the sentence says vision had not followed.

Sentence 2 names the two ways vision had used attention so far. Both keep the convolutional network in charge. To see why that matters, we need the two words.

The figure shows the real difference between the two operations. A convolution is local (each output looks at a small neighbourhood) and shares weights (the same filter is used everywhere). Self-attention is global (each token can look at every token) and its weights depend on the content, not on the position. Hold on to those two words, local and global; the introduction will come back to them under the name "inductive bias".

Sentence 3 is the thesis. "Pure transformer" means no convolution layers inside the network (Part 2 will show that the very first step, the patch embedding, can be written as one convolution, but the authors mean no CNN body). "Sequences of image patches" is the title again.

Sentence 4 is the method and the result, and it introduces the two-step recipe that the whole paper follows.

If you have read the BERT breakdown, this recipe is familiar: BERT pre-trains on text once and fine-tunes a copy per task. ViT does the same with pictures, with one difference. BERT's pre-training needs no labels (it hides words and guesses them). ViT's pre-training in this paper is supervised: every pre-training picture comes with a class label.

The abstract also says what the paper does not do. It says "image classification tasks", not detection or segmentation.

Let us run it#

Before reading the introduction, it helps to see the object the paper is talking about. We loaded the released ViT-B/16 model (google/vit-base-patch16-224: pre-trained on ImageNet-21k, fine-tuned on ImageNet) and gave it one picture, the test photo from the huggingface/cats-image dataset.

The code is in vit_part1.py:

proc = AutoImageProcessor.from_pretrained('google/vit-base-patch16-224')
vit = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224').eval()
x = proc(images=image, return_tensors='pt').pixel_values            # resized to 224 x 224, scaled to [-1, 1]
C, H, W = x.shape[1:]
P = vit.config.patch_size
N = (H // P) * (W // P)

ViT-B/16 (google/vit-base-patch16-224)
  image tensor: (1, 3, 224, 224)  = batch of 1, 3 channels, 224 x 224 pixels
  numbers in one image: 3 x 224 x 224 = 150,528
  patch size P = 16; patches per side = 224 // 16 = 14; number of patches N = 14 x 14 = 196
  numbers in one patch: 16 x 16 x 3 = 768
  sequence length with the [class] token: 196 + 1 = 197
  hidden size D = 768, layers = 12, heads = 12
  parameters: 86,567,656
  token matrix after the embedding step: (1, 197, 768)  (1 image, 197 tokens, 768 numbers each)
  output scores: (1, 1000)  (one score per ImageNet class)

Every number in that output is one line of arithmetic, and it is worth doing the arithmetic once by hand. The picture is a block of numbers of shape channels × height × width:

where:

• $C = 3$ is the number of colour channels (red, green, blue);
• $H = W = 224$ are the height and width in pixels, after the model's preprocessing resized the photo;
• $150{,}528$ is how many numbers the model receives for one picture.

Cutting it into patches of side $P = 16$ gives

and the sequence the Transformer reads has one extra token at the front, the [class] token (explained in Part 2), so its length is

The 196 × 768 = 150,528 numbers of the patches are exactly the 150,528 numbers of the picture: nothing is lost or added by cutting, only rearranged. The name ViT-B/16 encodes two of these numbers: "B" for Base (the BERT-base size: 12 layers, 768-wide vectors, 12 attention heads) and "/16" for the patch side. The model has 86.6 million parameters, which the paper rounds to 86M in its Table 1.

Now the answer. The model outputs 1,000 scores, one per ImageNet class; a softmax turns them into probabilities that add up to 1:

where $s_k$ is the raw score for class $k$ and $p_k$ is its probability. We printed the five largest, and then did the same with a classic convolutional network, ResNet-50 (microsoft/resnet-50), on the same picture:

rproc = AutoImageProcessor.from_pretrained('microsoft/resnet-50')
resnet = ResNetForImageClassification.from_pretrained('microsoft/resnet-50').eval()
xr = rproc(images=image, return_tensors='pt').pixel_values
with torch.no_grad():
    ro = resnet(pixel_values=xr, output_hidden_states=True)
p_res = torch.softmax(ro.logits[0], -1)

  top-5 classes:                       (ViT-B/16)
     0.9374  Egyptian cat
     0.0384  tabby
     0.0144  tiger cat
     0.0033  lynx
     0.0007  Siamese cat

ResNet-50 (microsoft/resnet-50), a convolutional network
  image tensor: (1, 3, 224, 224)
  parameters: 25,557,032
  feature map after stage 0: (64, 56, 56)  (channels, height, width)
  feature map after stage 1: (256, 56, 56)  (channels, height, width)
  feature map after stage 2: (512, 28, 28)  (channels, height, width)
  feature map after stage 3: (1024, 14, 14)  (channels, height, width)
  feature map after stage 4: (2048, 7, 7)  (channels, height, width)
  top-5 classes:
     0.9416  tiger cat
     0.0344  tabby
     0.0016  remote control
     0.0014  Egyptian cat
     0.0008  jinrikisha

both models agree on the top class: False  (ViT: Egyptian cat, ResNet-50: tiger cat)

Three things to notice:

• Both models are right, in the sense that matters: the picture shows cats, and both put about 94% on a cat class. ImageNet has several house-cat classes (Egyptian cat, tabby, tiger cat, Persian, Siamese), and the picture does not clearly belong to one, so the disagreement is about a fine label, not about the object.
• ResNet-50 is a different shape of machine. Its printed "feature maps" shrink from 56×56 to 7×7 as the picture passes through five stages, while the number of channels grows from 64 to 2,048. That is what a CNN does: it summarises larger and larger neighbourhoods. ViT keeps its 197 tokens of 768 numbers at every layer; nothing shrinks.
• ResNet-50 has 25.6 million parameters, ViT-B/16 86.6 million. The paper's fair comparison is against much bigger CNNs (Part 4). This run is only a first look, not a contest.

Transformers took over language#

Now the introduction. Its first paragraph is about language, not pictures.

Each of the four citations stands for one idea, and the paper leans on all four. We screenshot the sentences it points to.

The paragraph's last sentence is the one to remember: "there is still no sign of saturating performance". In language, the curve of accuracy against model size and data size had not flattened. The rest of the introduction asks whether the same curve exists for pictures, and answers yes, with a catch.

In vision, convolutions still ruled#

Let us take the three sentences in turn.

"Convolutional architectures remain dominant." The three citations are the history of CNNs in three steps. LeCun et al. (1989) trained a convolutional network to read handwritten postcodes, the first practical CNN. Krizhevsky et al. (2012), the AlexNet paper, won the 2012 ImageNet contest by a large margin with a deep CNN trained on GPUs, which started the deep-learning era in vision. He et al. (2016), the ResNet paper, made CNNs of 100+ layers trainable.

"Combining CNN-like architectures with self-attention." The second sentence lists what people had tried. Non-local networks (Wang et al., 2018) added an attention layer inside a CNN for video. DETR (Carion et al., 2020) put a Transformer on top of a CNN's output to detect objects. Stand-alone self-attention (Ramachandran et al., 2019) and Axial-DeepLab (Wang et al., 2020a) went further and replaced every convolution with a local form of attention. The abstract's two categories are these: attention with a CNN, and attention replacing parts of a CNN while "keeping their overall structure in place".

"Specialized attention patterns." The third sentence is the technical heart of the paragraph. Why did the convolution-free models need special patterns at all? Because plain attention over pixels is impossibly expensive. The paper spells this out in its related work (Part 2): "each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes." Let us compute the cost, because the whole design of ViT follows from it.

The quadratic cost of attention over pixels#

Self-attention computes, for every token, a score against every token. With $n$ tokens that is an $n \times n$ table of scores per attention head per layer. The attention series derives the formula; here it is with the shapes written in:

where:

• $Q$, $K$, $V$ are the queries, keys and values, three $n \times d$ tables made from the $n$ token vectors (for ViT-B, $d = 64$ per head);
• $QK^\top$ is the $n \times n$ table of scores: entry $(i, j)$ says how much token $i$ should look at token $j$;
• the softmax turns every row into weights that add up to 1;
• the result is $n \times d$: one new vector per token.

The expensive part is the $n \times n$ table. Its size is

We computed this table for several choices of what a token is (vit_part1_math.py, part (a)):

for name, n in [('32 x 32 CIFAR pixels', 32 * 32), ('224 x 224 ImageNet pixels', 224 * 224),
                ('ViT-B/32 patches + [class] (7x7+1)', 7 * 7 + 1), ('ViT-B/16 patches + [class] (14x14+1)', 14 * 14 + 1),
                ('ViT-B/16 at 384 px (24x24+1)', 24 * 24 + 1)]:
    nn_ = n * n
    bytes_ = nn_ * 4

    tokens are                                n            n x n  float32 memory
    32 x 32 CIFAR pixels                  1,024        1,048,576         4.19 MB
    224 x 224 ImageNet pixels            50,176    2,517,630,976        10.07 GB
    ViT-B/32 patches + [class] (7x7+1)       50            2,500         10.0 kB
    ViT-B/16 patches + [class] (14x14+1)      197           38,809        155.2 kB
    ViT-B/16 at 384 px (24x24+1)            577          332,929         1.33 MB
    pixels / patch tokens = 50,176 / 197 = 254.7 x more tokens, so 64,872 x more attention weights
    ViT-B/16 has 12 heads x 12 layers = 144 attention tables of 197 x 197 = 5,588,496 weights for one image

Read the second row. If every pixel of a 224×224 picture attended to every other pixel, one attention table of one head of one layer would hold $50{,}176^2 \approx 2.5$ billion numbers, about 10 GB. ViT-B has 144 such tables (12 heads × 12 layers), and training needs several copies of each. It cannot be done. Even for the tiny 32×32 pictures of CIFAR, one table has a million entries. That is why the earlier convolution-free models restricted each pixel to a local neighbourhood, and why those restrictions needed "specialized attention patterns" that ran poorly on accelerators.

Now read the fourth row. With 16×16 patches as tokens, $n = 197$ and one table has 38,809 entries. The ratio between the two is

Patches make attention about 65,000 times cheaper than pixels. That single fact is the engineering reason the title says "16×16 words" rather than "50,176 words". And 197 tokens is a very ordinary sequence length for a Transformer (BERT was trained with 512), so the standard, dense, accelerator-friendly attention can be used unchanged. The paper's related work (Part 2) credits Cordonnier et al. (2020) with the patch idea, using 2×2 patches on small pictures; the ViT authors use bigger patches, bigger pictures and far more data.

"Classic ResNet-like architectures are still state of the art." The paragraph closes with the three results that defined the top of the ImageNet leaderboard in 2020: Mahajan et al. (2018) pre-trained ResNeXt CNNs on 3.5 billion Instagram pictures; Xie et al. (2020), Noisy Student, used 300 million unlabelled pictures with a self-training loop; and Kolesnikov et al. (2020), Big Transfer (BiT), pre-trained very large ResNets on the same ImageNet-21k and JFT-300M datasets that ViT will use. All three are CNNs made better with more data, which is the same lever ViT pulls.

The fewest possible modifications#

The four sentences are the whole method. We illustrate each.

"Split an image into patches." The picture is cut into a grid of equal squares and the squares are read off in order, row by row, like text.

"The sequence of linear embeddings of these patches." Each patch is 16 × 16 × 3 = 768 raw pixel numbers. A "linear embedding" multiplies those 768 numbers by a learned matrix to give the patch's token vector. In the paper's notation (Equation 1, which Part 2 covers in full) a flattened patch $\mathbf{x}_p^i$ becomes

where $D$ is the Transformer's width ($D = 768$ for ViT-B, so $\mathbf{E}$ happens to be square, 768 × 768). "Linear" means no non-linearity: it is one matrix multiplication, the simplest possible way to turn a patch into a vector. The output has the same shape as a word embedding in BERT, which is the point.

Our code found this layer in the released model. In the implementation it is stored as a convolution with a 16×16 filter and a stride of 16, which is exactly the same computation as "cut into 16×16 patches and multiply each by $\mathbf{E}$", because a stride-16 filter of size 16 touches each pixel exactly once:

    patch embedding layer: Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
    patch vectors: (196, 768) = 196 patches x 768 numbers, arranged on a 14 x 14 grid

"Treated the same way as tokens (words)." After the embedding, the Transformer sees 196 vectors of 768 numbers. It would see the same thing for a 196-word sentence in BERT. Nothing in the attention layers, the feed-forward layers or the normalisation knows about pixels, rows or columns.

"In supervised fashion." Pre-training is ordinary classification with labels, on a very large labelled set. BERT's label-free pre-training was its big trick; ViT does not need a trick, because Google had hundreds of millions of labelled pictures. (Part 5 shows a first attempt at label-free pre-training for ViT, and Part 6 shows how later papers made it work.)

Mid-sized data, and the inductive-bias explanation#

The fourth paragraph starts at the bottom of page 1 and ends at the top of page 2. It reports the first, disappointing result, and explains it.

Four technical words appear here, and the rest of the series uses all of them, so we take them slowly.

The ResNet-50 run above showed locality in action: its feature maps shrink from 56×56 to 7×7 stage by stage, which is how a CNN gradually lets far-away pixels meet. ViT's 197 tokens all meet in the first layer.

Let us write these two properties down, because they are often confused. A picture is a function $x[i, j]$ of its row $i$ and column $j$. A two-dimensional convolution with a filter $w$ (here 3×3) computes

where:

• $x[i + u, j + v]$ are the 9 pixels under the filter when its top-left corner sits at $(i, j)$;
• $w[u, v]$ are the 9 filter weights, the same at every position (that is weight sharing);
• the sum is one output number; sliding over all $(i, j)$ gives the filtered picture.

Define the shift operator $T_{t}$ that moves a picture $t$ pixels to the right:

Then equivariance and invariance of a function $f$ are

In words: an equivariant $f$ lets the shift pass through (shift then filter equals filter then shift); an invariant $f$ swallows the shift. A convolution is equivariant because the same weights $w$ are used at every position: the sum at output position $(i, j + t)$ of the shifted picture sees exactly the pixels that the sum at $(i, j)$ saw in the original.

Equivariance with real numbers. We built a 12×12 toy picture with a bright 5×5 square in it, applied a 3×3 vertical-edge filter, then shifted the picture 2 pixels right and applied the same filter (vit_part1_math.py, part (b)):

img = torch.zeros(1, 1, 12, 12)
img[0, 0, 3:8, 2:7] = 1.0                                           # a bright 5 x 5 square at rows 3..7, columns 2..6
k = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]]).view(1, 1, 3, 3)   # a 3 x 3 vertical-edge filter (Sobel)
shift = 2
shifted = torch.zeros_like(img)
shifted[..., shift:] = img[..., :-shift]                             # the same square, 2 pixels to the right
out = Fn.conv2d(img, k)                                              # 10 x 10 output (no padding)
out_s = Fn.conv2d(shifted, k)
diff = (out_s[..., shift:] - out[..., :-shift]).abs().max().item()   # move the second output back by 2 and compare

    toy image (12, 12), bright square at rows 3-7, columns 2-6; filter (3, 3) (vertical edges)
    filter weights: [[-1.0, 0.0, 1.0], [-2.0, 0.0, 2.0], [-1.0, 0.0, 1.0]]
    output (10, 10); strongest response in the original at column 0, in the shifted at column 2
    max |shifted output moved back by 2 - original output| = 0.0
    row 5 of the output, original:    4    4    0    0    0   -4   -4    0    0    0
    row 5 of the output, shifted:     0    0    4    4    0    0    0   -4   -4    0
    the same filter weights (9 numbers) are used at every one of the 100 output positions (weight sharing)

Read the two printed rows. The filter fires +4 where the square's left edge is (dark to bright) and -4 where its right edge is (bright to dark). In the shifted picture the very same values appear, two columns to the right. Moving the second output back by two columns and subtracting the first gives a maximum difference of exactly 0.0. That is equivariance: the feature moved with the object, nothing else changed. The filter never had to learn anything about "the square at column 4" versus "the square at column 6"; one set of 9 weights handles every position.

This is why a CNN can learn from fewer examples. A cat in the top-left corner teaches the same filters as a cat in the bottom-right. The network does not need to see cats everywhere; the design guarantees that what it learns in one place applies in every place.

Patch tokens are not shift-equivariant#

Now the other side. ViT's first step cuts the picture along a fixed 16-pixel grid. Shift the picture by less than 16 pixels and every patch now contains a different mix of pixels, so every patch vector changes. Shift by exactly 16 and each patch's content moves, whole, into the neighbouring slot. We measured this on the real cat picture and the real ViT-B/16 patch-embedding layer, using cosine similarity to compare vectors:

where $\mathbf{a}$ and $\mathbf{b}$ are two 768-number patch vectors, the dot is their dot product and $\lVert \cdot \rVert$ is the length. The cosine is 1 for identical directions, 0 for unrelated ones, and negative for opposite ones. For each shift $k$ we compared patch $(i, j)$ of the original with patch $(i, j + \lfloor k/16 \rfloor)$ of the shifted picture, that is, we move the token grid back by the whole number of patches the picture moved (0 for shifts under 16, 1 for a shift of 16):

embed = vit.vit.embeddings.patch_embeddings                          # one 16 x 16 convolution with stride 16 = the linear patch projection
with torch.no_grad():
    ek = embed(shift_right(x, k))[0].view(14, 14, -1)
back = k // P                                                    # whole patches the picture moved
a, b = grid[:, :14 - back], ek[:, back:]                        # compare patch (i, j) of the original with patch (i, j + back) of the shifted
cos = Fn.cosine_similarity(a.reshape(-1, 768), b.reshape(-1, 768), dim=-1)

     shift k  grid moved back  mean cosine  min cosine   ViT-B/16 top-1             ResNet-50 top-1
           0                0        1.000       1.000   Egyptian cat 0.937         tiger cat 0.942
           1                0        0.935       0.643   Egyptian cat 0.888         tiger cat 0.939
           4                0        0.649      -0.219   Egyptian cat 0.861         tiger cat 0.847
           8                0        0.507      -0.353   Egyptian cat 0.939         tiger cat 0.826
          12                0        0.457      -0.446   Egyptian cat 0.847         tiger cat 0.607
          16                1        1.000       1.000   Egyptian cat 0.908         tiger cat 0.737
    for k = 16 compared at the SAME grid position (no moving back): mean cosine = 0.428  (each slot now holds its left neighbour)

Two lessons sit in this table.

• The patch tokens are not equivariant. A 1-pixel shift, invisible to a person, already changes the average patch vector's cosine to 0.935 and the worst one to 0.643. By 12 pixels the mean is 0.457 and some vectors point in the opposite direction. Only a shift of exactly one patch (16 pixels) restores a cosine of 1.000, and then only because we moved the token grid back by one slot. Compared at the same slot, the 16-pixel shift gives 0.428. A convolution has none of this: its output moved by exactly 2 pixels and changed by exactly 0.0.
• ViT still gets the answer right. At every shift its top class stays "Egyptian cat" with probability between 0.847 and 0.939. The model was never told that shifts do not matter; it learned it, from millions of pictures in which cats sat in different places. That is the paper's argument in one table: what the CNN has by design, the Transformer can learn from data, if there is enough data. ResNet-50 also keeps "tiger cat" throughout, as expected for a CNN.

An honesty note on the ResNet column: its confidence drops to 0.607 at a 12-pixel shift and 0.737 at 16. A real ResNet is not perfectly equivariant either, because its strided layers and pooling sample the picture on coarse grids, and our shift fills the strip that enters on the left by repeating the edge column, which adds a small artificial band. The toy filter above is exactly equivariant; a whole ResNet is only approximately so. The qualitative point stands: both networks keep their answer under shifts that scramble every ViT patch vector.

Large scale training trumps inductive bias#

The two datasets named here are the paper's fuel.

"14M-300M images" refers to these two: 14 million for ImageNet-21k, 303 million for JFT-300M. Against them, the 1.3 million pictures of ImageNet are "mid-sized", which is a startling thing to call the dataset that defined the field for a decade.

The paper proves the sketch in Section 4.3, which Part 4 reads in full. Two sentences from it are worth seeing now, because they are the evidence behind "the picture changes".

The exact numbers are in the paper's Table 5 (appendix), which Part 4 redraws in full. Here is the table, and the ImageNet rows for the two 16-pixel-patch models as a chart.

Now the four headline numbers. They all belong to the paper's biggest model, ViT-H/14 (632 million parameters, 14-pixel patches), pre-trained on JFT-300M.

These numbers are compared with the best CNNs in Table 2 of the paper (Part 4). The short version: ViT-H/14 beat the Big Transfer ResNet and Noisy Student on every one of these benchmarks, and did so with less pre-training compute. Part 4 goes through every row and every error bar.

Who's who: the papers the introduction is talking to#

The introduction cites sixteen earlier works in five paragraphs. Here they are in one place, so the rest of the series can refer back to them. Each is listed in full in the references at the end of this part.

Next, in Part 2: the related work the paper builds on (local attention, Sparse Transformers, iGPT, and the 2×2-patch model of Cordonnier et al.), and then the model itself: Figure 1 block by block, Equations 1 to 4 with the real numbers of ViT-B/16 (the patch embedding, the class token, the position embeddings, one encoder layer), the Appendix A formulas, and where the 86 million parameters live.

Run it yourself

The two scripts behind this part are code/papers/vit/vit_part1.py (the cat picture through ViT-B/16 and ResNet-50) and code/papers/vit/vit_part1_math.py (the quadratic cost table, the equivariance check on a toy convolution, and the patch-shift experiment). Both run on a laptop CPU in under a minute each and download google/vit-base-patch16-224 (about 350 MB) and microsoft/resnet-50 (about 100 MB) the first time.

bashCopy

pip install torch transformers datasets pillow
python vit_part1.py         # shapes, patch arithmetic, top-5 of both models; writes results/part1.json
python vit_part1_math.py    # quadratic cost, equivariance, patch shifts; writes results/part1_math.json

References#

The ViT paper

1. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT). ICLR 2021 (OpenReview). arXiv:2010.11929, version 2 (June 2021), which is the version shown in the screenshots.
2. Google Research. Vision Transformer code and pre-trained models. GitHub, 2020.

Papers the ViT paper cites in this part

3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin. Attention Is All You Need (the Transformer). NeurIPS 2017.
4. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (BERT). NAACL 2019.
5. T. B. Brown et al. Language Models are Few-Shot Learners (GPT-3). NeurIPS 2020.
6. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, Z. Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (GShard). ICLR 2021.
7. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation 1(4), 1989.
8. A. Krizhevsky, I. Sutskever, G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks (AlexNet). NeurIPS 2012.
9. K. He, X. Zhang, S. Ren, J. Sun. Deep Residual Learning for Image Recognition (ResNet). CVPR 2016.
10. X. Wang, R. Girshick, A. Gupta, K. He. Non-local Neural Networks. CVPR 2018.
11. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko. End-to-End Object Detection with Transformers (DETR). ECCV 2020.
12. P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, J. Shlens. Stand-Alone Self-Attention in Vision Models. NeurIPS 2019.
13. H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, L.-C. Chen. Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation ("Wang et al., 2020a" in the paper). ECCV 2020.
14. D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, L. van der Maaten. Exploring the Limits of Weakly Supervised Pretraining. ECCV 2018.
15. Q. Xie, M.-T. Luong, E. Hovy, Q. V. Le. Self-training with Noisy Student improves ImageNet classification (Noisy Student). CVPR 2020.
16. A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, N. Houlsby. Big Transfer (BiT): General Visual Representation Learning (BiT). ECCV 2020.
17. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR 2009.
18. C. Sun, A. Shrivastava, S. Singh, A. Gupta. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (JFT-300M). ICCV 2017.

Other sources used in this part

19. X. Zhai, J. Puigcerver, A. Kolesnikov, et al. A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark (VTAB). arXiv 2019. The benchmark behind the 77.63% number.
20. L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, A. van den Oord. Are we done with ImageNet? (ImageNet-ReaL). arXiv 2020. The cleaned-up labels behind the 90.72% number.
21. A. Krizhevsky. The CIFAR-10 and CIFAR-100 datasets. University of Toronto, 2009.
22. Hugging Face model cards: google/vit-base-patch16-224 and microsoft/resnet-50; the sample picture is the huggingface/cats-image dataset.
23. Code for this part: vit_part1.py and vit_part1_math.py.