# Fine-tuning, Higher Resolution and the Setup

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/vit/part-3-fine-tuning-and-setup/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vit/part-3-fine-tuning-and-setup.md


[Part 2](https://ishwarj.com/papers/vit/part-2-the-model/) built ViT from pixels to a class label. A model is only half of a paper, though. The other half is how it is trained, adapted and measured, and that half decides every number in the results.

## Fine-tuning: a new head for a new task {§3.2}

> [!PAPER] · §3.2
> [![The opening of Section 3.2, Fine-tuning and higher resolution: typically, we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks. For this, we remove the pre-trained prediction head and attach a zero-initialized D by K feedforward layer, where K is the number of downstream classes](https://ishwarj.com/img/papers/vit/p3-finetune-head.png3-finetune-head.png)](/img/papers/vit/p3-finetune-head.png)
>
> **What it says:** "we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks". To do so, "we remove the pre-trained prediction head and attach a zero-initialized $$D \times K$$ feedforward layer, where $$K$$ is the number of downstream classes."
>
> **What it means:**
>
> Think of a doctor who trained for years in a big hospital and then joins a small clinic. The clinic does not retrain the doctor from zero. It keeps everything the doctor knows and only teaches the clinic's own forms. The pre-trained body of ViT is the doctor; the head is the form.
>
> The old head answered one question: which of the 21,843 ImageNet-21k classes (or 18,000 JFT classes) is this? A new task asks a different question, with $$K$$ answers, so the old head is useless and is thrown away. The new head is one matrix of $$D \times K$$ numbers, 768 × 10 for ViT-B on a 10-class task, and every one of them starts at zero.
>
> The word "zero-initialized" looks like a detail, but it has a clean effect that we can measure. At the first step every class gets the same score, and the body of the network receives no gradient at all. We check this below.

> [!DEFINITION] Downstream task
> The task you actually care about, after pre-training: classifying pets, flowers or medical scans. "Downstream" because it comes after the big pre-training run, and usually has far fewer labelled pictures. [Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/) defines pre-training and fine-tuning.

> [!DEFINITION] Zero initialisation
> Starting a layer with every weight and bias equal to 0, instead of small random numbers. A zero layer outputs 0 for every input, so at the start it says nothing and adds no noise.

The released ViT-B/16 checkpoint pre-trained on ImageNet-21k shows the old head. It ends in a "pre-logits" layer, a $$768 \times 768$$ matrix with a tanh (Part 2 met it as the library's "pooler"). The 21,843-class layer on top of it was not released. Appendix B.1.1 says what is removed:

> [!PAPER] · Appendix B.1.1
> [![Appendix B.1.1: when transferring ViT models to another dataset, we remove the whole head (two linear layers) and replace it by a single, zero-initialized linear layer outputting the number of classes required by the target dataset. We found this to be a little more robust than simply re-initializing the very last layer](https://ishwarj.com/img/papers/vit/p3-b11-head.pngvit/p3-b11-head.png)](/img/papers/vit/p3-b11-head.png)
>
> **What it says:** "we remove the whole head (two linear layers) and replace it by a single, zero-initialized linear layer". This was "a little more robust than simply re-initializing the very last layer."
>
> **What it means:**
>
> The pre-training head has two layers: the pre-logits layer and the class layer. A common habit is to keep the first and replace only the second. The paper removes both, so the new task reads the [class] token's representation directly, as Equation 4 of Part 2 defines it.
>
> "A little more robust" is an honest phrase. The authors do not claim it is much better, only that it worked a little more reliably. One way to read it: the pre-logits layer was shaped to serve 21,843 classes. For a new task, it is one more layer that must unlearn that shape.

`vit_part3_finetune.py` loads the ImageNet-21k checkpoint into a 10-class classifier, sets the new head to zero and runs one batch of 32 Imagenette pictures through it. (Imagenette is a small public set of 10 ImageNet classes, described in the fine-tuning run below.)

```python
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224-in21k', num_labels=10)
torch.nn.init.zeros_(model.classifier.weight); torch.nn.init.zeros_(model.classifier.bias)
loss = F.cross_entropy(model(pixel_values=x).logits, y); loss.backward()
```

```text
== 0. data: Imagenette, 9469 training and 3925 validation pictures, 10 classes; device mps ==
  loaded google/vit-base-patch16-224-in21k into a 10-class classifier
  the pre-logits layer (768 x 768 + tanh) is dropped: model.vit.pooler is None
  new head: Linear(in_features=768, out_features=10, bias=True) with every weight and bias set to 0 (7,690 numbers)

== 1. step 0 with the zero head ==
  logits of the first picture: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]  -> softmax gives 0.1 to each of the 10 classes
  loss = -ln(0.1) = ln 10 = 2.302585;  measured 2.302585
  gradient of the head weight by hand, (softmax - one_hot)^T y / batch: norm 1.2152; autograd: 1.2152; max |difference| 0.0e+00
  gradient reaching the body (all 85.8M other numbers): norm 0.0  (W_head^T (softmax - one_hot) = 0 because W_head = 0)
  for comparison, a random head (normal, std 0.02): loss 2.3175, gradient reaching the body: norm 0.561, logits of picture 1 range -0.13 to 0.13
```

The loss of the first step is exactly $$\ln 10$$, because ten equal scores give each class a probability of 0.1. The more surprising line is the last one: the body gets a gradient of exactly 0. The chain rule explains it. The gradient that flows back into the [class] token is the head's weight matrix times the error of each class:

$$
\frac{\partial \mathcal{L}}{\partial \mathbf{y}} = W_{\text{head}}^\top\,(\mathbf{p} - \mathbf{t}) = \mathbf{0} \quad \text{when } W_{\text{head}} = \mathbf{0}
$$

Here $$\mathbf{y}$$ is the image representation (768 numbers) and $$W_{\text{head}}$$ the new $$10 \times 768$$ head. $$\mathbf{p}$$ holds the 10 predicted probabilities, and $$\mathbf{t}$$ is the one-hot target (1 for the right class, 0 for the rest). The head itself does get a gradient, $$(\mathbf{p} - \mathbf{t})\,\mathbf{y}^\top$$ averaged over the batch, because $$\mathbf{y}$$ is not zero. So step 1 changes only the head. From step 2 on, the head is no longer zero and the body starts to move.

A random head does the opposite. Its scores are noise (−0.13 to 0.13 here), and its loss starts above $$\ln 10$$, at 2.3175. That noise sends a gradient of size 0.561 into all 85.8 million pre-trained numbers on the very first step. The zero head lets the pre-trained body wait until the head has learned something sensible.

<figure class="fig"><svg viewBox="0 0 800 362" role="img" aria-label="The head swap: the ImageNet-21k checkpoint ends in a pre-logits layer and a 21,843-class layer, both removed; fine-tuning attaches one zero-initialised 768 by 10 layer. At step 0 the loss is ln 10 and only the head receives a gradient."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Fine-tuning swaps the head: two pre-trained layers out, one zero layer in</text><text class="t-note" x="170.0" y="56.0" text-anchor="middle">pre-trained on ImageNet-21k</text><rect class="box-1" x="70.0" y="270.0" width="200.0" height="34.0" rx="6"/><text class="t-tick" x="170.0" y="291.5" text-anchor="middle">encoder: 12 layers, 197 × 768</text><line class="edge" x1="170.0" y1="268.0" x2="170.0" y2="240.0" marker-end="url(#ah)"/><rect class="box" x="70.0" y="206.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="170.0" y="226.5" text-anchor="middle">y = LN(z_L⁰): 768</text><line class="edge" x1="170.0" y1="204.0" x2="170.0" y2="176.0" marker-end="url(#ah)"/><rect class="box-ghost" x="70.0" y="142.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="170.0" y="162.5" text-anchor="middle">pre-logits 768×768, tanh</text><line class="edge" x1="170.0" y1="140.0" x2="170.0" y2="112.0" marker-end="url(#ah)"/><rect class="box-ghost" x="70.0" y="78.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="170.0" y="98.5" text-anchor="middle">21,843-class layer</text><line class="bad-line" x1="64.0" y1="70.0" x2="276.0" y2="182.0"/><line class="bad-line" x1="276.0" y1="70.0" x2="64.0" y2="182.0"/><text class="t-muted" x="170.0" y="194.0" text-anchor="start"></text><text class="t-bad" x="282.0" y="116.0" text-anchor="start">removed</text><text class="t-note" x="560.0" y="56.0" text-anchor="middle">fine-tuned on a task with K = 10 classes</text><rect class="box-1" x="460.0" y="270.0" width="200.0" height="34.0" rx="6"/><text class="t-tick" x="560.0" y="291.5" text-anchor="middle">the same encoder</text><line class="edge" x1="560.0" y1="268.0" x2="560.0" y2="240.0" marker-end="url(#ah)"/><rect class="box" x="460.0" y="206.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="560.0" y="226.5" text-anchor="middle">y: 768</text><line class="edge" x1="560.0" y1="204.0" x2="560.0" y2="176.0" marker-end="url(#ah)"/><rect class="box-on" x="460.0" y="142.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="560.0" y="162.5" text-anchor="middle">W_head 768 × 10 = 0, b = 0</text><line class="edge" x1="560.0" y1="140.0" x2="560.0" y2="112.0" marker-end="url(#ah)"/><rect class="box" x="460.0" y="78.0" width="200.0" height="32.0" rx="6"/><text class="t-tick" x="560.0" y="98.5" text-anchor="middle">10 scores, all 0</text><text class="t-tick" x="680.0" y="90.0" text-anchor="start">step 0:</text><text class="t-tick" x="680.0" y="108.0" text-anchor="start">loss = ln 10 = 2.3026</text><text class="t-tick" x="680.0" y="126.0" text-anchor="start">head gradient: 1.215</text><text class="t-tick" x="680.0" y="144.0" text-anchor="start">body gradient: 0.0</text><line class="edge" x1="300.0" y1="287.0" x2="452.0" y2="287.0" marker-end="url(#ah)"/><text class="t-muted" x="20.0" y="336.0" text-anchor="start">The body keeps its pre-trained weights. At the first step the zero head gives every class the same score, so only the head</text><text class="t-muted" x="20.0" y="352.0" text-anchor="start">learns; the body starts to move at step 2, once the head is no longer zero.</text></svg><figcaption>The head swap. Left: the ImageNet-21k checkpoint ends in a pre-logits layer (768 × 768, tanh) and a 21,843-class layer. Both are removed. Right: one new 768 × 10 layer of zeros reads the [class] representation. At step 0 all ten scores are 0, the loss is ln 10 = 2.3026, the head gets a gradient and the body gets none.</figcaption></figure>

## Fine-tuning at a higher resolution {§3.2}

> [!PAPER] · §3.2
> [![Section 3.2 continued: it is often beneficial to fine-tune at higher resolution than pre-training (Touvron et al., 2019; Kolesnikov et al., 2020). When feeding images of higher resolution, we keep the patch size the same, which results in a larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints), however, the pre-trained position embeddings may no longer be meaningful](https://ishwarj.com/img/papers/vit/p3-higher-res.pngt/p3-higher-res.png)](/img/papers/vit/p3-higher-res.png)
>
> **What it says:** "It is often beneficial to fine-tune at higher resolution than pre-training". The patch size stays the same, "which results in a larger effective sequence length". ViT "can handle arbitrary sequence lengths", but "the pre-trained position embeddings may no longer be meaningful."
>
> **What it means:**
>
> Pre-training runs at 224 × 224 pixels because it is long and expensive. Fine-tuning is short, so it can afford bigger pictures, and bigger pictures show finer detail. Table 4 sets the usual fine-tuning size at 384 × 384.
>
> Keep the patch at 16 × 16 pixels and the picture at 384, and you get $$384/16 = 24$$ patches per side, so 576 patches instead of 196. The weights of the encoder do not care: the same matrices act on every token, however many there are. Only one part of the model has a fixed size, the table of position embeddings. It has 197 rows, and the longer sequence needs 577.

> [!DEFINITION] Resolution
> The size of the picture in pixels, here width × height: 224 × 224 or 384 × 384. More pixels show smaller details, and for ViT they also mean more patches, so more tokens and more compute.

The arithmetic, with the symbols of Part 2 ($$N = HW/P^2$$):

$$
N_{224} = \frac{224 \times 224}{16^2} = 14 \times 14 = 196, \qquad N_{384} = \frac{384 \times 384}{16^2} = 24 \times 24 = 576
$$

So the sequence grows from 197 to 577 tokens, about 2.9 times longer. Attention compares every pair of tokens, so its pairs grow from 38,809 to 332,929, about 8.6 times. The whole forward pass of ViT-B/16 grows from about 34.9 to 110.3 billion operations. That is our estimate from the shapes, printed in the Model variants section below.

The paper cites two sources for "often beneficial". One is BiT, the ResNet baseline we meet below. The other is a paper about exactly this effect:

> [!PAPER] Touvron et al. (2019), Fixing the train-test resolution discrepancy · Abstract
> [![The abstract of Touvron et al. (2019): data-augmentation is key; existing augmentations induce a significant discrepancy between the size of the objects seen by the classifier at train and test time: in fact, a lower train resolution improves the classification at test time. We propose a simple strategy that employs different train and test resolutions, relying on a computationally cheap fine-tuning of the network at the test resolution. A ResNet-50 trained on 128 by 128 images reaches 77.1% top-1 accuracy on ImageNet, and 79.8% when trained at 224 by 224](https://ishwarj.com/img/papers/vit/p3-ext-touvron.png/p3-ext-touvron.png)](/img/papers/vit/p3-ext-touvron.png)
>
> **What it says:** common augmentations create "a significant discrepancy between the size of the objects seen by the classifier at train and test time"; "a lower train resolution improves the classification at test time"; the fix is "a computationally cheap fine-tuning of the network at the test resolution".
>
> **What it means:**
>
> During training, pictures are randomly cropped and zoomed, so objects look larger than in the plain test pictures. Testing at a higher resolution makes test objects larger too, so the two match again. A short fine-tuning at the test resolution then repairs the rest.
>
> This is the habit ViT follows: train long at a small size, fine-tune briefly at a larger one. For a CNN the change is easy, because a convolution slides over any size of grid. For ViT one piece does not stretch by itself, and the next sentence of the paper fixes it.

> [!PAPER] · §3.2
> [![The end of Section 3.2: we therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer](https://ishwarj.com/img/papers/vit/p3-interpolation.png3-interpolation.png)](/img/papers/vit/p3-interpolation.png)
>
> **What it says:** "We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image." This step and the patch cutting "are the only points at which an inductive bias about the 2D structure of the images is manually injected".
>
> **What it means:**
>
> The 196 position rows were learned for a 14 × 14 grid. Lay them out on that grid, like 196 pins on a map. Now stretch the map to 24 × 24. Each new pin gets a value made from the old pins nearest to its place, and the stretched map has 576 pins.
>
> "2D" is the key word. The rows are not stretched as one long list, which would mix the end of one grid row with the start of the next. They are stretched as a grid, rows and columns separately. Together with the cut into patches, it is the only moment when the model is told that its patches form a picture.

> [!DEFINITION] Interpolation
> Filling in values between known values. If a thermometer reads 10° at noon and 14° at 2 pm, linear interpolation guesses 12° at 1 pm. **Bilinear** interpolation does the same on a grid: first along the rows, then along the columns. **Bicubic** uses 4 × 4 neighbours and a smoother curve.

**A tiny example first.** The official code resizes the grid with `scipy.ndimage.zoom(..., order=1)`, which is linear interpolation in each direction with the corner values kept. On a 2 × 2 grid of single numbers, resized to 3 × 3 ([`vit_part3_posemb.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part3_posemb.py)):

```text
== 1. resize a 2x2 grid to 3x3 the way the official code does (scipy.ndimage.zoom, order=1 = linear) ==
  grid 2x2: [[0.0, 1.0], [2.0, 3.0]]
    0.00  0.50  1.00
    1.00  1.50  2.00
    2.00  2.50  3.00
  corners stay, the new middle row and column are averages of their neighbours: (0+1)/2 = 0.5, (0+2)/2 = 1, (0+1+2+3)/4 = 1.5
```

Check it by hand. The four corners keep their values 0, 1, 2 and 3. The new top-middle value sits halfway between 0 and 1, so it is 0.5. The centre sits halfway between all four, so it is $$(0 + 1 + 2 + 3)/4 = 1.5$$.

<figure class="fig"><svg viewBox="0 0 800 250" role="img" aria-label="A 2 by 2 grid of the values 0, 1, 2 and 3 stretched to a 3 by 3 grid by bilinear interpolation: the corners keep their values, the edge middles are 0.5, 1, 2 and 2.5, and the centre is 1.5."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Bilinear interpolation: stretch a 2 × 2 grid to 3 × 3</text><text class="t-note" x="146.0" y="60.0" text-anchor="middle">old grid, 2 × 2</text><rect class="cell" x="81.0" y="71.0" width="64" height="64" rx="4" style="fill-opacity:0.10"/><text class="t-val" x="113.0" y="108.0" text-anchor="middle">0</text><rect class="cell" x="147.0" y="71.0" width="64" height="64" rx="4" style="fill-opacity:0.30"/><text class="t-val" x="179.0" y="108.0" text-anchor="middle">1</text><rect class="cell" x="81.0" y="137.0" width="64" height="64" rx="4" style="fill-opacity:0.50"/><text class="t-val" x="113.0" y="174.0" text-anchor="middle">2</text><rect class="cell" x="147.0" y="137.0" width="64" height="64" rx="4" style="fill-opacity:0.70"/><text class="t-val" x="179.0" y="174.0" text-anchor="middle">3</text><line class="edge" x1="240.0" y1="136.0" x2="330.0" y2="136.0" marker-end="url(#ah)"/><text class="t-tick" x="285.0" y="126.0" text-anchor="middle">zoom, order 1</text><text class="t-note" x="447.0" y="40.0" text-anchor="middle">new grid, 3 × 3</text><rect class="cell" x="361.0" y="51.0" width="56" height="56" rx="4" style="fill-opacity:0.10"/><text class="t-val" x="389.0" y="84.0" text-anchor="middle">0</text><rect class="cell" x="419.0" y="51.0" width="56" height="56" rx="4" style="fill-opacity:0.20"/><text class="t-val" x="447.0" y="84.0" text-anchor="middle">0.5</text><rect class="cell" x="477.0" y="51.0" width="56" height="56" rx="4" style="fill-opacity:0.30"/><text class="t-val" x="505.0" y="84.0" text-anchor="middle">1</text><rect class="cell" x="361.0" y="109.0" width="56" height="56" rx="4" style="fill-opacity:0.30"/><text class="t-val" x="389.0" y="142.0" text-anchor="middle">1</text><rect class="cell" x="419.0" y="109.0" width="56" height="56" rx="4" style="fill-opacity:0.40"/><text class="t-val" x="447.0" y="142.0" text-anchor="middle">1.5</text><rect class="cell" x="477.0" y="109.0" width="56" height="56" rx="4" style="fill-opacity:0.50"/><text class="t-val" x="505.0" y="142.0" text-anchor="middle">2</text><rect class="cell" x="361.0" y="167.0" width="56" height="56" rx="4" style="fill-opacity:0.50"/><text class="t-val" x="389.0" y="200.0" text-anchor="middle">2</text><rect class="cell" x="419.0" y="167.0" width="56" height="56" rx="4" style="fill-opacity:0.60"/><text class="t-val" x="447.0" y="200.0" text-anchor="middle">2.5</text><rect class="cell" x="477.0" y="167.0" width="56" height="56" rx="4" style="fill-opacity:0.70"/><text class="t-val" x="505.0" y="200.0" text-anchor="middle">3</text><text class="t-tick" x="560.0" y="80.0" text-anchor="start">corners keep their values</text><text class="t-tick" x="560.0" y="98.0" text-anchor="start">edge middles: average of 2</text><text class="t-tick" x="560.0" y="116.0" text-anchor="start">  (0 + 1)/2 = 0.5</text><text class="t-tick" x="560.0" y="134.0" text-anchor="start">centre: average of 4</text><text class="t-tick" x="560.0" y="152.0" text-anchor="start">  (0 + 1 + 2 + 3)/4 = 1.5</text><text class="t-tick" x="560.0" y="170.0" text-anchor="start"></text><text class="t-tick" x="560.0" y="188.0" text-anchor="start">In ViT each cell is a vector</text><text class="t-tick" x="560.0" y="206.0" text-anchor="start">of 768 numbers, and the grid</text><text class="t-tick" x="560.0" y="224.0" text-anchor="start">goes from 14 × 14 to 24 × 24.</text></svg><figcaption>Bilinear interpolation on the smallest grid. Left: a 2×2 grid of values 0, 1, 2, 3. Right: the same grid stretched to 3×3. The corners keep their values; each new cell is the average of the old cells around its place (0.5, 1, 1.5, 2, 2.5).</figcaption></figure>

**The formula.** For one new grid row $$i$$ out of $$n_{\text{new}}$$, the place on the old grid of $$n_{\text{old}}$$ rows is

$$
y = i \cdot \frac{n_{\text{old}} - 1}{n_{\text{new}} - 1}, \qquad \text{row}_{\text{new}}(i) = (1 - \delta)\,\text{row}_{\text{old}}(\lfloor y \rfloor) + \delta\,\text{row}_{\text{old}}(\lfloor y \rfloor + 1), \quad \delta = y - \lfloor y \rfloor
$$

where $$\lfloor y \rfloor$$ is $$y$$ rounded down, and $$\delta$$ is how far the new row sits past that old row. The same rule then runs along the columns. Each "row" here is a whole vector of 768 numbers, so the mix is done number by number. For ViT-B/16 at 384, $$n_{\text{old}} = 14$$ and $$n_{\text{new}} = 24$$. New row 5 sits at $$5 \times 13/23 = 2.8261$$, so it takes 0.1739 of old row 2 and 0.8261 of old row 3.

**The real table.** We wrote this rule as a plain loop and ran it on the 14 × 14 × 768 position grid of `google/vit-base-patch16-224`. Then we compared it with SciPy and with the library:

```text
== 2. the position table of google/vit-base-patch16-224 ==
  E_pos (197, 768) = 1 row for [class] + 196 rows = a 14x14 grid of 768 numbers
  at 384 pixels: 384/16 = 24 patches per side, N = 576, so 577 rows are needed: 380 rows do not exist
  our bilinear 14x14 -> 24x24: (24, 24, 768); max |difference| vs scipy.ndimage.zoom(order=1): 2.3e-15
  new row 5 sits at old position 5 x 13/23 = 2.8261: 0.1739 of old row 2 + 0.8261 of old row 3
  library interpolate_pos_encoding: bicubic, align_corners=False; our bicubic call matches it: max |difference| 0.0e+00
  bilinear (official) vs bicubic (library) position rows: mean |difference| 0.0131, mean |value| 0.0784
  corner row (0,0) kept by bilinear: max |difference| 0.0e+00
```

Our loop equals SciPy's resize to $$2 \times 10^{-15}$$, which is the rounding of 64-bit numbers. The Hugging Face library does something slightly different: it uses bicubic interpolation, not bilinear. The two tables differ by 0.0131 on average, against an average size of 0.0784 for the numbers themselves. So the official and the library versions are close but not the same.

To see what the stretch keeps, take one position row and compare it, by cosine similarity, with every other row. In the 14 × 14 grid, row (3, 10) is most like its four neighbours (0.79 to 0.81) and unlike the far corner (−0.29). After the stretch, the same place on the 24 × 24 grid shows the same pattern, only smoother. Its neighbours are now closer (0.92 to 0.95), because they are made from the same old rows.

<figure class="fig"><svg viewBox="0 0 800 392" role="img" aria-label="Two heatmaps of cosine similarity: one position row of the 14 by 14 grid of ViT-B/16 against all 196 rows, and the same place on the stretched 24 by 24 grid against all 576 rows; both are bright near the chosen place and faint far away."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Cosine similarity of one position row with every other row</text><text class="t-note" x="200.0" y="50.0" text-anchor="middle">14 × 14, as trained: row (3, 10)</text><rect class="cell" x="60.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.227"/><rect class="cell" x="80.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.226"/><rect class="cell" x="100.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.149"/><rect class="cell" x="120.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.280"/><rect class="cell" x="140.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.425"/><rect class="cell" x="160.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.408"/><rect class="cell" x="180.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.222"/><rect class="cell" x="200.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.128"/><rect class="cell" x="220.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.266"/><rect class="cell" x="240.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.518"/><rect class="cell" x="260.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.638"/><rect class="cell" x="280.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.505"/><rect class="cell" x="300.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.324"/><rect class="cell" x="320.00" y="60.00" width="19.40" height="19.40" style="fill-opacity:0.315"/><rect class="cell" x="60.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.213"/><rect class="cell" x="80.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.205"/><rect class="cell" x="100.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.113"/><rect class="cell" x="120.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.272"/><rect class="cell" x="140.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.417"/><rect class="cell" x="160.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.399"/><rect class="cell" x="180.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.202"/><rect class="cell" x="200.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.093"/><rect class="cell" x="220.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.230"/><rect class="cell" x="240.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.518"/><rect class="cell" x="260.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.650"/><rect class="cell" x="280.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.503"/><rect class="cell" x="300.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.294"/><rect class="cell" x="320.00" y="80.00" width="19.40" height="19.40" style="fill-opacity:0.292"/><rect class="cell" x="60.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.423"/><rect class="cell" x="80.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.433"/><rect class="cell" x="100.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.342"/><rect class="cell" x="120.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.487"/><rect class="cell" x="140.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.618"/><rect class="cell" x="160.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.610"/><rect class="cell" x="180.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.420"/><rect class="cell" x="200.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.310"/><rect class="cell" x="220.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.444"/><rect class="cell" x="240.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.717"/><rect class="cell" x="260.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.857"/><rect class="cell" x="280.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.718"/><rect class="cell" x="300.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.518"/><rect class="cell" x="320.00" y="100.00" width="19.40" height="19.40" style="fill-opacity:0.521"/><rect class="cell" x="60.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.522"/><rect class="cell" x="80.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.550"/><rect class="cell" x="100.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.449"/><rect class="cell" x="120.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.590"/><rect class="cell" x="140.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.742"/><rect class="cell" x="160.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.730"/><rect class="cell" x="180.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.544"/><rect class="cell" x="200.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.441"/><rect class="cell" x="220.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.584"/><rect class="cell" x="240.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.850"/><rect class="cell" x="260.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:1.000"/><rect class="cell" x="280.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.848"/><rect class="cell" x="300.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.642"/><rect class="cell" x="320.00" y="120.00" width="19.40" height="19.40" style="fill-opacity:0.621"/><rect class="cell" x="60.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.424"/><rect class="cell" x="80.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.437"/><rect class="cell" x="100.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.338"/><rect class="cell" x="120.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.479"/><rect class="cell" x="140.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.634"/><rect class="cell" x="160.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.623"/><rect class="cell" x="180.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.425"/><rect class="cell" x="200.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.326"/><rect class="cell" x="220.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.462"/><rect class="cell" x="240.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.738"/><rect class="cell" x="260.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.865"/><rect class="cell" x="280.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.728"/><rect class="cell" x="300.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.524"/><rect class="cell" x="320.00" y="140.00" width="19.40" height="19.40" style="fill-opacity:0.514"/><rect class="cell" x="60.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.196"/><rect class="cell" x="80.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.190"/><rect class="cell" x="100.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.107"/><rect class="cell" x="120.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.237"/><rect class="cell" x="140.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.389"/><rect class="cell" x="160.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.377"/><rect class="cell" x="180.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.192"/><rect class="cell" x="200.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.078"/><rect class="cell" x="220.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.227"/><rect class="cell" x="240.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.501"/><rect class="cell" x="260.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.626"/><rect class="cell" x="280.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.492"/><rect class="cell" x="300.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.289"/><rect class="cell" x="320.00" y="160.00" width="19.40" height="19.40" style="fill-opacity:0.292"/><rect class="cell" x="60.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="80.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="100.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="120.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.062"/><rect class="cell" x="140.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.199"/><rect class="cell" x="160.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.176"/><rect class="cell" x="180.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="200.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="220.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.036"/><rect class="cell" x="240.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.311"/><rect class="cell" x="260.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.429"/><rect class="cell" x="280.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.302"/><rect class="cell" x="300.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.089"/><rect class="cell" x="320.00" y="180.00" width="19.40" height="19.40" style="fill-opacity:0.112"/><rect class="cell" x="60.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="80.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="100.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="120.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="140.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.109"/><rect class="cell" x="160.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.087"/><rect class="cell" x="180.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="200.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="220.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="240.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.205"/><rect class="cell" x="260.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.336"/><rect class="cell" x="280.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.195"/><rect class="cell" x="300.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="320.00" y="200.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="60.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.046"/><rect class="cell" x="80.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="100.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="120.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.075"/><rect class="cell" x="140.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.224"/><rect class="cell" x="160.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.201"/><rect class="cell" x="180.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="200.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="220.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.054"/><rect class="cell" x="240.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.320"/><rect class="cell" x="260.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.461"/><rect class="cell" x="280.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.316"/><rect class="cell" x="300.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.120"/><rect class="cell" x="320.00" y="220.00" width="19.40" height="19.40" style="fill-opacity:0.136"/><rect class="cell" x="60.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.231"/><rect class="cell" x="80.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.234"/><rect class="cell" x="100.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.142"/><rect class="cell" x="120.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.298"/><rect class="cell" x="140.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.425"/><rect class="cell" x="160.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.412"/><rect class="cell" x="180.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.239"/><rect class="cell" x="200.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.128"/><rect class="cell" x="220.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.268"/><rect class="cell" x="240.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.534"/><rect class="cell" x="260.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.660"/><rect class="cell" x="280.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.534"/><rect class="cell" x="300.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.340"/><rect class="cell" x="320.00" y="240.00" width="19.40" height="19.40" style="fill-opacity:0.331"/><rect class="cell" x="60.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.324"/><rect class="cell" x="80.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.332"/><rect class="cell" x="100.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.244"/><rect class="cell" x="120.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.395"/><rect class="cell" x="140.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.528"/><rect class="cell" x="160.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.510"/><rect class="cell" x="180.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.324"/><rect class="cell" x="200.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.215"/><rect class="cell" x="220.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.366"/><rect class="cell" x="240.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.633"/><rect class="cell" x="260.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.761"/><rect class="cell" x="280.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.633"/><rect class="cell" x="300.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.433"/><rect class="cell" x="320.00" y="260.00" width="19.40" height="19.40" style="fill-opacity:0.406"/><rect class="cell" x="60.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.205"/><rect class="cell" x="80.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.203"/><rect class="cell" x="100.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.105"/><rect class="cell" x="120.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.254"/><rect class="cell" x="140.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.398"/><rect class="cell" x="160.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.378"/><rect class="cell" x="180.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.189"/><rect class="cell" x="200.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.067"/><rect class="cell" x="220.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.221"/><rect class="cell" x="240.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.498"/><rect class="cell" x="260.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.624"/><rect class="cell" x="280.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.483"/><rect class="cell" x="300.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.293"/><rect class="cell" x="320.00" y="280.00" width="19.40" height="19.40" style="fill-opacity:0.290"/><rect class="cell" x="60.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="80.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="100.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="120.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="140.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.133"/><rect class="cell" x="160.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.109"/><rect class="cell" x="180.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="200.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="220.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="240.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.225"/><rect class="cell" x="260.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.353"/><rect class="cell" x="280.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.219"/><rect class="cell" x="300.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="320.00" y="300.00" width="19.40" height="19.40" style="fill-opacity:0.062"/><rect class="cell" x="60.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.081"/><rect class="cell" x="80.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.050"/><rect class="cell" x="100.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="120.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.082"/><rect class="cell" x="140.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.227"/><rect class="cell" x="160.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.209"/><rect class="cell" x="180.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.037"/><rect class="cell" x="200.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.030"/><rect class="cell" x="220.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.061"/><rect class="cell" x="240.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.314"/><rect class="cell" x="260.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.447"/><rect class="cell" x="280.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.314"/><rect class="cell" x="300.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.137"/><rect class="cell" x="320.00" y="320.00" width="19.40" height="19.40" style="fill-opacity:0.159"/><rect class="box-on" x="260.00" y="120.00" width="20.00" height="20.00" style="fill-opacity:0"/><line class="edge" x1="352.0" y1="200.0" x2="418.0" y2="200.0" marker-end="url(#ah)"/><text class="t-tick" x="385.0" y="190.0" text-anchor="middle">2D stretch</text><text class="t-note" x="570.0" y="50.0" text-anchor="middle">24 × 24, after: the same place (5, 18)</text><rect class="cell" x="430.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.222"/><rect class="cell" x="441.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.227"/><rect class="cell" x="453.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.227"/><rect class="cell" x="465.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.170"/><rect class="cell" x="476.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.167"/><rect class="cell" x="488.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.224"/><rect class="cell" x="500.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.301"/><rect class="cell" x="511.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.383"/><rect class="cell" x="523.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.401"/><rect class="cell" x="535.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.390"/><rect class="cell" x="546.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.297"/><rect class="cell" x="558.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.214"/><rect class="cell" x="570.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.151"/><rect class="cell" x="581.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.163"/><rect class="cell" x="593.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.229"/><rect class="cell" x="605.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.361"/><rect class="cell" x="616.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.494"/><rect class="cell" x="628.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.589"/><rect class="cell" x="640.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.625"/><rect class="cell" x="651.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.571"/><rect class="cell" x="663.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.487"/><rect class="cell" x="675.00" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.374"/><rect class="cell" x="686.67" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.339"/><rect class="cell" x="698.33" y="60.00" width="11.07" height="11.07" style="fill-opacity:0.317"/><rect class="cell" x="430.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.229"/><rect class="cell" x="441.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.235"/><rect class="cell" x="453.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.233"/><rect class="cell" x="465.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.162"/><rect class="cell" x="476.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.159"/><rect class="cell" x="488.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.233"/><rect class="cell" x="500.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.321"/><rect class="cell" x="511.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.411"/><rect class="cell" x="523.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.433"/><rect class="cell" x="535.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.419"/><rect class="cell" x="546.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.315"/><rect class="cell" x="558.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.218"/><rect class="cell" x="570.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.142"/><rect class="cell" x="581.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.152"/><rect class="cell" x="593.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.224"/><rect class="cell" x="605.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.382"/><rect class="cell" x="616.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.540"/><rect class="cell" x="628.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.652"/><rect class="cell" x="640.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.691"/><rect class="cell" x="651.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.631"/><rect class="cell" x="663.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.531"/><rect class="cell" x="675.00" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.394"/><rect class="cell" x="686.67" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.353"/><rect class="cell" x="698.33" y="71.67" width="11.07" height="11.07" style="fill-opacity:0.326"/><rect class="cell" x="430.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.271"/><rect class="cell" x="441.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.283"/><rect class="cell" x="453.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.278"/><rect class="cell" x="465.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.208"/><rect class="cell" x="476.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.205"/><rect class="cell" x="488.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.278"/><rect class="cell" x="500.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.362"/><rect class="cell" x="511.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.443"/><rect class="cell" x="523.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.467"/><rect class="cell" x="535.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.454"/><rect class="cell" x="546.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.355"/><rect class="cell" x="558.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.263"/><rect class="cell" x="570.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.187"/><rect class="cell" x="581.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.195"/><rect class="cell" x="593.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.260"/><rect class="cell" x="605.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.414"/><rect class="cell" x="616.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.566"/><rect class="cell" x="628.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.677"/><rect class="cell" x="640.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.716"/><rect class="cell" x="651.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.658"/><rect class="cell" x="663.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.561"/><rect class="cell" x="675.00" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.425"/><rect class="cell" x="686.67" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.388"/><rect class="cell" x="698.33" y="83.33" width="11.07" height="11.07" style="fill-opacity:0.359"/><rect class="cell" x="430.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.396"/><rect class="cell" x="441.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.430"/><rect class="cell" x="453.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.418"/><rect class="cell" x="465.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.349"/><rect class="cell" x="476.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.344"/><rect class="cell" x="488.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.413"/><rect class="cell" x="500.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.499"/><rect class="cell" x="511.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.570"/><rect class="cell" x="523.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.603"/><rect class="cell" x="535.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.588"/><rect class="cell" x="546.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.495"/><rect class="cell" x="558.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.399"/><rect class="cell" x="570.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.321"/><rect class="cell" x="581.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.331"/><rect class="cell" x="593.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.393"/><rect class="cell" x="605.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.551"/><rect class="cell" x="616.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.694"/><rect class="cell" x="628.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.818"/><rect class="cell" x="640.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.858"/><rect class="cell" x="651.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.803"/><rect class="cell" x="663.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.709"/><rect class="cell" x="675.00" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.568"/><rect class="cell" x="686.67" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.541"/><rect class="cell" x="698.33" y="95.00" width="11.07" height="11.07" style="fill-opacity:0.498"/><rect class="cell" x="430.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.478"/><rect class="cell" x="441.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.526"/><rect class="cell" x="453.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.513"/><rect class="cell" x="465.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.445"/><rect class="cell" x="476.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.437"/><rect class="cell" x="488.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.501"/><rect class="cell" x="500.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.588"/><rect class="cell" x="511.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.655"/><rect class="cell" x="523.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.692"/><rect class="cell" x="535.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.676"/><rect class="cell" x="546.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.588"/><rect class="cell" x="558.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.493"/><rect class="cell" x="570.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.417"/><rect class="cell" x="581.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.430"/><rect class="cell" x="593.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.489"/><rect class="cell" x="605.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.646"/><rect class="cell" x="616.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.780"/><rect class="cell" x="628.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.910"/><rect class="cell" x="640.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.949"/><rect class="cell" x="651.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.896"/><rect class="cell" x="663.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.804"/><rect class="cell" x="675.00" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.662"/><rect class="cell" x="686.67" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.640"/><rect class="cell" x="698.33" y="106.67" width="11.07" height="11.07" style="fill-opacity:0.585"/><rect class="cell" x="430.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.513"/><rect class="cell" x="441.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.570"/><rect class="cell" x="453.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.556"/><rect class="cell" x="465.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.488"/><rect class="cell" x="476.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.477"/><rect class="cell" x="488.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.539"/><rect class="cell" x="500.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.629"/><rect class="cell" x="511.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.701"/><rect class="cell" x="523.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.739"/><rect class="cell" x="535.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.720"/><rect class="cell" x="546.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.636"/><rect class="cell" x="558.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.542"/><rect class="cell" x="570.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.470"/><rect class="cell" x="581.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.486"/><rect class="cell" x="593.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.546"/><rect class="cell" x="605.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.701"/><rect class="cell" x="616.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.831"/><rect class="cell" x="628.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.962"/><rect class="cell" x="640.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:1.000"/><rect class="cell" x="651.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.946"/><rect class="cell" x="663.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.853"/><rect class="cell" x="675.00" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.710"/><rect class="cell" x="686.67" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.686"/><rect class="cell" x="698.33" y="118.33" width="11.07" height="11.07" style="fill-opacity:0.620"/><rect class="cell" x="430.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.478"/><rect class="cell" x="441.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.529"/><rect class="cell" x="453.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.516"/><rect class="cell" x="465.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.446"/><rect class="cell" x="476.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.435"/><rect class="cell" x="488.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.497"/><rect class="cell" x="500.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.590"/><rect class="cell" x="511.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.668"/><rect class="cell" x="523.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.707"/><rect class="cell" x="535.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.688"/><rect class="cell" x="546.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.599"/><rect class="cell" x="558.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.502"/><rect class="cell" x="570.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.430"/><rect class="cell" x="581.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.446"/><rect class="cell" x="593.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.506"/><rect class="cell" x="605.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.665"/><rect class="cell" x="616.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.800"/><rect class="cell" x="628.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.927"/><rect class="cell" x="640.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.965"/><rect class="cell" x="651.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.911"/><rect class="cell" x="663.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.817"/><rect class="cell" x="675.00" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.672"/><rect class="cell" x="686.67" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.646"/><rect class="cell" x="698.33" y="130.00" width="11.07" height="11.07" style="fill-opacity:0.584"/><rect class="cell" x="430.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.404"/><rect class="cell" x="441.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.438"/><rect class="cell" x="453.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.427"/><rect class="cell" x="465.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.359"/><rect class="cell" x="476.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.349"/><rect class="cell" x="488.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.410"/><rect class="cell" x="500.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.498"/><rect class="cell" x="511.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.578"/><rect class="cell" x="523.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.612"/><rect class="cell" x="535.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.596"/><rect class="cell" x="546.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.503"/><rect class="cell" x="558.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.410"/><rect class="cell" x="570.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.341"/><rect class="cell" x="581.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.354"/><rect class="cell" x="593.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.413"/><rect class="cell" x="605.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.565"/><rect class="cell" x="616.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.702"/><rect class="cell" x="628.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.814"/><rect class="cell" x="640.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.848"/><rect class="cell" x="651.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.796"/><rect class="cell" x="663.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.706"/><rect class="cell" x="675.00" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.572"/><rect class="cell" x="686.67" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.545"/><rect class="cell" x="698.33" y="141.67" width="11.07" height="11.07" style="fill-opacity:0.501"/><rect class="cell" x="430.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.279"/><rect class="cell" x="441.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.294"/><rect class="cell" x="453.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.289"/><rect class="cell" x="465.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.218"/><rect class="cell" x="476.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.210"/><rect class="cell" x="488.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.273"/><rect class="cell" x="500.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.359"/><rect class="cell" x="511.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.449"/><rect class="cell" x="523.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.478"/><rect class="cell" x="535.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.467"/><rect class="cell" x="546.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.368"/><rect class="cell" x="558.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.273"/><rect class="cell" x="570.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.196"/><rect class="cell" x="581.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.209"/><rect class="cell" x="593.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.279"/><rect class="cell" x="605.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.433"/><rect class="cell" x="616.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.582"/><rect class="cell" x="628.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.692"/><rect class="cell" x="640.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.730"/><rect class="cell" x="651.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.674"/><rect class="cell" x="663.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.580"/><rect class="cell" x="675.00" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.445"/><rect class="cell" x="686.67" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.410"/><rect class="cell" x="698.33" y="153.33" width="11.07" height="11.07" style="fill-opacity:0.381"/><rect class="cell" x="430.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.155"/><rect class="cell" x="441.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.150"/><rect class="cell" x="453.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.152"/><rect class="cell" x="465.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.085"/><rect class="cell" x="476.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.081"/><rect class="cell" x="488.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.140"/><rect class="cell" x="500.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.217"/><rect class="cell" x="511.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.308"/><rect class="cell" x="523.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.328"/><rect class="cell" x="535.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.322"/><rect class="cell" x="546.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.226"/><rect class="cell" x="558.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.139"/><rect class="cell" x="570.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.062"/><rect class="cell" x="581.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.073"/><rect class="cell" x="593.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.149"/><rect class="cell" x="605.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.291"/><rect class="cell" x="616.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.441"/><rect class="cell" x="628.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.539"/><rect class="cell" x="640.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.580"/><rect class="cell" x="651.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.526"/><rect class="cell" x="663.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.434"/><rect class="cell" x="675.00" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.307"/><rect class="cell" x="686.67" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.267"/><rect class="cell" x="698.33" y="165.00" width="11.07" height="11.07" style="fill-opacity:0.258"/><rect class="cell" x="430.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.063"/><rect class="cell" x="441.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.044"/><rect class="cell" x="453.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.050"/><rect class="cell" x="465.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.046"/><rect class="cell" x="500.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.120"/><rect class="cell" x="511.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.212"/><rect class="cell" x="523.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.224"/><rect class="cell" x="535.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.219"/><rect class="cell" x="546.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.120"/><rect class="cell" x="558.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.034"/><rect class="cell" x="570.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.047"/><rect class="cell" x="605.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.191"/><rect class="cell" x="616.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.349"/><rect class="cell" x="628.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.443"/><rect class="cell" x="640.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.486"/><rect class="cell" x="651.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.433"/><rect class="cell" x="663.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.337"/><rect class="cell" x="675.00" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.204"/><rect class="cell" x="686.67" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.160"/><rect class="cell" x="698.33" y="176.67" width="11.07" height="11.07" style="fill-opacity:0.167"/><rect class="cell" x="430.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="441.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="453.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="465.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.046"/><rect class="cell" x="511.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.140"/><rect class="cell" x="523.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.146"/><rect class="cell" x="535.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.142"/><rect class="cell" x="546.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.041"/><rect class="cell" x="558.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.111"/><rect class="cell" x="616.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.272"/><rect class="cell" x="628.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.362"/><rect class="cell" x="640.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.406"/><rect class="cell" x="651.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.352"/><rect class="cell" x="663.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.256"/><rect class="cell" x="675.00" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.127"/><rect class="cell" x="686.67" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.082"/><rect class="cell" x="698.33" y="188.33" width="11.07" height="11.07" style="fill-opacity:0.099"/><rect class="cell" x="430.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="441.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="453.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="465.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="511.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.098"/><rect class="cell" x="523.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.103"/><rect class="cell" x="535.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.101"/><rect class="cell" x="546.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="558.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.058"/><rect class="cell" x="616.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.221"/><rect class="cell" x="628.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.314"/><rect class="cell" x="640.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.361"/><rect class="cell" x="651.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.301"/><rect class="cell" x="663.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.205"/><rect class="cell" x="675.00" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.084"/><rect class="cell" x="686.67" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.038"/><rect class="cell" x="698.33" y="200.00" width="11.07" height="11.07" style="fill-opacity:0.061"/><rect class="cell" x="430.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="441.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="453.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="465.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="511.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.125"/><rect class="cell" x="523.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.132"/><rect class="cell" x="535.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.128"/><rect class="cell" x="546.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="558.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.083"/><rect class="cell" x="616.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.246"/><rect class="cell" x="628.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.346"/><rect class="cell" x="640.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.394"/><rect class="cell" x="651.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.332"/><rect class="cell" x="663.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.235"/><rect class="cell" x="675.00" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.114"/><rect class="cell" x="686.67" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.068"/><rect class="cell" x="698.33" y="211.67" width="11.07" height="11.07" style="fill-opacity:0.087"/><rect class="cell" x="430.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.062"/><rect class="cell" x="441.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.038"/><rect class="cell" x="453.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.038"/><rect class="cell" x="465.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.031"/><rect class="cell" x="500.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.109"/><rect class="cell" x="511.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.204"/><rect class="cell" x="523.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.214"/><rect class="cell" x="535.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.206"/><rect class="cell" x="546.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.104"/><rect class="cell" x="558.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.035"/><rect class="cell" x="605.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.169"/><rect class="cell" x="616.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.321"/><rect class="cell" x="628.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.426"/><rect class="cell" x="640.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.473"/><rect class="cell" x="651.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.413"/><rect class="cell" x="663.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.318"/><rect class="cell" x="675.00" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.198"/><rect class="cell" x="686.67" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.156"/><rect class="cell" x="698.33" y="223.33" width="11.07" height="11.07" style="fill-opacity:0.160"/><rect class="cell" x="430.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.152"/><rect class="cell" x="441.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.146"/><rect class="cell" x="453.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.146"/><rect class="cell" x="465.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.072"/><rect class="cell" x="476.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.070"/><rect class="cell" x="488.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.142"/><rect class="cell" x="500.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.222"/><rect class="cell" x="511.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.309"/><rect class="cell" x="523.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.326"/><rect class="cell" x="535.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.318"/><rect class="cell" x="546.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.220"/><rect class="cell" x="558.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.132"/><rect class="cell" x="570.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.055"/><rect class="cell" x="581.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.065"/><rect class="cell" x="593.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.142"/><rect class="cell" x="605.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.285"/><rect class="cell" x="616.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.436"/><rect class="cell" x="628.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.545"/><rect class="cell" x="640.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.590"/><rect class="cell" x="651.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.535"/><rect class="cell" x="663.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.441"/><rect class="cell" x="675.00" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.314"/><rect class="cell" x="686.67" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.273"/><rect class="cell" x="698.33" y="235.00" width="11.07" height="11.07" style="fill-opacity:0.259"/><rect class="cell" x="430.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.250"/><rect class="cell" x="441.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.265"/><rect class="cell" x="453.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.264"/><rect class="cell" x="465.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.195"/><rect class="cell" x="476.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.193"/><rect class="cell" x="488.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.264"/><rect class="cell" x="500.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.340"/><rect class="cell" x="511.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.411"/><rect class="cell" x="523.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.434"/><rect class="cell" x="535.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.425"/><rect class="cell" x="546.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.341"/><rect class="cell" x="558.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.257"/><rect class="cell" x="570.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.183"/><rect class="cell" x="581.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.192"/><rect class="cell" x="593.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.258"/><rect class="cell" x="605.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.399"/><rect class="cell" x="616.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.539"/><rect class="cell" x="628.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.642"/><rect class="cell" x="640.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.682"/><rect class="cell" x="651.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.635"/><rect class="cell" x="663.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.550"/><rect class="cell" x="675.00" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.426"/><rect class="cell" x="686.67" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.388"/><rect class="cell" x="698.33" y="246.67" width="11.07" height="11.07" style="fill-opacity:0.357"/><rect class="cell" x="430.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.285"/><rect class="cell" x="441.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.304"/><rect class="cell" x="453.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.301"/><rect class="cell" x="465.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.231"/><rect class="cell" x="476.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.228"/><rect class="cell" x="488.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.300"/><rect class="cell" x="500.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.381"/><rect class="cell" x="511.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.455"/><rect class="cell" x="523.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.480"/><rect class="cell" x="535.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.467"/><rect class="cell" x="546.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.376"/><rect class="cell" x="558.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.286"/><rect class="cell" x="570.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.210"/><rect class="cell" x="581.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.222"/><rect class="cell" x="593.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.293"/><rect class="cell" x="605.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.442"/><rect class="cell" x="616.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.586"/><rect class="cell" x="628.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.697"/><rect class="cell" x="640.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.737"/><rect class="cell" x="651.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.689"/><rect class="cell" x="663.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.599"/><rect class="cell" x="675.00" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.466"/><rect class="cell" x="686.67" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.424"/><rect class="cell" x="698.33" y="258.33" width="11.07" height="11.07" style="fill-opacity:0.384"/><rect class="cell" x="430.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.285"/><rect class="cell" x="441.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.302"/><rect class="cell" x="453.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.298"/><rect class="cell" x="465.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.228"/><rect class="cell" x="476.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.225"/><rect class="cell" x="488.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.295"/><rect class="cell" x="500.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.375"/><rect class="cell" x="511.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.452"/><rect class="cell" x="523.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.475"/><rect class="cell" x="535.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.461"/><rect class="cell" x="546.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.367"/><rect class="cell" x="558.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.276"/><rect class="cell" x="570.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.201"/><rect class="cell" x="581.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.214"/><rect class="cell" x="593.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.287"/><rect class="cell" x="605.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.434"/><rect class="cell" x="616.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.578"/><rect class="cell" x="628.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.686"/><rect class="cell" x="640.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.725"/><rect class="cell" x="651.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.676"/><rect class="cell" x="663.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.586"/><rect class="cell" x="675.00" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.456"/><rect class="cell" x="686.67" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.413"/><rect class="cell" x="698.33" y="270.00" width="11.07" height="11.07" style="fill-opacity:0.374"/><rect class="cell" x="430.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.203"/><rect class="cell" x="441.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.207"/><rect class="cell" x="453.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.205"/><rect class="cell" x="465.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.130"/><rect class="cell" x="476.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.125"/><rect class="cell" x="488.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.197"/><rect class="cell" x="500.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.278"/><rect class="cell" x="511.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.363"/><rect class="cell" x="523.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.381"/><rect class="cell" x="535.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.369"/><rect class="cell" x="546.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.271"/><rect class="cell" x="558.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.179"/><rect class="cell" x="570.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.097"/><rect class="cell" x="581.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.109"/><rect class="cell" x="593.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.187"/><rect class="cell" x="605.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.336"/><rect class="cell" x="616.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.487"/><rect class="cell" x="628.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.591"/><rect class="cell" x="640.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.632"/><rect class="cell" x="651.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.575"/><rect class="cell" x="663.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.483"/><rect class="cell" x="675.00" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.358"/><rect class="cell" x="686.67" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.317"/><rect class="cell" x="698.33" y="281.67" width="11.07" height="11.07" style="fill-opacity:0.295"/><rect class="cell" x="430.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.093"/><rect class="cell" x="441.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.073"/><rect class="cell" x="453.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.074"/><rect class="cell" x="465.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.064"/><rect class="cell" x="500.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.145"/><rect class="cell" x="511.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.240"/><rect class="cell" x="523.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.251"/><rect class="cell" x="535.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.243"/><rect class="cell" x="546.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.139"/><rect class="cell" x="558.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.049"/><rect class="cell" x="570.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.058"/><rect class="cell" x="605.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.202"/><rect class="cell" x="616.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.362"/><rect class="cell" x="628.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.462"/><rect class="cell" x="640.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.504"/><rect class="cell" x="651.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.444"/><rect class="cell" x="663.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.350"/><rect class="cell" x="675.00" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.229"/><rect class="cell" x="686.67" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.187"/><rect class="cell" x="698.33" y="293.33" width="11.07" height="11.07" style="fill-opacity:0.188"/><rect class="cell" x="430.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="441.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="453.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="465.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="511.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.091"/><rect class="cell" x="523.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.094"/><rect class="cell" x="535.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.092"/><rect class="cell" x="546.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="558.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.045"/><rect class="cell" x="616.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.206"/><rect class="cell" x="628.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.297"/><rect class="cell" x="640.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.344"/><rect class="cell" x="651.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.286"/><rect class="cell" x="663.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.193"/><rect class="cell" x="675.00" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.078"/><rect class="cell" x="686.67" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.037"/><rect class="cell" x="698.33" y="305.00" width="11.07" height="11.07" style="fill-opacity:0.063"/><rect class="cell" x="430.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="441.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="453.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="465.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="511.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.093"/><rect class="cell" x="523.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.097"/><rect class="cell" x="535.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.096"/><rect class="cell" x="546.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="558.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.046"/><rect class="cell" x="616.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.213"/><rect class="cell" x="628.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.313"/><rect class="cell" x="640.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.363"/><rect class="cell" x="651.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.302"/><rect class="cell" x="663.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.205"/><rect class="cell" x="675.00" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.086"/><rect class="cell" x="686.67" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.043"/><rect class="cell" x="698.33" y="316.67" width="11.07" height="11.07" style="fill-opacity:0.071"/><rect class="cell" x="430.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.069"/><rect class="cell" x="441.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.042"/><rect class="cell" x="453.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.035"/><rect class="cell" x="465.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="476.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="488.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="500.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.084"/><rect class="cell" x="511.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.176"/><rect class="cell" x="523.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.185"/><rect class="cell" x="535.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.182"/><rect class="cell" x="546.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.090"/><rect class="cell" x="558.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="570.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="581.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="593.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.030"/><rect class="cell" x="605.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.137"/><rect class="cell" x="616.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.280"/><rect class="cell" x="628.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.374"/><rect class="cell" x="640.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.419"/><rect class="cell" x="651.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.364"/><rect class="cell" x="663.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.280"/><rect class="cell" x="675.00" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.176"/><rect class="cell" x="686.67" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.141"/><rect class="cell" x="698.33" y="328.33" width="11.07" height="11.07" style="fill-opacity:0.151"/><rect class="box-on" x="640.00" y="118.33" width="11.67" height="11.67" style="fill-opacity:0"/><text class="t-muted" x="60.0" y="364.0" text-anchor="start">Bright = similar (up to 1), faint = unlike (down to about −0.3). The neighbours of a row are similar to it, far rows are not;</text><text class="t-muted" x="60.0" y="380.0" text-anchor="start">the stretched grid keeps that map and only makes it smoother.</text></svg><figcaption>Cosine similarity of one position row with all others, before and after the 2D interpolation. Left: the 14×14 grid of google/vit-base-patch16-224, row (3, 10), bright near itself and dark far away. Right: the 24×24 grid after the bilinear stretch, the same place (5, 18): the same map, stretched. Part 5 reads these maps in full.</figcaption></figure>

**Does it work?** We ran the ImageNet-fine-tuned ViT-B/16 on the 3,925 validation pictures of Imagenette. We used 224 and 384 pixels, and five versions of the position table. There is no training here at all; only the position table changes.

```text
== 4. accuracy on the Imagenette validation set: 3925 pictures (160 px on the short side), device mps ==
  the ImageNet head of the checkpoint is read at the 10 Imagenette classes; no training; "of 1000" = the top class of all 1000 must be right
  224, the training resolution (196 patches)     accuracy 99.54% (of 10)   83.13% (of 1000)   [65 s]
  384, 2D bilinear (the official code)           accuracy 99.69% (of 10)   84.76% (of 1000)   [196 s]
  384, 2D bicubic (the library)                  accuracy 99.69% (of 10)   84.97% (of 1000)   [1582 s]
  384, 1D: the 196 rows stretched as one list    accuracy 92.00% (of 10)   48.38% (of 1000)   [1583 s]
  384, no position rows at all (zeros)           accuracy 52.64% (of 10)   13.99% (of 1000)   [297 s]
```

The two 2D stretches keep the model working at 384. Over all 1,000 ImageNet classes, accuracy is 84.76% with the official bilinear table and 84.97% with the library's bicubic one, against 83.13% at 224. Read at only the 10 Imagenette classes, all three are almost perfect (99.54% to 99.69%).

The wrong stretch is a disaster. Stretching the 196 rows as one list of 576 drops the accuracy to 48.38%. A 1D stretch treats the last patch of one grid row and the first patch of the next as neighbours. So most new rows land in the wrong place. With no position rows at all the model falls to 13.99%; [Part 2](https://ishwarj.com/papers/vit/part-2-the-model/) showed why, since attention alone sees only a bag of patches.

Two honest limits. This model was never fine-tuned at 384, and the paper always fine-tunes after the stretch. Our numbers show only that the stretched table is a good start. And Imagenette stores its pictures at 160 pixels, so at 384 they are only upscaled, with no new detail. The small gain at 384 cannot come from detail. It fits the explanation of Touvron et al. that objects simply look bigger.

The times in brackets include waiting for another job on the same GPU, so they say nothing about speed.

<figure class="fig"><svg viewBox="0 0 800 270" role="img" aria-label="Bars of accuracy for five position tables: 224 pixels, 384 with 2D bilinear, 384 with 2D bicubic, 384 with a 1D stretch and 384 with no position rows; the two 2D versions keep the accuracy, the other two lose much of it."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="46.0" text-anchor="start">ViT-B/16 (ImageNet fine-tuned) on 3,925 Imagenette pictures, top-1 of all 1000 classes, no training</text><text class="t-tick" x="232.0" y="77.0" text-anchor="end">224: the training size</text><g class="mark"><title>224: the training size: 83.13%</title><rect class="s1" x="240.0" y="59.0" width="332.5" height="26.0" rx="3"/></g><text class="t-val" x="578.5" y="77.0" text-anchor="start">83.13%</text><text class="t-tick" x="232.0" y="111.0" text-anchor="end">384: 2D bilinear (official)</text><g class="mark"><title>384: 2D bilinear (official): 84.76%</title><rect class="s1" x="240.0" y="93.0" width="339.1" height="26.0" rx="3"/></g><text class="t-val" x="585.1" y="111.0" text-anchor="start">84.76%</text><text class="t-tick" x="232.0" y="145.0" text-anchor="end">384: 2D bicubic (library)</text><g class="mark"><title>384: 2D bicubic (library): 84.97%</title><rect class="s1" x="240.0" y="127.0" width="339.9" height="26.0" rx="3"/></g><text class="t-val" x="585.9" y="145.0" text-anchor="start">84.97%</text><text class="t-tick" x="232.0" y="179.0" text-anchor="end">384: 1D stretch of the list</text><g class="mark"><title>384: 1D stretch of the list: 48.38%</title><rect class="s1" x="240.0" y="161.0" width="193.5" height="26.0" rx="3"/></g><text class="t-val" x="439.5" y="179.0" text-anchor="start">48.38%</text><text class="t-tick" x="232.0" y="213.0" text-anchor="end">384: no position rows</text><g class="mark"><title>384: no position rows: 13.99%</title><rect class="s1" x="240.0" y="195.0" width="55.9" height="26.0" rx="3"/></g><text class="t-val" x="301.9" y="213.0" text-anchor="start">13.99%</text><text class="t-muted" x="20.0" y="250.0" text-anchor="start">Read at only the 10 Imagenette classes: 99.54%, 99.69%, 99.69%, 92.00%, 52.64% (same order).</text></svg><figcaption>Accuracy of the released ViT-B/16 (ImageNet fine-tuned) on the 3,925 Imagenette validation pictures, scored over all 1,000 ImageNet classes, with five versions of the position table and no training. The 2D stretches keep the model working at 384; a 1D stretch of the list or no position rows at all break it.</figcaption></figure>

## The experiments: three model families {§4}

> [!PAPER] · §4
> [![The opening of Section 4, Experiments: we evaluate the representation learning capabilities of ResNet, Vision Transformer (ViT), and the hybrid. To understand the data requirements of each model, we pre-train on datasets of varying size and evaluate many benchmark tasks. When considering the computational cost of pre-training the model, ViT performs very favourably, attaining state of the art on most recognition benchmarks at a lower pre-training cost. Lastly, we perform a small experiment using self-supervision](https://ishwarj.com/img/papers/vit/p3-experiments-intro.pngperiments-intro.png)](/img/papers/vit/p3-experiments-intro.png)
>
> **What it says:** the paper evaluates "the representation learning capabilities of ResNet, Vision Transformer (ViT), and the hybrid", pre-trained on "datasets of varying size". ViT reaches the state of the art "at a lower pre-training cost". A small experiment uses "self-supervision".
>
> **What it means:**
>
> This paragraph is the table of contents of the results. Three model families, three dataset sizes, many tasks, and the cost of each run. [Part 4](https://ishwarj.com/papers/vit/part-4-results/) reads the results, and Part 5 reads the cost study and self-supervision.
>
> "Representation learning" is the frame for everything. The real product of pre-training is not the head, which is thrown away. It is the body, which turns any picture into a useful vector. Every test that follows asks how good that vector is for a new task.

> [!DEFINITION] Representation
> The vector a model produces for an input, before any task-specific head: for ViT, the 768 numbers of $$\mathbf{y}$$ (Equation 4). A good representation puts pictures of the same kind close together, so that a simple head can separate the classes.

## Datasets {§4.1}

> [!PAPER] · §4.1
> [![Section 4.1, Datasets: to explore model scalability, we use the ILSVRC-2012 ImageNet dataset with 1k classes and 1.3M images, its superset ImageNet-21k with 21k classes and 14M images, and JFT with 18k classes and 303M high-resolution images. We de-duplicate the pre-training datasets with respect to the test sets of the downstream tasks following Kolesnikov et al. (2020)](https://ishwarj.com/img/papers/vit/p3-datasets.pngvit/p3-datasets.png)](/img/papers/vit/p3-datasets.png)
>
> **What it says:** three pre-training sets of growing size: ImageNet ("1k classes and 1.3M images"), ImageNet-21k ("21k classes and 14M images") and JFT ("18k classes and 303M high-resolution images"). "We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks."
>
> **What it means:**
>
> Each step up is 10 to 20 times bigger in pictures: 1.3 million, 14 million, 303 million. That ladder is the paper's main experiment. Train the same model on each rung and see when ViT starts to beat ResNets ([Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/) defines the three sets).
>
> De-duplication is a quiet but important step. A collection of 303 million pictures can easily hold copies of test pictures of CIFAR or Pets. A model that saw the test picture in pre-training would get an unfair head start. So near-copies of every downstream test picture are removed from the pre-training data first.

> [!DEFINITION] De-duplication
> Removing near-copies of the test pictures from the training data, so that a test measures what the model learned, not what it memorised. The method follows the BiT paper (Kolesnikov et al., 2020).

<figure class="fig"><svg viewBox="0 0 800 244" role="img" aria-label="Bars on a log scale of the number of training pictures: Imagenette 9,469 (ours), ImageNet 1.3 million, ImageNet-21k 14 million and JFT-300M 303 million."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Training pictures, log scale</text><line class="grid" x1="170.0" y1="40" x2="170.0" y2="210"/><text class="t-tick" x="170.0" y="228.0" text-anchor="middle">1k</text><line class="grid" x1="256.7" y1="40" x2="256.7" y2="210"/><text class="t-tick" x="256.7" y="228.0" text-anchor="middle">10k</text><line class="grid" x1="343.3" y1="40" x2="343.3" y2="210"/><text class="t-tick" x="343.3" y="228.0" text-anchor="middle">100k</text><line class="grid" x1="430.0" y1="40" x2="430.0" y2="210"/><text class="t-tick" x="430.0" y="228.0" text-anchor="middle">1M</text><line class="grid" x1="516.7" y1="40" x2="516.7" y2="210"/><text class="t-tick" x="516.7" y="228.0" text-anchor="middle">10M</text><line class="grid" x1="603.3" y1="40" x2="603.3" y2="210"/><text class="t-tick" x="603.3" y="228.0" text-anchor="middle">100M</text><line class="grid" x1="690.0" y1="40" x2="690.0" y2="210"/><text class="t-tick" x="690.0" y="228.0" text-anchor="middle">1B</text><text class="t-tick t-strong" x="162.0" y="68.0" text-anchor="end">Imagenette (ours)</text><g class="mark"><title>Imagenette (ours): 9,469 pictures</title><rect class="s2" x="170" y="54" width="84.6" height="20" rx="3"/></g><text class="t-tick" x="260.6" y="69.0" text-anchor="start">9,469, 10 classes</text><text class="t-tick t-strong" x="162.0" y="108.0" text-anchor="end">ImageNet</text><g class="mark"><title>ImageNet: 1,300,000 pictures</title><rect class="s1" x="170" y="94" width="269.9" height="20" rx="3"/></g><text class="t-tick" x="445.9" y="109.0" text-anchor="start">1.3M, 1k classes</text><text class="t-tick t-strong" x="162.0" y="148.0" text-anchor="end">ImageNet-21k</text><g class="mark"><title>ImageNet-21k: 14,000,000 pictures</title><rect class="s1" x="170" y="134" width="359.3" height="20" rx="3"/></g><text class="t-tick" x="535.3" y="149.0" text-anchor="start">14M, 21k classes</text><text class="t-tick t-strong" x="162.0" y="188.0" text-anchor="end">JFT-300M</text><g class="mark"><title>JFT-300M: 303,000,000 pictures</title><rect class="s1" x="170" y="174" width="475.1" height="20" rx="3"/></g><text class="t-tick" x="651.1" y="189.0" text-anchor="start">303M, 18k classes</text></svg><figcaption>The three pre-training datasets of the paper on a log scale, with our fine-tuning set for comparison. ImageNet: 1.3M pictures, 1k classes. ImageNet-21k: 14M, 21k classes. JFT-300M: 303M, 18k classes. Imagenette, which we use: 9,469 training pictures, 10 classes, 31,999 times fewer than JFT.</figcaption></figure>

> [!PAPER] · §4.1
> [![Section 4.1 continued: we transfer the models trained on these dataset to several benchmark tasks: ImageNet on the original validation labels and the cleaned-up ReaL labels (Beyer et al., 2020), CIFAR-10/100 (Krizhevsky, 2009), Oxford-IIIT Pets (Parkhi et al., 2012), and Oxford Flowers-102 (Nilsback and Zisserman, 2008). For these datasets, pre-processing follows Kolesnikov et al. (2020)](https://ishwarj.com/img/papers/vit/p3-transfer-tasks.png-transfer-tasks.png)](/img/papers/vit/p3-transfer-tasks.png)
>
> **What it says:** the pre-trained models are transferred to ImageNet (with its original labels and "the cleaned-up ReaL labels"), "CIFAR-10/100", "Oxford-IIIT Pets" and "Oxford Flowers-102". The "pre-processing follows Kolesnikov et al. (2020)".
>
> **What it means:**
>
> These are the downstream tasks of Table 2. There is a big one (ImageNet), two with tiny 32 × 32 pictures (CIFAR) and two fine-grained ones (pet breeds and flower species). Together they test whether one pre-trained body helps on very different problems.
>
> Copying the pre-processing of BiT matters for a fair fight. BiT is the ResNet baseline, so both families see the same crops and the same picture sizes.

> [!DEFINITION] ImageNet ReaL
> "Reassessed Labels" (Beyer et al., 2020): new labels for the ImageNet validation pictures, collected again by people. Some original labels were wrong or too narrow (a picture with a dog and a ball has one label). ReaL allows several correct labels, so it rewards models that are right and were marked wrong.

> [!DEFINITION] CIFAR-10, Oxford-IIIT Pets, Oxford Flowers-102
> Three standard benchmarks. CIFAR-10 has small pictures of 32 × 32 pixels in 10 classes; CIFAR-100 ([Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/)) has the same pictures sorted into 100 classes. Pets has 37 breeds of cats and dogs, and Flowers-102 has 102 flower species. Both are small, fine-grained sets.

> [!PAPER] · §4.1
> [![Section 4.1 continued: we also evaluate on the 19-task VTAB classification suite (Zhai et al., 2019b). VTAB evaluates low-data transfer to diverse tasks, using 1 000 training examples per task. The tasks are divided into three groups: Natural, tasks like the above, Pets, CIFAR, etc.; Specialized, medical and satellite imagery; and Structured, tasks that require geometric understanding like localization](https://ishwarj.com/img/papers/vit/p3-vtab.pngers/vit/p3-vtab.png)](/img/papers/vit/p3-vtab.png)
>
> **What it says:** "the 19-task VTAB classification suite" tests "low-data transfer to diverse tasks, using 1 000 training examples per task", in three groups: "Natural", "Specialized" (medical and satellite pictures) and "Structured" ("geometric understanding like localization").
>
> **What it means:**
>
> VTAB ([Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/)) is the hard exam. Each task gives only 1,000 labelled pictures. Some tasks look nothing like photos: a microscope slide, a satellite tile, or a rendered scene that asks how far away an object is.
>
> The three groups matter because a model can be good at one and poor at another. Natural tasks reward knowing objects. Structured tasks reward knowing where things are and how many there are, which a classifier of photos never practised. Figure 2 in Part 4 breaks the score down by group.

## Model variants and the "/16" in the name {§4.1}

> [!PAPER] · §4.1, Model Variants
> [![The Model Variants paragraph: we base ViT configurations on those used for BERT, as summarized in Table 1. The Base and Large models are directly adopted from BERT and we add the larger Huge model. We use brief notation to indicate the model size and the input patch size: for instance, ViT-L/16 means the Large variant with 16 by 16 input patch size. Note that the Transformer's sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive](https://ishwarj.com/img/papers/vit/p3-variants.pngvit/p3-variants.png)](/img/papers/vit/p3-variants.png)
> [![Table 1, details of Vision Transformer model variants: ViT-Base 12 layers, hidden size 768, MLP size 3072, 12 heads, 86M parameters; ViT-Large 24, 1024, 4096, 16, 307M; ViT-Huge 32, 1280, 5120, 16, 632M](https://ishwarj.com/img/papers/vit/p3-table1.pngs/vit/p3-table1.png)](/img/papers/vit/p3-table1.png)
>
> **What it says:** "Base" and "Large" are "directly adopted from BERT", and the paper adds "the larger 'Huge' model". "ViT-L/16 means the 'Large' variant with 16×16 input patch size." "The Transformer's sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive."
>
> **What it means:**
>
> A ViT name has two parts. The letter is the size of the Transformer (Table 1: Base, Large, Huge). The number is the side of a patch in pixels. ViT-B/32 and ViT-B/16 are the same network, fed differently cut pictures.
>
> "Inversely proportional to the square" is the key to reading every result table. Halve the patch side, and each side of the picture holds twice as many patches, so there are four times as many patches. The weights barely change, but the work grows about four times. A smaller patch is a more expensive model, not a bigger one.

[Part 2](https://ishwarj.com/papers/vit/part-2-the-model/) counted the parameters of each size from a formula. Here we add the patch size and the resolution. `vit_part3_setup.py` computes the sequence length and an estimate of the forward compute from the shapes, and counts the numbers in each released weight file:

```text
== 1. Table 1 and the patch-size notation: ViT-<size>/<patch> ==
  N = (resolution / patch)^2 patches, plus 1 [class] token; attention compares every pair: (N+1)^2 pairs per head
  ViT-B/32 at 224: grid  7x7  N =   49  sequence   50  pairs per head     2,500  forward compute about    8.6 GFLOPs
  ViT-B/32 at 384: grid 12x12 N =  144  sequence  145  pairs per head    21,025  forward compute about   25.4 GFLOPs
  ViT-B/16 at 224: grid 14x14 N =  196  sequence  197  pairs per head    38,809  forward compute about   34.9 GFLOPs
  ViT-B/16 at 384: grid 24x24 N =  576  sequence  577  pairs per head   332,929  forward compute about  110.3 GFLOPs
  ViT-L/32 at 224: grid  7x7  N =   49  sequence   50  pairs per head     2,500  forward compute about   30.4 GFLOPs
  ViT-L/32 at 384: grid 12x12 N =  144  sequence  145  pairs per head    21,025  forward compute about   89.6 GFLOPs
  ViT-L/16 at 224: grid 14x14 N =  196  sequence  197  pairs per head    38,809  forward compute about  122.8 GFLOPs
  ViT-L/16 at 384: grid 24x24 N =  576  sequence  577  pairs per head   332,929  forward compute about  381.2 GFLOPs
  ViT-H/14 at 224: grid 16x16 N =  256  sequence  257  pairs per head    66,049  forward compute about  334.2 GFLOPs
  ViT-H/14 at 384: grid 27x27 N =  729  sequence  730  pairs per head   532,900  forward compute about 1005.9 GFLOPs
  ViT-B/16 vs ViT-B/32 at 224: 196 vs 49 patches = x4 (patch side halved, so (1/2)^-2 = 4); compute x4.06
  Table 2 fine-tuning resolution: ViT-L/16 at 512: grid 32x32, N = 1024, sequence 1025
  Table 2 fine-tuning resolution: ViT-H/14 at 518: grid 37x37, N = 1369, sequence 1370

-- parameters of the released checkpoints, read from the weight files (no head, no pre-logits layer) --
  ViT-B/32 (vit-base-patch32-224-in21k, weight file): backbone  87,455,232  =   87.5M   pre-logits layer   590,592   Table 1: 86M
  ViT-B/16 (vit-base-patch16-224-in21k, weight file): backbone  85,798,656  =   85.8M   pre-logits layer   590,592   Table 1: 86M
  ViT-L/32 (vit-large-patch32-224-in21k, weight file): backbone 305,510,400  =  305.5M   pre-logits layer 1,049,600   Table 1: 307M
  ViT-L/16 (vit-large-patch16-224, weight file): backbone 303,301,632  =  303.3M   pre-logits layer         0   Table 1: 307M
  ViT-H/14 (vit-huge-patch14-224-in21k, weight file): backbone 630,764,800  =  630.8M   pre-logits layer 1,639,680   Table 1: 632M
  ViT-B/32 minus ViT-B/16: 1,656,576 parameters (1.9% more), because E grows from 768x768 to 3072x768 (+1,769,472) and the position table shrinks from 197 to 50 rows (-112,896)
  ViT-L/32 with its pre-logits layer: 306.6M; ViT-L/16 has 303.3M without one
```

Three things stand out. First, ViT-B/16 needs 4.06 times the compute of ViT-B/32 at 224 pixels, almost exactly the factor 4 of the paper's sentence. The extra 0.06 comes from attention, whose cost grows with the square of the sequence.

Second, ViT-B/32 has 1.9% *more* parameters than ViT-B/16, not fewer: its patch embedding is $$3{,}072 \times 768$$ instead of $$768 \times 768$$. Parameters and compute are separate dials.

Third, the 307M of ViT-Large in Table 1 is close to ViT-L/32 with its pre-logits layer (306.6M). ViT-L/16 has 303.3M without one; its only cached file is the ImageNet fine-tuned one, which has no pre-logits layer. Part 2 already flagged this gap. The table may count a different version; we cannot tell which.

<figure class="fig"><svg viewBox="0 0 800 328" role="img" aria-label="Horizontal bars of the estimated forward compute for ViT-B/32, B/16, L/32, L/16 and H/14 at 224 and 384 pixels, with the sequence length of each; smaller patches and larger pictures both multiply the work."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Forward compute of one picture (billions of operations), at 224 and 384 pixels</text><text class="t-tick t-strong" x="102.0" y="70.0" text-anchor="end">ViT-B/32</text><g class="mark"><title>ViT-B/32 at 224: 8.6 GFLOPs, sequence 50</title><rect class="s1" x="110" y="52" width="4.4" height="14" rx="3"/></g><text class="t-tick" x="120.4" y="64.0" text-anchor="start">8.6   (50 tokens)</text><g class="mark"><title>ViT-B/32 at 384: 25.4 GFLOPs, sequence 145</title><rect class="s2" x="110" y="70" width="13.1" height="14" rx="3"/></g><text class="t-tick" x="129.1" y="82.0" text-anchor="start">25.4   (145 tokens)</text><text class="t-tick t-strong" x="102.0" y="116.0" text-anchor="end">ViT-B/16</text><g class="mark"><title>ViT-B/16 at 224: 34.9 GFLOPs, sequence 197</title><rect class="s1" x="110" y="98" width="18.0" height="14" rx="3"/></g><text class="t-tick" x="134.0" y="110.0" text-anchor="start">34.9   (197 tokens)</text><g class="mark"><title>ViT-B/16 at 384: 110.3 GFLOPs, sequence 577</title><rect class="s2" x="110" y="116" width="57.0" height="14" rx="3"/></g><text class="t-tick" x="173.0" y="128.0" text-anchor="start">110.3   (577 tokens)</text><text class="t-tick t-strong" x="102.0" y="162.0" text-anchor="end">ViT-L/32</text><g class="mark"><title>ViT-L/32 at 224: 30.4 GFLOPs, sequence 50</title><rect class="s1" x="110" y="144" width="15.7" height="14" rx="3"/></g><text class="t-tick" x="131.7" y="156.0" text-anchor="start">30.4   (50 tokens)</text><g class="mark"><title>ViT-L/32 at 384: 89.6 GFLOPs, sequence 145</title><rect class="s2" x="110" y="162" width="46.3" height="14" rx="3"/></g><text class="t-tick" x="162.3" y="174.0" text-anchor="start">89.6   (145 tokens)</text><text class="t-tick t-strong" x="102.0" y="208.0" text-anchor="end">ViT-L/16</text><g class="mark"><title>ViT-L/16 at 224: 122.8 GFLOPs, sequence 197</title><rect class="s1" x="110" y="190" width="63.5" height="14" rx="3"/></g><text class="t-tick" x="179.5" y="202.0" text-anchor="start">122.8   (197 tokens)</text><g class="mark"><title>ViT-L/16 at 384: 381.2 GFLOPs, sequence 577</title><rect class="s2" x="110" y="208" width="197.1" height="14" rx="3"/></g><text class="t-tick" x="313.1" y="220.0" text-anchor="start">381.2   (577 tokens)</text><text class="t-tick t-strong" x="102.0" y="254.0" text-anchor="end">ViT-H/14</text><g class="mark"><title>ViT-H/14 at 224: 334.2 GFLOPs, sequence 257</title><rect class="s1" x="110" y="236" width="172.8" height="14" rx="3"/></g><text class="t-tick" x="288.8" y="248.0" text-anchor="start">334.2   (257 tokens)</text><g class="mark"><title>ViT-H/14 at 384: 1005.9 GFLOPs, sequence 730</title><rect class="s2" x="110" y="254" width="520.0" height="14" rx="3"/></g><text class="t-tick" x="636.0" y="266.0" text-anchor="start">1,005.9   (730 tokens)</text><rect class="s1" x="560" y="40" width="12" height="12" rx="2"/><text class="t-tick" x="578.0" y="51.0" text-anchor="start">224 × 224</text><rect class="s2" x="660" y="40" width="12" height="12" rx="2"/><text class="t-tick" x="678.0" y="51.0" text-anchor="start">384 × 384</text><text class="t-muted" x="20.0" y="294.0" text-anchor="start">Estimate from the shapes: 2 × (12 N D² + 2 N² D) per layer, N tokens of width D. Halving the patch side</text><text class="t-muted" x="20.0" y="310.0" text-anchor="start">gives 4 times the tokens; ViT-B/16 costs 4.06 times ViT-B/32 while having 1.9% fewer parameters.</text></svg><figcaption>Patch size and resolution set the work, not the size of the model. For each variant, the bar is the forward compute of one picture in billions of operations (our estimate from the shapes) at 224 and at 384 pixels, with the sequence length written beside it. ViT-B/16 costs about 4 times ViT-B/32; going from 224 to 384 costs about 3 times more again.</figcaption></figure>

> [!DEFINITION] FLOPs
> Floating-point operations: the count of additions and multiplications a model performs. One multiply-add counts as 2. "GFLOPs" means billions. Our estimate counts the large matrix products of each layer: $$2 \times (12ND^2 + 2N^2D)$$ for $$N$$ tokens of width $$D$$ with an MLP of $$4D$$.

## The CNN baselines: ResNet (BiT) and Noisy Student {§4.1}

> [!PAPER] · §4.1, Model Variants
> [![For the baseline CNNs, we use ResNet (He et al., 2016), but replace the Batch Normalization layers (Ioffe and Szegedy, 2015) with Group Normalization (Wu and He, 2018), and used standardized convolutions (Qiao et al., 2019). These modifications improve transfer (Kolesnikov et al., 2020), and we denote the modified model ResNet (BiT)](https://ishwarj.com/img/papers/vit/p3-bit-baseline.pngp3-bit-baseline.png)](/img/papers/vit/p3-bit-baseline.png)
>
> **What it says:** the baseline is a ResNet in which the authors "replace the Batch Normalization layers" with "Group Normalization" and use "standardized convolutions". "These modifications improve transfer", and the result is called "ResNet (BiT)".
>
> **What it means:**
>
> A fair race needs a strong opponent. A plain ResNet ([Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/)) is not the best CNN for transfer. So the paper uses the version that the BiT paper tuned for exactly this job: pre-train big, fine-tune small.
>
> The two changes are both about normalisation, the step that keeps numbers inside a network in a sane range. The standard version, Batch Norm, depends on the other pictures in the batch. The BiT version depends only on the picture itself. The tiny example below shows the difference.

> [!DEFINITION] Batch Normalization (BN)
> A layer that, for each channel, subtracts the mean and divides by the spread computed over all pictures in the current batch and all pixels. During training, a picture's output depends on which other pictures share its batch. At test time it uses running averages saved during training.

> [!DEFINITION] Group Normalization (GN)
> The same idea, but the mean and spread are computed inside one picture, over a group of channels. A picture's output never depends on the batch, so small batches and fine-tuning do not upset it.

> [!DEFINITION] Weight Standardization (WS)
> Normalising the weights instead of the activations: before a convolution uses a filter, its weights are shifted to mean 0 and scaled to spread 1. The paper calls it "standardized convolutions". It works well together with Group Norm.

The difference fits on a few numbers. Picture A has 4 channels of 2 pixels each. We normalise it once together with one partner picture and once with another:

```text
== 3. Batch Norm vs Group Norm on a tiny batch (4 channels, 2 pixels per channel, 2 groups) ==
  picture A, channel 1 = [1.0, 3.0]
  Batch Norm, A batched with partner 1: A channel 1 -> [-1.260, -0.700]
  Batch Norm, A batched with partner 2: A channel 1 -> [ 0.632,  1.265]   (same picture, different output)
  Group Norm, group 1 = channels 1 and 2 of A = [1.0, 3.0, 2.0, 2.0]: mean 2.000, std 0.707
  Group Norm: A channel 1 -> [-1.414,  1.414] with any partner (it never looks at the batch); torch GroupNorm agrees: True
  Weight Standardization: filter 2 = [1.0, 2.0, 3.0, 2.0]: mean 2.00, std 0.7071 -> [-1.414,  0.000,  1.414,  0.000]
  filter 1 = [0.2, -0.1, 0.5, 0.0] -> [ 0.218, -1.091,  1.528, -0.655]   (every filter now has mean 0 and std 1 before the convolution uses it)
```

With Batch Norm, the same picture A gives $$[-1.260, -0.700]$$ in one batch and $$[0.632, 1.265]$$ in another. Its output depends on a stranger. With Group Norm, channels 1 and 2 of A form a group with values $$1, 3, 2, 2$$: mean 2, spread $$\sqrt{(1 + 1 + 0 + 0)/4} = 0.707$$. So channel 1 becomes $$(1 - 2)/0.707 = -1.414$$ and $$(3 - 2)/0.707 = 1.414$$, whatever the batch. The released `microsoft/resnet-50` that Part 1 ran is a standard ResNet with 53 Batch Norm layers; ResNet (BiT) swaps each of them for Group Norm.

<figure class="fig"><svg viewBox="0 0 800 296" role="img" aria-label="Two batches of two pictures with four channels of two pixels. Batch Norm highlights channel 1 in both pictures, averaged together; Group Norm highlights channels 1 and 2 of picture A only."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Which numbers are averaged: Batch Norm against Group Norm</text><text class="t-note" x="170.0" y="56.0" text-anchor="middle">Batch Norm: channel 1 across the batch</text><text class="t-tick" x="90.0" y="80.0" text-anchor="middle">picture A</text><rect class="box-on" x="50.0" y="90.0" width="38.0" height="30.0" rx="4"/><rect class="box-on" x="90.0" y="90.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="40.0" y="110.0" text-anchor="end">c1</text><rect class="box" x="50.0" y="124.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="90.0" y="124.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="40.0" y="144.0" text-anchor="end">c2</text><rect class="box" x="50.0" y="158.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="90.0" y="158.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="40.0" y="178.0" text-anchor="end">c3</text><rect class="box" x="50.0" y="192.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="90.0" y="192.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="40.0" y="212.0" text-anchor="end">c4</text><text class="t-tick" x="250.0" y="80.0" text-anchor="middle">picture B</text><rect class="box-on" x="210.0" y="90.0" width="38.0" height="30.0" rx="4"/><rect class="box-on" x="250.0" y="90.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="200.0" y="110.0" text-anchor="end">c1</text><rect class="box" x="210.0" y="124.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="250.0" y="124.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="200.0" y="144.0" text-anchor="end">c2</text><rect class="box" x="210.0" y="158.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="250.0" y="158.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="200.0" y="178.0" text-anchor="end">c3</text><rect class="box" x="210.0" y="192.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="250.0" y="192.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="200.0" y="212.0" text-anchor="end">c4</text><text class="t-note" x="570.0" y="56.0" text-anchor="middle">Group Norm: channels 1 and 2 of one picture</text><text class="t-tick" x="490.0" y="80.0" text-anchor="middle">picture A</text><rect class="box-on" x="450.0" y="90.0" width="38.0" height="30.0" rx="4"/><rect class="box-on" x="490.0" y="90.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="440.0" y="110.0" text-anchor="end">c1</text><rect class="box-on" x="450.0" y="124.0" width="38.0" height="30.0" rx="4"/><rect class="box-on" x="490.0" y="124.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="440.0" y="144.0" text-anchor="end">c2</text><rect class="box" x="450.0" y="158.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="490.0" y="158.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="440.0" y="178.0" text-anchor="end">c3</text><rect class="box" x="450.0" y="192.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="490.0" y="192.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="440.0" y="212.0" text-anchor="end">c4</text><text class="t-tick" x="650.0" y="80.0" text-anchor="middle">picture B</text><rect class="box" x="610.0" y="90.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="650.0" y="90.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="600.0" y="110.0" text-anchor="end">c1</text><rect class="box" x="610.0" y="124.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="650.0" y="124.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="600.0" y="144.0" text-anchor="end">c2</text><rect class="box" x="610.0" y="158.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="650.0" y="158.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="600.0" y="178.0" text-anchor="end">c3</text><rect class="box" x="610.0" y="192.0" width="38.0" height="30.0" rx="4"/><rect class="box" x="650.0" y="192.0" width="38.0" height="30.0" rx="4"/><text class="t-tick" x="600.0" y="212.0" text-anchor="end">c4</text><text class="t-muted" x="20.0" y="250.0" text-anchor="start">Tiny example, picture A channel 1 = [1.0, 3.0]: with Batch Norm it becomes [-1.260, -0.700] with one partner and [0.632, 1.265] with another.</text><text class="t-muted" x="20.0" y="268.0" text-anchor="start">With Group Norm (group mean 2.0, std 0.707) it is always [-1.414, 1.414]: the batch never enters.</text></svg><figcaption>Which numbers are averaged. A batch of two pictures, each with four channels. Batch Norm (left) takes the mean of one channel across both pictures, so picture A's result depends on picture B. Group Norm (right) takes the mean of a group of channels inside one picture. The numbers are the tiny example: A's first channel becomes −1.260, −0.700 or 0.632, 1.265 with Batch Norm, and always −1.414, 1.414 with Group Norm.</figcaption></figure>

The BiT paper gives the reason in its own words:

> [!PAPER] Kolesnikov et al. (2020), Big Transfer (BiT) · §2.1
> [![The BiT paper, Section 2.1: the second component is Group Normalization (GN) and Weight Standardization (WS). Batch Normalization (BN) is used in most state-of-the-art vision models to stabilize training. However, we find that BN is detrimental to Big Transfer for two reasons. First, when training large models with small per-device batches, BN performs poorly or incurs inter-device synchronization cost. Second, due to the requirement to update running statistics, BN is detrimental for transfer. GN, when combined with WS, has been shown to improve performance on small-batch training](https://ishwarj.com/img/papers/vit/p3-ext-bit-gnws.pngp3-ext-bit-gnws.png)](/img/papers/vit/p3-ext-bit-gnws.png)
>
> **What it says:** "BN is detrimental to Big Transfer for two reasons." With "small per-device batches" it "performs poorly or incurs inter-device synchronization cost", and the "requirement to update running statistics" hurts transfer. GN with WS fixes both.
>
> **What it means:**
>
> A big model trained on many chips sees only a few pictures per chip. Batch Norm then either averages over too few pictures, or must stop and share its averages across chips, which is slow.
>
> The second reason is about fine-tuning. Batch Norm carries running averages from pre-training, and a new task with different pictures makes them wrong. Group Norm has no running averages to go stale. ViT has the same property: its LayerNorm (Part 2) also works inside one token at a time.

> [!DEFINITION] BiT (Big Transfer)
> A recipe from the same Google team (Kolesnikov et al., 2020). Large ResNets with Group Norm and Weight Standardization are pre-trained on ImageNet-21k (BiT-M) or JFT-300M (BiT-L), then fine-tuned with one fixed rule. BiT-L is the main opponent in Table 2. The paper names ResNets by depth and width: R152x4 is 152 layers, four times wider than usual.

The second opponent appears in Section 4.2 ([Part 4](https://ishwarj.com/papers/vit/part-4-results/)): Noisy Student, which the paper calls the state of the art on ImageNet at the time. It is a different kind of CNN trained in a different way:

> [!PAPER] Xie et al. (2020), Self-training with Noisy Student · Abstract
> [![The abstract of Noisy Student: we present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant; it achieves 88.4% top-1 accuracy on ImageNet. We first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images, and iterate. During the learning of the student, we inject noise such as dropout, stochastic depth, and data augmentation via RandAugment](https://ishwarj.com/img/papers/vit/p3-ext-noisy.pngit/p3-ext-noisy.png)](/img/papers/vit/p3-ext-noisy.png)
>
> **What it says:** "a semi-supervised learning approach" that reaches 88.4% on ImageNet. A teacher EfficientNet "generate[s] pseudo labels for 300M unlabeled images"; a "larger EfficientNet as a student model" learns from both, with noise such as "dropout, stochastic depth" and RandAugment; the student then becomes the next teacher.
>
> **What it means:**
>
> Noisy Student uses the same 300 million JFT pictures as ViT, but without their labels. A teacher model labels them itself, and a bigger student learns from those guesses while being made to work under noise. Repeat a few times, and the student passes the teacher.
>
> So the two opponents of ViT are both strong in different ways. BiT is the fair comparison: same team, same data, same transfer recipe, only the architecture differs. Noisy Student is the record to beat on ImageNet, with a very different method.

> [!DEFINITION] Semi-supervised learning and pseudo labels
> Training with a few labelled pictures and many unlabelled ones. A **pseudo label** is a label that a trained model guesses for an unlabelled picture and that is then used as if it were true.

> [!DEFINITION] EfficientNet
> A family of CNNs (Tan and Le, 2019) whose depth, width and picture size are scaled together by one rule. Noisy Student's largest model is EfficientNet-L2.

## The hybrids: which ResNet stage feeds ViT {§4.1}

> [!PAPER] · §4.1, Model Variants
> [![For the hybrids, we feed the intermediate feature maps into ViT with patch size of one pixel. To experiment with different sequence lengths, we either (i) take the output of stage 4 of a regular ResNet50 or (ii) remove stage 4, place the same number of layers in stage 3 (keeping the total number of layers), and take the output of this extended stage 3. Option (ii) results in a 4x longer sequence length, and a more expensive ViT model](https://ishwarj.com/img/papers/vit/p3-hybrid-stages.png3-hybrid-stages.png)](/img/papers/vit/p3-hybrid-stages.png)
>
> **What it says:** the hybrids read a CNN feature map "with patch size of one 'pixel'". Either "(i) take the output of stage 4 of a regular ResNet50" or "(ii) remove stage 4, place the same number of layers in stage 3", and "take the output of this extended stage 3". Option (ii) "results in a 4x longer sequence length".
>
> **What it means:**
>
> [Part 2](https://ishwarj.com/papers/vit/part-2-the-model/) defined the hybrid: a ResNet does the first steps of seeing, and each position of its feature map becomes a token. This paragraph says which feature map. A ResNet halves its grid at each stage, so the later the stage, the shorter the sequence.
>
> Option (ii) is a clever trade. Moving stage 4's layers into stage 3 keeps the CNN's depth but skips the last halving. The grid stays at 14 × 14, so the hybrid gets 196 tokens, the same as ViT-B/16. The Transformer part then costs about four times as much as in option (i).

The real ResNet-50 shows the four grids on one Imagenette picture:

```text
== 2. the hybrid: which ResNet-50 stage feeds ViT ==
  stage 1 output:  256 channels x 56 x 56 -> as 1x1 patches:  3136 tokens of 256 numbers
  stage 2 output:  512 channels x 28 x 28 -> as 1x1 patches:   784 tokens of 512 numbers
  stage 3 output: 1024 channels x 14 x 14 -> as 1x1 patches:   196 tokens of 1024 numbers
  stage 4 output: 2048 channels x  7 x 7  -> as 1x1 patches:    49 tokens of 2048 numbers
  option (i), stage 4: 49 tokens; option (ii), extended stage 3: 196 tokens = x4 (the "4x longer sequence")
  microsoft/resnet-50 is a standard ResNet: it has 53 BatchNorm2d layers (BiT replaces each with Group Norm)
```

<figure class="fig"><svg viewBox="0 0 800 344" role="img" aria-label="The four stage outputs of ResNet-50 drawn as shrinking squares: 56 by 56, 28 by 28, 14 by 14 and 7 by 7, with 3,136, 784, 196 and 49 tokens; stage 3 and stage 4 are highlighted as the two hybrid options."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">ResNet-50 on a 224 × 224 picture: each stage halves the grid</text><rect class="box-1" x="30.0" y="60.0" width="160.0" height="160.0" rx="4"/><text class="t-tick t-strong" x="110.0" y="246.0" text-anchor="middle">stage 1</text><text class="t-tick" x="110.0" y="264.0" text-anchor="middle">56 × 56 × 256</text><text class="t-tick" x="110.0" y="282.0" text-anchor="middle">3,136 tokens</text><line class="edge" x1="198.0" y1="140.0" x2="282.0" y2="140.0" marker-end="url(#ah)"/><rect class="box-1" x="290.0" y="100.0" width="80.0" height="80.0" rx="4"/><text class="t-tick t-strong" x="330.0" y="246.0" text-anchor="middle">stage 2</text><text class="t-tick" x="330.0" y="264.0" text-anchor="middle">28 × 28 × 512</text><text class="t-tick" x="330.0" y="282.0" text-anchor="middle">784 tokens</text><line class="edge" x1="378.0" y1="140.0" x2="462.0" y2="140.0" marker-end="url(#ah)"/><rect class="box-on" x="470.0" y="120.0" width="40.0" height="40.0" rx="4"/><text class="t-tick t-strong" x="490.0" y="246.0" text-anchor="middle">stage 3</text><text class="t-tick" x="490.0" y="264.0" text-anchor="middle">14 × 14 × 1024</text><text class="t-tick" x="490.0" y="282.0" text-anchor="middle">196 tokens</text><line class="edge" x1="518.0" y1="140.0" x2="632.0" y2="140.0" marker-end="url(#ah)"/><rect class="box-4" x="640.0" y="130.0" width="20.0" height="20.0" rx="4"/><text class="t-tick t-strong" x="650.0" y="246.0" text-anchor="middle">stage 4</text><text class="t-tick" x="650.0" y="264.0" text-anchor="middle">7 × 7 × 2048</text><text class="t-tick" x="650.0" y="282.0" text-anchor="middle">49 tokens</text><text class="t-muted" x="30.0" y="316.0" text-anchor="start">Option (i) reads stage 4: 49 tokens of 2,048 numbers. Option (ii) moves stage 4's layers into stage 3 and reads its 14 × 14 grid:</text><text class="t-muted" x="30.0" y="332.0" text-anchor="start">196 tokens, 4 times longer, the same length as ViT-B/16. Each token becomes one "1 × 1 patch" for E.</text></svg><figcaption>The four stages of ResNet-50 on a 224×224 picture. Each stage halves the grid: 56×56, 28×28, 14×14, 7×7. Option (i) reads stage 4: 49 tokens. Option (ii) moves stage 4's layers into stage 3 and reads its 14×14 grid: 196 tokens, four times longer, as many as ViT-B/16.</figcaption></figure>

## Pre-training: Adam, a big batch and strong weight decay {§4.1}

> [!PAPER] · §4.1, Training and Fine-tuning
> [![Training and Fine-tuning: we train all models, including ResNets, using Adam (Kingma and Ba, 2015) with beta1 = 0.9, beta2 = 0.999, a batch size of 4096 and apply a high weight decay of 0.1, which we found to be useful for transfer of all models (Appendix D.1 shows that, in contrast to common practices, Adam works slightly better than SGD for ResNets in our setting). We use a linear learning rate warmup and decay, see Appendix B.1 for details](https://ishwarj.com/img/papers/vit/p3-training.pngvit/p3-training.png)](/img/papers/vit/p3-training.png)
>
> **What it says:** all models, "including ResNets", are trained "using Adam" with "$$\beta_1 = 0.9, \beta_2 = 0.999$$, a batch size of 4096" and "a high weight decay of 0.1", which was "useful for transfer of all models". The learning rate follows "a linear learning rate warmup and decay".
>
> **What it means:**
>
> Every model in the race gets the same training recipe, ResNets included. That is unusual: ResNets are normally trained with plain SGD. The authors checked that Adam suits their ResNets slightly better (Appendix D.1, in Part 4), so the baseline is not handicapped.
>
> Each piece of the sentence is a dial. Adam decides how each weight moves. The batch says how many pictures each step averages over. Weight decay pulls every weight a little towards zero, and the warm-up and decay set the step size over time. The boxes below define them, and a tiny example shows Adam's arithmetic.

> [!DEFINITION] Batch size
> How many pictures the model sees before it takes one step. The gradient of each step is the average over the batch. A batch of 4,096 gives a smooth, reliable direction, and spreads well over many chips.

> [!DEFINITION] Learning rate, warm-up and decay
> The **learning rate** scales every step. A **warm-up** starts it at 0 and raises it in a straight line over the first steps. The first gradients of a fresh model are wild. **Decay** lowers it again towards the end, so the last steps are small and careful. "Linear" decay is a straight line down; "cosine" follows half a cosine wave.

> [!DEFINITION] Weight decay
> At every step, shrink each weight a little towards zero. A model must then "pay" to keep a weight large, so it keeps only the weights that really help. It is a form of regularization ([Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/)). The paper calls its value of 0.1 "high".

> [!DEFINITION] Adam
> An optimizer (Kingma and Ba, 2015). For every weight it keeps a running average of the gradients (the direction) and of the squared gradients (the size). It divides the first by the square root of the second. Each weight gets its own step size.

**Adam, by hand.** The intuition: a hiker in fog keeps a running sense of which way is downhill ($$m$$), and of how bumpy the ground has been ($$v$$). Steps are long on smooth ground and short on bumpy ground. For one weight $$w$$ with gradient $$g_t$$ at step $$t$$:

$$
m_t = \beta_1 m_{t-1} + (1 - \beta_1)\,g_t, \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2)\,g_t^2
$$

$$
\hat m_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat v_t = \frac{v_t}{1 - \beta_2^t}, \qquad w_t = w_{t-1} - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon}
$$

Here $$m_t$$ is the running average of the gradient with memory $$\beta_1 = 0.9$$, and $$v_t$$ the running average of the squared gradient with memory $$\beta_2 = 0.999$$. $$\hat m_t$$ and $$\hat v_t$$ are the same averages, corrected for starting at 0. $$\eta$$ is the learning rate, and $$\epsilon$$ a tiny number ($$10^{-8}$$) that avoids a division by zero. With three gradients $$0.5, -0.1, 0.3$$ and $$\eta = 0.001$$:

```text
  Adam t=1: g= 0.50  m=0.0500  v=0.000250  m_hat=0.5000  v_hat=0.2500  step = lr*m_hat/sqrt(v_hat) = 0.001000  w=0.999000
  Adam t=2: g=-0.10  m=0.0350  v=0.000260  m_hat=0.1842  v_hat=0.1299  step = lr*m_hat/sqrt(v_hat) = 0.000511  w=0.998489
  Adam t=3: g= 0.30  m=0.0615  v=0.000349  m_hat=0.2269  v_hat=0.1166  step = lr*m_hat/sqrt(v_hat) = 0.000665  w=0.997824
  torch.optim.Adam after the same three gradients: w=0.997824  (ours 0.997824)
```

Check step 1 with a pencil: $$m_1 = 0.1 \times 0.5 = 0.05$$ and $$v_1 = 0.001 \times 0.25 = 0.00025$$. The corrections divide by $$1 - 0.9 = 0.1$$ and $$1 - 0.999 = 0.001$$, giving $$\hat m_1 = 0.5$$ and $$\hat v_1 = 0.25$$. The step is $$0.001 \times 0.5 / \sqrt{0.25} = 0.001$$. The first step of Adam is always exactly $$\eta$$ long, whatever the size of the gradient. That is why a warm-up matters: without it, every weight jumps by the full learning rate at once.

> [!PAPER] · Table 3
> [![Table 3, hyperparameters for training: ViT-B/16 and B/32 on JFT-300M 7 epochs, base LR 8 times 10 to the minus 4, linear decay, weight decay 0.1, dropout 0.0; ViT-L/32 6e-4; ViT-L/16 7 or 14 epochs, 4e-4; ViT-H/14 14 epochs, 3e-4; ResNets and hybrids similar; ViT-B on ImageNet-21k 90 epochs, 1e-3, weight decay 0.03, dropout 0.1; ViT-L on ImageNet-21k 30 or 90 epochs; ViT on ImageNet 300 epochs, 3e-3, cosine decay, weight decay 0.3, dropout 0.1. Caption: all models are trained with a batch size of 4096 and learning rate warmup of 10k steps. For ImageNet we found it beneficial to additionally apply gradient clipping at global norm 1. Training resolution is 224](https://ishwarj.com/img/papers/vit/p3-table3.pngs/vit/p3-table3.png)](/img/papers/vit/p3-table3.png)
>
> **What it says:** on JFT-300M every model trains for 7 or 14 epochs with linear decay, weight decay 0.1 and no dropout; the base learning rate falls as models grow ($$8 \cdot 10^{-4}$$ for ViT-B, $$3 \cdot 10^{-4}$$ for ViT-H). On ImageNet-21k: 30 or 90 epochs, weight decay 0.03, dropout 0.1. On ImageNet: 300 epochs, cosine decay, weight decay 0.3, dropout 0.1. All use "a batch size of 4096 and learning rate warmup of 10k steps"; ImageNet adds "gradient clipping at global norm 1".
>
> **What it means:**
>
> Read the table from top to bottom and a pattern appears. The smaller the dataset, the more epochs and the more regularization. JFT is so big that 7 passes are enough and nothing needs to be held back. ImageNet is small for ViT, so the model sees it 300 times, and needs dropout and three times the weight decay to avoid memorising it.
>
> The learning rate shrinks as the model grows. The paper gives no reason. A common one: a bigger model has more weights moving at once, so the same step shakes it more. Note also the single line "Training resolution is 224": all pre-training uses small pictures, and the bigger ones come only at fine-tuning.

> [!DEFINITION] Epoch
> One full pass over the training set. Seven epochs of JFT-300M means each of its 303 million pictures is seen about seven times.

Turned into steps, the table reads like this (`vit_part3_setup.py`; the ImageNet training set is 1.28 million pictures):

```text
  pre-training ViT-B/16 on JFT-300M      :   7 epochs x 303.00M images / batch 4096 =   517,822 steps; warm-up 10,000 =  1.9% of them; base LR 0.0008, linear decay
  pre-training ViT-H/14 on JFT-300M      :  14 epochs x 303.00M images / batch 4096 = 1,035,645 steps; warm-up 10,000 =  1.0% of them; base LR 0.0003, linear decay
  pre-training ViT-B/16 on ImageNet-21k  :  90 epochs x  14.00M images / batch 4096 =   307,617 steps; warm-up 10,000 =  3.3% of them; base LR 0.001, linear decay
  pre-training ViT on ImageNet           : 300 epochs x   1.28M images / batch 4096 =    93,750 steps; warm-up 10,000 = 10.7% of them; base LR 0.003, cosine decay
  ViT-B/16 on JFT: LR at step 5,000 = 4.0e-04 (half way up the warm-up), at 10,000 = 8.0e-04, at the middle of the decay = 4.0e-04
```

The 10,000 warm-up steps are under 2% of a JFT run, but almost 11% of an ImageNet run. One rule fits very different runs. The learning rate itself is a ramp up and a long line down:

<figure class="fig"><svg viewBox="0 0 800 270" role="img" aria-label="Three learning-rate curves: pre-training on JFT-300M with a short linear warm-up and a long linear decay, ImageNet training with warm-up and cosine decay, and ImageNet fine-tuning with cosine decay from 0.03 to 0 over 20,000 steps."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Learning rate over training (Tables 3 and 4)</text><text class="t-note" x="70.0" y="52.0" text-anchor="start">ViT-B/16 on JFT-300M</text><line class="grid" x1="70" y1="214.0" x2="260" y2="214.0"/><text class="t-tick" x="64.0" y="218.0" text-anchor="end">0</text><line class="grid" x1="70" y1="139.0" x2="260" y2="139.0"/><text class="t-tick" x="64.0" y="143.0" text-anchor="end">4e-4</text><line class="grid" x1="70" y1="64.0" x2="260" y2="64.0"/><text class="t-tick" x="64.0" y="68.0" text-anchor="end">8e-4</text><line class="axis" x1="70" y1="214" x2="260" y2="214"/><polyline class="l1" points="70.0,214.0 71.6,149.3 73.2,84.6 74.7,64.9 76.3,66.1 77.9,67.4 79.5,68.7 81.1,70.0 82.7,71.2 84.2,72.5 85.8,73.8 87.4,75.1 89.0,76.3 90.6,77.6 92.2,78.9 93.7,80.2 95.3,81.4 96.9,82.7 98.5,84.0 100.1,85.3 101.7,86.5 103.2,87.8 104.8,89.1 106.4,90.4 108.0,91.6 109.6,92.9 111.2,94.2 112.7,95.5 114.3,96.7 115.9,98.0 117.5,99.3 119.1,100.6 120.7,101.8 122.2,103.1 123.8,104.4 125.4,105.7 127.0,106.9 128.6,108.2 130.2,109.5 131.7,110.8 133.3,112.0 134.9,113.3 136.5,114.6 138.1,115.9 139.7,117.1 141.2,118.4 142.8,119.7 144.4,121.0 146.0,122.2 147.6,123.5 149.2,124.8 150.7,126.1 152.3,127.3 153.9,128.6 155.5,129.9 157.1,131.2 158.7,132.4 160.2,133.7 161.8,135.0 163.4,136.2 165.0,137.5 166.6,138.8 168.2,140.1 169.7,141.3 171.3,142.6 172.9,143.9 174.5,145.2 176.1,146.4 177.7,147.7 179.2,149.0 180.8,150.3 182.4,151.5 184.0,152.8 185.6,154.1 187.2,155.4 188.7,156.6 190.3,157.9 191.9,159.2 193.5,160.5 195.1,161.7 196.7,163.0 198.2,164.3 199.8,165.6 201.4,166.8 203.0,168.1 204.6,169.4 206.2,170.7 207.7,171.9 209.3,173.2 210.9,174.5 212.5,175.8 214.1,177.0 215.7,178.3 217.2,179.6 218.8,180.9 220.4,182.1 222.0,183.4 223.6,184.7 225.2,186.0 226.7,187.2 228.3,188.5 229.9,189.8 231.5,191.1 233.1,192.3 234.7,193.6 236.2,194.9 237.8,196.2 239.4,197.4 241.0,198.7 242.6,200.0 244.2,201.3 245.7,202.5 247.3,203.8 248.9,205.1 250.5,206.4 252.1,207.6 253.7,208.9 255.2,210.2 256.8,211.5 258.4,212.7 260.0,214.0"/><text class="t-tick" x="70.0" y="230.0" text-anchor="middle">0k</text><text class="t-tick" x="165.0" y="230.0" text-anchor="middle">259k</text><text class="t-tick" x="260.0" y="230.0" text-anchor="middle">518k</text><text class="t-muted" x="165.0" y="248.0" text-anchor="middle">steps; warm-up 10k, linear decay</text><text class="t-note" x="340.0" y="52.0" text-anchor="start">ViT on ImageNet</text><line class="grid" x1="340" y1="214.0" x2="530" y2="214.0"/><text class="t-tick" x="334.0" y="218.0" text-anchor="end">0</text><line class="grid" x1="340" y1="139.0" x2="530" y2="139.0"/><text class="t-tick" x="334.0" y="143.0" text-anchor="end">1.5e-3</text><line class="grid" x1="340" y1="64.0" x2="530" y2="64.0"/><text class="t-tick" x="334.0" y="68.0" text-anchor="end">3.0e-3</text><line class="axis" x1="340" y1="214" x2="530" y2="214"/><polyline class="l1" points="340.0,214.0 341.6,202.3 343.2,190.6 344.7,178.9 346.3,167.1 347.9,155.4 349.5,143.7 351.1,132.0 352.7,120.2 354.2,108.5 355.8,96.8 357.4,85.1 359.0,73.4 360.6,64.0 362.2,64.0 363.7,64.2 365.3,64.3 366.9,64.6 368.5,64.9 370.1,65.2 371.7,65.7 373.2,66.2 374.8,66.7 376.4,67.3 378.0,68.0 379.6,68.7 381.2,69.5 382.7,70.4 384.3,71.3 385.9,72.3 387.5,73.3 389.1,74.4 390.7,75.6 392.2,76.8 393.8,78.0 395.4,79.3 397.0,80.7 398.6,82.1 400.2,83.5 401.7,85.0 403.3,86.6 404.9,88.2 406.5,89.8 408.1,91.5 409.7,93.2 411.2,95.0 412.8,96.8 414.4,98.6 416.0,100.5 417.6,102.4 419.2,104.3 420.7,106.3 422.3,108.3 423.9,110.3 425.5,112.3 427.1,114.4 428.7,116.5 430.2,118.6 431.8,120.7 433.4,122.9 435.0,125.0 436.6,127.2 438.2,129.4 439.7,131.5 441.3,133.7 442.9,135.9 444.5,138.1 446.1,140.3 447.7,142.5 449.2,144.7 450.8,146.9 452.4,149.1 454.0,151.3 455.6,153.4 457.2,155.6 458.7,157.7 460.3,159.8 461.9,161.9 463.5,164.0 465.1,166.1 466.7,168.1 468.2,170.1 469.8,172.1 471.4,174.1 473.0,176.0 474.6,177.9 476.2,179.7 477.7,181.6 479.3,183.4 480.9,185.1 482.5,186.8 484.1,188.5 485.7,190.1 487.2,191.7 488.8,193.3 490.4,194.8 492.0,196.2 493.6,197.6 495.2,198.9 496.7,200.2 498.3,201.5 499.9,202.7 501.5,203.8 503.1,204.9 504.7,205.9 506.2,206.9 507.8,207.8 509.4,208.6 511.0,209.4 512.6,210.1 514.2,210.8 515.7,211.4 517.3,211.9 518.9,212.4 520.5,212.8 522.1,213.2 523.7,213.5 525.2,213.7 526.8,213.9 528.4,214.0 530.0,214.0"/><text class="t-tick" x="340.0" y="230.0" text-anchor="middle">0k</text><text class="t-tick" x="435.0" y="230.0" text-anchor="middle">47k</text><text class="t-tick" x="530.0" y="230.0" text-anchor="middle">94k</text><text class="t-muted" x="435.0" y="248.0" text-anchor="middle">steps; warm-up 10k, cosine</text><text class="t-note" x="600.0" y="52.0" text-anchor="start">fine-tuning on ImageNet</text><line class="grid" x1="600" y1="214.0" x2="770" y2="214.0"/><text class="t-tick" x="594.0" y="218.0" text-anchor="end">0</text><line class="grid" x1="600" y1="139.0" x2="770" y2="139.0"/><text class="t-tick" x="594.0" y="143.0" text-anchor="end">0.015</text><line class="grid" x1="600" y1="64.0" x2="770" y2="64.0"/><text class="t-tick" x="594.0" y="68.0" text-anchor="end">0.030</text><line class="axis" x1="600" y1="214" x2="770" y2="214"/><polyline class="l1" points="600.0,64.0 601.4,64.0 602.8,64.1 604.2,64.2 605.7,64.4 607.1,64.6 608.5,64.9 609.9,65.3 611.3,65.6 612.8,66.1 614.2,66.6 615.6,67.1 617.0,67.7 618.4,68.3 619.8,69.0 621.2,69.7 622.7,70.5 624.1,71.3 625.5,72.2 626.9,73.1 628.3,74.0 629.8,75.1 631.2,76.1 632.6,77.2 634.0,78.3 635.4,79.5 636.8,80.7 638.2,82.0 639.7,83.3 641.1,84.6 642.5,86.0 643.9,87.4 645.3,88.8 646.8,90.3 648.2,91.8 649.6,93.3 651.0,94.9 652.4,96.5 653.8,98.1 655.2,99.8 656.7,101.5 658.1,103.2 659.5,105.0 660.9,106.7 662.3,108.5 663.8,110.3 665.2,112.1 666.6,114.0 668.0,115.8 669.4,117.7 670.8,119.6 672.2,121.5 673.7,123.4 675.1,125.3 676.5,127.3 677.9,129.2 679.3,131.2 680.8,133.1 682.2,135.1 683.6,137.0 685.0,139.0 686.4,141.0 687.8,142.9 689.2,144.9 690.7,146.8 692.1,148.8 693.5,150.7 694.9,152.7 696.3,154.6 697.8,156.5 699.2,158.4 700.6,160.3 702.0,162.2 703.4,164.0 704.8,165.9 706.2,167.7 707.7,169.5 709.1,171.3 710.5,173.0 711.9,174.8 713.3,176.5 714.8,178.2 716.2,179.8 717.6,181.5 719.0,183.1 720.4,184.7 721.8,186.2 723.2,187.7 724.7,189.2 726.1,190.6 727.5,192.0 728.9,193.4 730.3,194.7 731.8,196.0 733.2,197.3 734.6,198.5 736.0,199.7 737.4,200.8 738.8,201.9 740.2,202.9 741.7,203.9 743.1,204.9 744.5,205.8 745.9,206.7 747.3,207.5 748.8,208.3 750.2,209.0 751.6,209.7 753.0,210.3 754.4,210.9 755.8,211.4 757.2,211.9 758.7,212.4 760.1,212.7 761.5,213.1 762.9,213.4 764.3,213.6 765.8,213.8 767.2,213.9 768.6,214.0 770.0,214.0"/><text class="t-tick" x="600.0" y="230.0" text-anchor="middle">0k</text><text class="t-tick" x="685.0" y="230.0" text-anchor="middle">10k</text><text class="t-tick" x="770.0" y="230.0" text-anchor="middle">20k</text><text class="t-muted" x="685.0" y="248.0" text-anchor="middle">steps; cosine, base 0.03</text></svg><figcaption>The learning-rate schedules of Table 3 and Table 4, computed from their numbers. Left: pre-training ViT-B/16 on JFT-300M, a linear warm-up to 8·10⁻⁴ over 10,000 steps, then a straight line down over the remaining 507,822 steps (we assume it ends at 0; the paper does not say). Middle: ViT on ImageNet, the same warm-up to 3·10⁻³ and a cosine decay over 93,750 steps. Right: fine-tuning on ImageNet, 20,000 steps of cosine decay from a base of 0.03.</figcaption></figure>

> [!PAPER] · Appendix B.1
> [![Appendix B.1, Training: Table 3 summarizes our training setups for our different models. We found strong regularization to be key when training models from scratch on ImageNet. Dropout, when used, is applied after every dense layer except for the qkv-projections and directly after adding positional- to patch embeddings. Hybrid models are trained with the exact setup as their ViT counterparts. Finally, all training is done on resolution 224](https://ishwarj.com/img/papers/vit/p3-b1-training.png/p3-b1-training.png)](/img/papers/vit/p3-b1-training.png)
>
> **What it says:** "strong regularization" is "key when training models from scratch on ImageNet". Dropout is "applied after every dense layer except for the the qkv-projections and directly after adding positional- to patch embeddings". Hybrids use "the exact setup as their ViT counterparts", and "all training is done on resolution 224".
>
> **What it means:**
>
> "From scratch on ImageNet" is the case where ViT struggles. With only 1.3 million pictures and few built-in assumptions, it can memorise the training set. Dropout and weight decay make memorising harder.
>
> The exact placement of dropout is given because it matters for reproduction. Every dense layer gets it, except the one that makes queries, keys and values. The input sequence also gets it, right after the position embeddings are added. The released checkpoints set dropout to 0.0 in their configuration, which is what fine-tuning and inference use.

> [!DEFINITION] Dropout
> During training, set a random share of the numbers in a layer's output to zero (10% for a rate of 0.1). Scale the rest up to keep the average. The network cannot rely on any single number, so it learns more robust patterns. At test time dropout is off.

## Fine-tuning: SGD with momentum, cosine decay, clipping {§4.1}

> [!PAPER] · §4.1, Training and Fine-tuning
> [![For fine-tuning we use SGD with momentum, batch size 512, for all models, see Appendix B.1.1. For ImageNet results in Table 2, we fine-tuned at higher resolution: 512 for ViT-L/16 and 518 for ViT-H/14, and also used Polyak and Juditsky (1992) averaging with a factor of 0.9999](https://ishwarj.com/img/papers/vit/p3-finetune-recipe.pngfinetune-recipe.png)](/img/papers/vit/p3-finetune-recipe.png)
>
> **What it says:** "For fine-tuning we use SGD with momentum, batch size 512, for all models". For the ImageNet results of Table 2, the models are fine-tuned at "512 for ViT-L/16 and 518 for ViT-H/14", with "Polyak & Juditsky (1992) averaging with a factor of 0.9999".
>
> **What it means:**
>
> Pre-training and fine-tuning use different optimizers. Adam is good at finding its way from a random start. Fine-tuning starts from a good place and only needs small, steady steps, and plain SGD with momentum does that with less memory.
>
> The two odd resolutions follow from the patch sizes. 512 is $$32 \times 16$$, so ViT-L/16 gets a 32 × 32 grid of 1,024 patches. 518 is $$37 \times 14$$, so ViT-H/14 gets 37 × 37 = 1,369 patches. Both are chosen to divide evenly by the patch. Polyak averaging, defined below, is the last polish for the headline numbers.

> [!DEFINITION] SGD with momentum
> Stochastic gradient descent: step against the gradient of the current batch. **Momentum** keeps a running "velocity" that adds up past gradients, like a ball rolling downhill. It smooths out the noise of single batches and speeds up travel along steady slopes.

> [!DEFINITION] Polyak averaging
> Keeping a running average of the weights during training and using the average, not the last weights, at the end. The last weights jitter from step to step; their average sits in the middle of the jitter. A factor of 0.9999 means each step moves the average by 0.0001 of the way to the current weights.

**SGD with momentum and Polyak averaging, by hand.** For one weight with velocity $$u$$, momentum $$\mu = 0.9$$ and learning rate $$\eta$$:

$$
u_t = \mu\,u_{t-1} + g_t, \qquad w_t = w_{t-1} - \eta\,u_t, \qquad \bar w_t = \beta\,\bar w_{t-1} + (1 - \beta)\,w_t
$$

where $$u_t$$ is the velocity, a running sum of gradients that forgets 10% per step, and $$\bar w_t$$ is the Polyak average with factor $$\beta$$. The same three gradients as before, with $$\eta = 0.01$$, and an average with $$\beta = 0.9$$ of five weight values:

```text
  SGD momentum 0.9, lr 0.01, t=1: g= 0.50  velocity = 0.9*velocity + g = 0.5000  w = w - lr*velocity = 0.99500
  SGD momentum 0.9, lr 0.01, t=2: g=-0.10  velocity = 0.9*velocity + g = 0.3500  w = w - lr*velocity = 0.99150
  SGD momentum 0.9, lr 0.01, t=3: g= 0.30  velocity = 0.9*velocity + g = 0.6150  w = w - lr*velocity = 0.98535
  torch.optim.SGD(momentum=0.9) after the same gradients: w=0.98535  (ours 0.98535)
  Polyak (exponential) averaging, factor 0.9: weights [1.0, 1.2, 0.9, 1.1, 1.05] -> averages [1.0, 1.02, 1.008, 1.0172, 1.0205]
  a factor of 0.9999 averages over roughly the last 1/(1 - 0.9999) = 10,000 steps
```

At step 2 the gradient is $$-0.1$$, uphill, but the velocity is still $$0.9 \times 0.5 - 0.1 = 0.35$$, so the weight keeps moving down. That is momentum: one contrary batch does not reverse the direction. The Polyak average of the jittering weights $$1.00, 1.20, 0.90, 1.10, 1.05$$ moves only from 1.0 to 1.0205. A factor of 0.9999 averages over roughly the last 10,000 steps, half of an ImageNet fine-tuning run of 20,000 steps.

> [!PAPER] · Table 4
> [![Table 4, hyperparameters for fine-tuning: ImageNet 20 000 steps, base LR in 0.003, 0.01, 0.03, 0.06; CIFAR100 and CIFAR10 10 000 steps, base LR in 0.001, 0.003, 0.01, 0.03; Oxford-IIIT Pets and Oxford Flowers-102 500 steps, same range; VTAB (19 tasks) 2 500 steps, 0.01. Caption: all models are fine-tuned with cosine learning rate decay, a batch size of 512, no weight decay, and grad clipping at global norm 1. If not mentioned otherwise, fine-tuning resolution is 384](https://ishwarj.com/img/papers/vit/p3-table4.pngs/vit/p3-table4.png)](/img/papers/vit/p3-table4.png)
>
> **What it says:** ImageNet is fine-tuned for 20,000 steps, CIFAR for 10,000, Pets and Flowers for 500, each VTAB task for 2,500 at a learning rate of 0.01. All runs use "cosine learning rate decay, a batch size of 512, no weight decay, and grad clipping at global norm 1". The usual "fine-tuning resolution is 384".
>
> **What it means:**
>
> Fine-tuning is short. Pre-training ViT-B/16 on JFT took about 518,000 steps of 4,096 pictures; fine-tuning on Pets takes 500 steps of 512. The body already knows how to see. Fine-tuning only points it at a new question.
>
> "No weight decay" is a deliberate change from pre-training. Weight decay pulls weights towards zero, and in fine-tuning that means pulling them away from the pre-trained values that make the model good. Clipping, the last item, guards against single bad steps.

> [!DEFINITION] Gradient clipping at global norm 1
> Before each step, measure the length of the whole gradient, all weights together, as one long vector. If it is longer than 1, shrink every number by the same factor so the length becomes 1. The direction stays; only an unusually large step is cut down.

**Cosine decay and clipping, in numbers.** With base learning rate $$\eta_0$$ and $$T$$ steps, cosine decay sets step $$t$$ to

$$
\eta_t = \eta_0 \cdot \tfrac{1}{2}\left(1 + \cos\frac{\pi t}{T}\right)
$$

which is $$\eta_0$$ at $$t = 0$$, $$\eta_0/2$$ halfway, and 0 at the end. For ImageNet with $$\eta_0 = 0.03$$ and $$T = 20{,}000$$: 0.03, then 0.015 at step 10,000, 0.0044 at step 15,000, and 0 at step 20,000. Clipping uses the global norm $$\lVert g \rVert = \sqrt{\sum_i g_i^2}$$ over all weights, and multiplies the gradient by $$\min(1, 1/\lVert g \rVert)$$. With two tensors of gradients $$[3, 0]$$ and $$[4]$$:

```text
  clipping at global norm 1: gradients of two tensors [3, 0] and [4]; global norm sqrt(9 + 0 + 16) = 5.0 > 1, so every gradient is multiplied by 1/5: [0.6, 0.0] and [0.8]; direction kept, length 1
  torch.nn.utils.clip_grad_norm_: [0.6, 0.0] and [0.8]
```

The length is $$\sqrt{9 + 0 + 16} = 5$$, so every number is divided by 5: $$[0.6, 0]$$ and $$[0.8]$$, a vector of length 1 that points the same way. Table 4 in steps and passes over the data:

```text
  fine-tuning ImageNet           : 20,000 steps x batch 512 =  10.24M pictures seen =     8.0 passes over 1,280,000 training pictures
  fine-tuning CIFAR-100          : 10,000 steps x batch 512 =   5.12M pictures seen =   102.4 passes over 50,000 training pictures
  fine-tuning CIFAR-10           : 10,000 steps x batch 512 =   5.12M pictures seen =   102.4 passes over 50,000 training pictures
  fine-tuning Oxford-IIIT Pets   :    500 steps x batch 512 =   0.26M pictures seen =    69.6 passes over 3,680 training pictures
  fine-tuning Oxford Flowers-102 :    500 steps x batch 512 =   0.26M pictures seen =   125.5 passes over 2,040 training pictures
  fine-tuning VTAB task          :  2,500 steps x batch 512 =   1.28M pictures seen =  1280.0 passes over 1,000 training pictures
  ImageNet fine-tuning at base LR 0.03, cosine: step 0 -> 0.03, step 10,000 -> 0.015, step 15,000 -> 0.0044, step 20,000 -> 0
```

These counts use the standard training-set sizes of each benchmark. Pets is seen about 70 times, and each 1,000-example VTAB task 1,280 times. Small tasks get many passes but few steps.

> [!PAPER] · Appendix B.1.1
> [![Appendix B.1.1, Fine-tuning: we fine-tune all ViT models using SGD with a momentum of 0.9. We run a small grid search over learning rates, see learning rate ranges in Table 4. To do so, we use small sub-splits from the training set (10% for Pets and Flowers, 2% for CIFAR, 1% ImageNet) as development set and train on the remaining data. For final results we train on the entire training set and evaluate on the respective test data. For fine-tuning ResNets and hybrid models we use the exact same setup, with the only exception of ImageNet where we add another value 0.06 to the learning rate sweep](https://ishwarj.com/img/papers/vit/p3-b11-sweep.pngit/p3-b11-sweep.png)](/img/papers/vit/p3-b11-sweep.png)
>
> **What it says:** "SGD with a momentum of 0.9", and "a small grid search over learning rates". To choose, they hold out "10% for Pets and Flowers, 2% for CIFAR, 1% ImageNet" of the training set as a development set. "For final results we train on the entire training set". ResNets and hybrids get the same setup, plus a learning rate of 0.06 on ImageNet.
>
> **What it means:**
>
> Four learning rates are tried for each task, and the winner is chosen on held-out training pictures, never on the test set. Picking on the test set would be like a student who sees the exam before choosing how to study.
>
> After the choice, the held-out pictures go back in, and the final model trains on everything. The extra 0.06 for ResNets on ImageNet is a small gift to the baseline. If anything, it tilts the race against ViT.

> [!DEFINITION] Development set and grid search
> A **development set** (or validation set) is a slice of the training data kept aside to choose settings. A **grid search** tries every value in a short list, here four learning rates, and keeps the one that does best on the development set.

> [!PAPER] · Appendix B.1.1
> [![Appendix B.1.1 continued: for ResNets we also run the setup of Kolesnikov et al. (2020) and select the best results across this run and our sweep. Finally, if not mentioned otherwise, all fine-tuning experiments run at 384 resolution (running fine-tuning at different resolution than training is common practice (Kolesnikov et al., 2020))](https://ishwarj.com/img/papers/vit/p3-b11-res.png/vit/p3-b11-res.png)](/img/papers/vit/p3-b11-res.png)
>
> **What it says:** ResNets get BiT's own fine-tuning setup as well, and "the best results across this run and our sweep" count. "All fine-tuning experiments run at 384 resolution", which "is common practice".
>
> **What it means:**
>
> The baseline gets two chances: the paper's recipe and its own original recipe, whichever is better. This is how a careful paper protects itself from the charge "you tuned your model and not the opponent".
>
> The 384 rule ties back to Section 3.2. Unless the paper says otherwise, every ViT fine-tuning number comes from a model whose position table was stretched first. For a /16 model, that is 14 × 14 to 24 × 24.

> [!PAPER] · Appendix B.1.1
> [![Appendix B.1.1, VTAB: for VTAB we follow the protocol in Kolesnikov et al. (2020), and use the same hyperparameter setting for all tasks. We use a learning rate of 0.01 and train for 2500 steps. We chose this setting by running a small sweep over two learning rates and two schedules, and selecting the setting with the highest VTAB score on the 200-example validation sets. We follow the pre-processing used in Kolesnikov et al. (2020), except that we do not use task-specific input resolutions. Instead we find that Vision Transformer benefits most from a high resolution (384 by 384) for all tasks](https://ishwarj.com/img/papers/vit/p3-b11-vtab.pngvit/p3-b11-vtab.png)](/img/papers/vit/p3-b11-vtab.png)
>
> **What it says:** VTAB uses "the same hyperparameter setting for all tasks": "a learning rate of 0.01 and train for 2500 steps", chosen on "200-example validation sets". Unlike BiT, no task-specific picture sizes: ViT "benefits most from a high resolution (384 × 384) for all tasks".
>
> **What it means:**
>
> On VTAB, tuning each of 19 tasks separately would be both costly and unfair, since each task has only 1,000 pictures. So one setting serves all 19, chosen once on small validation slices of 200 pictures.
>
> The last sentence is a finding, not only a setting. BiT picks the picture size task by task. ViT does best at 384 on every task, so it needs no such rule.

### Our fine-tuning run

The paper's fine-tuning runs are far beyond one computer: 20,000 steps of 512 pictures at 384 pixels. We kept the pieces of the recipe and shrank the scale. We start from the ImageNet-21k checkpoint of ViT-B/16 with the zero head shown above. We fine-tune it on **Imagenette**, a public set of 10 easy ImageNet classes. They are tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball and parachute. It has 9,469 training and 3,925 validation pictures, stored at 160 pixels on the short side.

`vit_part3_finetune.py` runs 300 steps of 32 pictures at 224 pixels, so the model sees 9,600 pictures, about one pass over the training set. The pieces follow Table 4: SGD with momentum 0.9, no weight decay, cosine decay and clipping at global norm 1. We also keep a running average of the weights.

Three choices are ours. The batch is 32 instead of 512. There is a short linear warm-up of 30 steps, which Table 4 does not list. And the learning rate is 0.003, a value from Table 4's grid, picked without a search.

```python
opt = torch.optim.SGD(model.parameters(), lr=LR, momentum=0.9, weight_decay=0.0)
for step in range(1, STEPS + 1):
    for g in opt.param_groups:
        g['lr'] = lr_at(step - 1)                               # warm-up, then cosine to 0
    loss = F.cross_entropy(model(pixel_values=x).logits, y)
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)    # global norm 1
    opt.step()
    for pe, pm in zip(ema.parameters(), model.parameters()):   # Polyak average, factor 0.99
        pe.mul_(0.99).add_(pm.detach(), alpha=0.01)
```

```text
== 2. fine-tuning: 300 steps, batch 32, SGD momentum 0.9, no weight decay, base LR 0.003, linear warm-up 30 steps, cosine decay to 0, clip at global norm 1.0, weight average factor 0.99, random horizontal flips ==
  step   0: validation accuracy (1,000 pictures, 100 per class) 10.0%  (zero head: every class ties, the first wins)
  step   5: lr 0.00040  loss 2.3014  gradient norm before clipping   1.151  validation  98.3%   [39 s]
  step  10: lr 0.00090  loss 2.2914  gradient norm before clipping   1.197  validation  99.0%   [76 s]
  step  25: lr 0.00240  loss 2.1605  gradient norm before clipping   1.123  validation  99.0%   [137 s]
  step  50: lr 0.00296  loss 1.6761  gradient norm before clipping   1.280  validation  99.1%   [220 s]
  step 100: lr 0.00254  loss 0.9975  gradient norm before clipping   1.655  validation  99.4%   [359 s]
  step 150: lr 0.00178  loss 0.6214  gradient norm before clipping   1.523  validation  99.6%   [494 s]
  step 200: lr 0.00092  loss 0.4379  gradient norm before clipping   1.217  validation  99.7%   [599 s]
  step 250: lr 0.00026  loss 0.4833  gradient norm before clipping   1.825  validation  99.7%   [718 s]
  step 300: lr 0.00000  loss 0.3019  gradient norm before clipping   1.104  validation  99.7%   [850 s]
  clipping was active in 292 of 300 steps; median gradient norm: first 50 steps 1.197, last 50 steps 1.393
  mean training loss: steps 1 to 10 2.299, steps 291 to 300 0.350; pictures seen 9,600 = 1.01 passes over the training set
```

Read the loss and the accuracy side by side. After only 5 steps, validation accuracy jumps from 10.0% to 98.3%, while the loss has barely moved from $$\ln 10$$ (2.3014). Accuracy only asks whether the right class has the highest score. The loss asks how sure the model is. After 5 small steps the head already points the right way, but its scores are still small, since the loss is still close to $$\ln 10$$. The other 295 steps mostly make the model confident: the mean loss of the last 10 steps is 0.350.

<figure class="fig"><svg viewBox="0 0 800 310" role="img" aria-label="Left: the training loss of each step of our fine-tuning run, falling from ln 10 towards zero, with the learning-rate schedule dashed. Right: validation accuracy on 1,000 Imagenette pictures at a few steps, rising from 10 percent to near 100 percent."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Fine-tuning ViT-B/16 on Imagenette: 300 steps of 32 pictures</text><text class="t-note" x="70.0" y="46.0" text-anchor="start">training loss per step (and learning rate, dashed)</text><line class="grid" x1="70" y1="256.0" x2="370" y2="256.0"/><text class="t-tick" x="64.0" y="260.0" text-anchor="end">0.0</text><line class="grid" x1="70" y1="156.0" x2="370" y2="156.0"/><text class="t-tick" x="64.0" y="160.0" text-anchor="end">1.2</text><line class="grid" x1="70" y1="56.0" x2="370" y2="56.0"/><text class="t-tick" x="64.0" y="60.0" text-anchor="end">2.3</text><polyline class="l1" points="71.0,56.0 72.0,56.0 73.0,56.0 74.0,56.0 75.0,56.1 76.0,56.2 77.0,56.3 78.0,56.5 79.0,56.7 80.0,57.0 81.0,57.3 82.0,57.8 83.0,58.2 84.0,58.7 85.0,59.5 86.0,59.6 87.0,60.4 88.0,61.9 89.0,62.5 90.0,63.3 91.0,64.4 92.0,65.3 93.0,66.6 94.0,66.9 95.0,68.3 96.0,70.8 97.0,71.1 98.0,74.0 99.0,74.5 100.0,75.9 101.0,77.4 102.0,78.2 103.0,81.2 104.0,82.6 105.0,80.9 106.0,85.9 107.0,86.4 108.0,91.6 109.0,89.6 110.0,94.3 111.0,93.5 112.0,97.0 113.0,95.3 114.0,96.3 115.0,103.4 116.0,100.2 117.0,101.3 118.0,101.9 119.0,102.3 120.0,110.4 121.0,109.9 122.0,108.3 123.0,110.4 124.0,114.1 125.0,112.4 126.0,120.9 127.0,119.2 128.0,124.6 129.0,125.8 130.0,126.8 131.0,124.3 132.0,125.6 133.0,124.7 134.0,128.6 135.0,128.9 136.0,127.2 137.0,135.9 138.0,135.9 139.0,143.4 140.0,143.4 141.0,140.5 142.0,140.6 143.0,153.9 144.0,138.3 145.0,147.3 146.0,139.2 147.0,143.9 148.0,146.2 149.0,151.3 150.0,149.5 151.0,147.7 152.0,152.1 153.0,152.4 154.0,149.3 155.0,163.3 156.0,160.4 157.0,155.8 158.0,162.5 159.0,166.5 160.0,160.5 161.0,164.6 162.0,164.6 163.0,162.2 164.0,169.6 165.0,160.9 166.0,170.1 167.0,172.4 168.0,180.3 169.0,161.4 170.0,169.4 171.0,168.4 172.0,180.9 173.0,176.2 174.0,175.5 175.0,172.6 176.0,172.8 177.0,177.3 178.0,180.8 179.0,178.2 180.0,187.1 181.0,181.4 182.0,179.3 183.0,190.7 184.0,184.0 185.0,178.6 186.0,181.6 187.0,191.6 188.0,188.8 189.0,191.7 190.0,191.7 191.0,193.1 192.0,192.4 193.0,189.4 194.0,195.8 195.0,194.4 196.0,191.3 197.0,189.6 198.0,200.2 199.0,194.6 200.0,197.0 201.0,193.1 202.0,193.0 203.0,199.2 204.0,198.8 205.0,202.7 206.0,193.1 207.0,200.7 208.0,194.5 209.0,212.2 210.0,192.8 211.0,202.4 212.0,201.6 213.0,203.6 214.0,205.4 215.0,193.9 216.0,202.6 217.0,207.4 218.0,207.3 219.0,200.4 220.0,202.0 221.0,197.4 222.0,212.3 223.0,207.5 224.0,209.0 225.0,202.6 226.0,209.4 227.0,216.7 228.0,211.6 229.0,203.6 230.0,205.9 231.0,212.0 232.0,206.7 233.0,214.5 234.0,205.5 235.0,201.2 236.0,213.8 237.0,204.7 238.0,206.5 239.0,214.0 240.0,214.0 241.0,218.8 242.0,207.1 243.0,218.2 244.0,212.8 245.0,220.7 246.0,209.3 247.0,214.0 248.0,219.9 249.0,201.3 250.0,218.5 251.0,209.9 252.0,220.9 253.0,215.1 254.0,218.0 255.0,213.2 256.0,213.9 257.0,221.1 258.0,217.2 259.0,218.1 260.0,221.2 261.0,216.2 262.0,215.6 263.0,218.5 264.0,216.9 265.0,221.7 266.0,222.1 267.0,219.4 268.0,223.4 269.0,221.6 270.0,218.0 271.0,212.3 272.0,221.2 273.0,222.9 274.0,222.7 275.0,218.0 276.0,219.8 277.0,221.0 278.0,225.8 279.0,223.5 280.0,223.9 281.0,223.1 282.0,218.7 283.0,220.6 284.0,211.9 285.0,223.5 286.0,226.3 287.0,226.5 288.0,227.5 289.0,216.1 290.0,217.6 291.0,223.4 292.0,228.6 293.0,210.7 294.0,221.2 295.0,216.9 296.0,221.3 297.0,227.1 298.0,220.7 299.0,226.1 300.0,220.2 301.0,221.2 302.0,221.1 303.0,226.6 304.0,220.1 305.0,224.4 306.0,221.8 307.0,218.4 308.0,228.2 309.0,224.2 310.0,214.7 311.0,224.4 312.0,225.8 313.0,223.0 314.0,231.2 315.0,225.8 316.0,211.7 317.0,228.3 318.0,226.6 319.0,212.6 320.0,214.0 321.0,214.6 322.0,228.5 323.0,223.5 324.0,226.1 325.0,227.1 326.0,232.2 327.0,226.9 328.0,222.6 329.0,225.2 330.0,223.4 331.0,226.4 332.0,222.2 333.0,221.3 334.0,223.4 335.0,217.4 336.0,216.7 337.0,227.0 338.0,224.4 339.0,224.5 340.0,227.1 341.0,229.1 342.0,227.3 343.0,228.1 344.0,221.0 345.0,215.6 346.0,220.0 347.0,219.9 348.0,224.8 349.0,225.7 350.0,224.8 351.0,230.1 352.0,229.6 353.0,227.7 354.0,218.0 355.0,228.2 356.0,226.2 357.0,227.7 358.0,226.6 359.0,225.1 360.0,227.2 361.0,218.6 362.0,227.1 363.0,225.4 364.0,224.8 365.0,219.6 366.0,228.1 367.0,226.6 368.0,227.4 369.0,228.3 370.0,229.8"/><polyline class="edge-dim" points="71.0,256.0 72.0,249.3 73.0,242.7 74.0,236.0 75.0,229.3 76.0,222.7 77.0,216.0 78.0,209.3 79.0,202.7 80.0,196.0 81.0,189.3 82.0,182.7 83.0,176.0 84.0,169.3 85.0,162.7 86.0,156.0 87.0,149.3 88.0,142.7 89.0,136.0 90.0,129.3 91.0,122.7 92.0,116.0 93.0,109.3 94.0,102.7 95.0,96.0 96.0,89.3 97.0,82.7 98.0,76.0 99.0,69.3 100.0,62.7 101.0,56.0 102.0,56.0 103.0,56.0 104.0,56.1 105.0,56.1 106.0,56.2 107.0,56.2 108.0,56.3 109.0,56.4 110.0,56.5 111.0,56.7 112.0,56.8 113.0,57.0 114.0,57.1 115.0,57.3 116.0,57.5 117.0,57.7 118.0,57.9 119.0,58.2 120.0,58.4 121.0,58.7 122.0,59.0 123.0,59.3 124.0,59.6 125.0,59.9 126.0,60.2 127.0,60.5 128.0,60.9 129.0,61.3 130.0,61.6 131.0,62.0 132.0,62.4 133.0,62.9 134.0,63.3 135.0,63.7 136.0,64.2 137.0,64.6 138.0,65.1 139.0,65.6 140.0,66.1 141.0,66.6 142.0,67.2 143.0,67.7 144.0,68.3 145.0,68.8 146.0,69.4 147.0,70.0 148.0,70.6 149.0,71.2 150.0,71.8 151.0,72.5 152.0,73.1 153.0,73.8 154.0,74.4 155.0,75.1 156.0,75.8 157.0,76.5 158.0,77.2 159.0,77.9 160.0,78.7 161.0,79.4 162.0,80.1 163.0,80.9 164.0,81.7 165.0,82.5 166.0,83.3 167.0,84.1 168.0,84.9 169.0,85.7 170.0,86.5 171.0,87.4 172.0,88.2 173.0,89.1 174.0,90.0 175.0,90.8 176.0,91.7 177.0,92.6 178.0,93.5 179.0,94.4 180.0,95.4 181.0,96.3 182.0,97.2 183.0,98.2 184.0,99.1 185.0,100.1 186.0,101.0 187.0,102.0 188.0,103.0 189.0,104.0 190.0,105.0 191.0,106.0 192.0,107.0 193.0,108.0 194.0,109.1 195.0,110.1 196.0,111.1 197.0,112.2 198.0,113.2 199.0,114.3 200.0,115.3 201.0,116.4 202.0,117.5 203.0,118.5 204.0,119.6 205.0,120.7 206.0,121.8 207.0,122.9 208.0,124.0 209.0,125.1 210.0,126.2 211.0,127.3 212.0,128.4 213.0,129.6 214.0,130.7 215.0,131.8 216.0,132.9 217.0,134.1 218.0,135.2 219.0,136.3 220.0,137.5 221.0,138.6 222.0,139.8 223.0,140.9 224.0,142.1 225.0,143.2 226.0,144.4 227.0,145.5 228.0,146.7 229.0,147.9 230.0,149.0 231.0,150.2 232.0,151.3 233.0,152.5 234.0,153.7 235.0,154.8 236.0,156.0 237.0,157.2 238.0,158.3 239.0,159.5 240.0,160.7 241.0,161.8 242.0,163.0 243.0,164.1 244.0,165.3 245.0,166.5 246.0,167.6 247.0,168.8 248.0,169.9 249.0,171.1 250.0,172.2 251.0,173.4 252.0,174.5 253.0,175.7 254.0,176.8 255.0,177.9 256.0,179.1 257.0,180.2 258.0,181.3 259.0,182.4 260.0,183.6 261.0,184.7 262.0,185.8 263.0,186.9 264.0,188.0 265.0,189.1 266.0,190.2 267.0,191.3 268.0,192.4 269.0,193.5 270.0,194.5 271.0,195.6 272.0,196.7 273.0,197.7 274.0,198.8 275.0,199.8 276.0,200.9 277.0,201.9 278.0,202.9 279.0,204.0 280.0,205.0 281.0,206.0 282.0,207.0 283.0,208.0 284.0,209.0 285.0,210.0 286.0,211.0 287.0,211.9 288.0,212.9 289.0,213.8 290.0,214.8 291.0,215.7 292.0,216.6 293.0,217.6 294.0,218.5 295.0,219.4 296.0,220.3 297.0,221.2 298.0,222.0 299.0,222.9 300.0,223.8 301.0,224.6 302.0,225.5 303.0,226.3 304.0,227.1 305.0,227.9 306.0,228.7 307.0,229.5 308.0,230.3 309.0,231.1 310.0,231.9 311.0,232.6 312.0,233.3 313.0,234.1 314.0,234.8 315.0,235.5 316.0,236.2 317.0,236.9 318.0,237.6 319.0,238.2 320.0,238.9 321.0,239.5 322.0,240.2 323.0,240.8 324.0,241.4 325.0,242.0 326.0,242.6 327.0,243.2 328.0,243.7 329.0,244.3 330.0,244.8 331.0,245.4 332.0,245.9 333.0,246.4 334.0,246.9 335.0,247.4 336.0,247.8 337.0,248.3 338.0,248.7 339.0,249.1 340.0,249.6 341.0,250.0 342.0,250.4 343.0,250.7 344.0,251.1 345.0,251.5 346.0,251.8 347.0,252.1 348.0,252.4 349.0,252.7 350.0,253.0 351.0,253.3 352.0,253.6 353.0,253.8 354.0,254.1 355.0,254.3 356.0,254.5 357.0,254.7 358.0,254.9 359.0,255.0 360.0,255.2 361.0,255.3 362.0,255.5 363.0,255.6 364.0,255.7 365.0,255.8 366.0,255.8 367.0,255.9 368.0,255.9 369.0,256.0 370.0,256.0"/><text class="t-tick" x="70.0" y="272.0" text-anchor="middle">0</text><text class="t-tick" x="220.0" y="272.0" text-anchor="middle">150</text><text class="t-tick" x="370.0" y="272.0" text-anchor="middle">300</text><text class="t-muted" x="220.0" y="290.0" text-anchor="middle">step</text><text class="t-note" x="460.0" y="46.0" text-anchor="start">validation accuracy (1,000 pictures)</text><line class="grid" x1="460" y1="256.0" x2="760" y2="256.0"/><text class="t-tick" x="454.0" y="260.0" text-anchor="end">0%</text><line class="grid" x1="460" y1="156.0" x2="760" y2="156.0"/><text class="t-tick" x="454.0" y="160.0" text-anchor="end">50%</text><line class="grid" x1="460" y1="56.0" x2="760" y2="56.0"/><text class="t-tick" x="454.0" y="60.0" text-anchor="end">100%</text><polyline class="l3" points="460.0,236.0 465.0,59.4 470.0,58.0 485.0,58.0 510.0,57.8 560.0,57.2 610.0,56.8 660.0,56.6 710.0,56.6 760.0,56.6"/><g class="mark"><title>step 0: 10.0%</title><circle class="s3 ring" cx="460.0" cy="236.0" r="4.5"/></g><g class="mark"><title>step 5: 98.3%</title><circle class="s3 ring" cx="465.0" cy="59.4" r="4.5"/></g><g class="mark"><title>step 10: 99.0%</title><circle class="s3 ring" cx="470.0" cy="58.0" r="4.5"/></g><g class="mark"><title>step 25: 99.0%</title><circle class="s3 ring" cx="485.0" cy="58.0" r="4.5"/></g><g class="mark"><title>step 50: 99.1%</title><circle class="s3 ring" cx="510.0" cy="57.8" r="4.5"/></g><g class="mark"><title>step 100: 99.4%</title><circle class="s3 ring" cx="560.0" cy="57.2" r="4.5"/></g><g class="mark"><title>step 150: 99.6%</title><circle class="s3 ring" cx="610.0" cy="56.8" r="4.5"/></g><g class="mark"><title>step 200: 99.7%</title><circle class="s3 ring" cx="660.0" cy="56.6" r="4.5"/></g><g class="mark"><title>step 250: 99.7%</title><circle class="s3 ring" cx="710.0" cy="56.6" r="4.5"/></g><g class="mark"><title>step 300: 99.7%</title><circle class="s3 ring" cx="760.0" cy="56.6" r="4.5"/></g><text class="t-tick" x="475.0" y="81.4" text-anchor="start">98.3% at step 5</text><text class="t-tick" x="470.0" y="240.0" text-anchor="start">10.0% at step 0</text><text class="t-tick" x="756.0" y="46.6" text-anchor="end">99.7%</text><text class="t-tick" x="460.0" y="272.0" text-anchor="middle">0</text><text class="t-tick" x="610.0" y="272.0" text-anchor="middle">150</text><text class="t-tick" x="760.0" y="272.0" text-anchor="middle">300</text><text class="t-muted" x="610.0" y="290.0" text-anchor="middle">step</text></svg><figcaption>Our fine-tuning run of ViT-B/16 (ImageNet-21k weights) on Imagenette, 300 steps of 32 pictures. Left: the training loss of every step, from ln 10 down to about 0.3, with the learning rate dashed (30 warm-up steps, then cosine to 0). Right: accuracy on 1,000 validation pictures, 10.0% at step 0, 98.3% at step 5 and 99.7% at the end.</figcaption></figure>

Two numbers deserve a comment. Clipping was active in 292 of the 300 steps. The median gradient norm was 1.197 over the first 50 steps and 1.393 over the last 50, above the cap of 1. At our batch of 32, the cap of 1 works more like a fixed step length than like a rare guard. A batch of 512 averages more pictures, so its gradient is less noisy and may hit the cap less often; we did not measure that.

The weight average with factor 0.99 ended at 99.52%, against 99.57% for the last weights. On a task this easy, the average has no jitter to remove.

What this run cannot show: Imagenette is tiny and easy. Its ten classes are among the labels of ImageNet-21k, so the body has seen them before. We used one run, one learning rate and 224 pixels instead of 384. The paper's fine-tuning numbers come from much harder tasks, many runs and a learning-rate search. Our run shows the mechanics of the recipe, not its value.

## Metrics: fine-tuning accuracy and few-shot accuracy {§4.1}

> [!PAPER] · §4.1, Metrics
> [![Metrics: we report results on downstream datasets either through few-shot or fine-tuning accuracy. Fine-tuning accuracies capture the performance of each model after fine-tuning it on the respective dataset. Few-shot accuracies are obtained by solving a regularized least-squares regression problem that maps the (frozen) representation of a subset of training images to minus one, one to the K target vectors. This formulation allows us to recover the exact solution in closed form. Though we mainly focus on fine-tuning performance, we sometimes use linear few-shot accuracies for fast on-the-fly evaluation where fine-tuning would be too costly](https://ishwarj.com/img/papers/vit/p3-metrics.png/vit/p3-metrics.png)](/img/papers/vit/p3-metrics.png)
>
> **What it says:** results are "few-shot or fine-tuning accuracy". Few-shot accuracy comes from "a regularized least-squares regression problem that maps the (frozen) representation of a subset of training images to $$\{-1, 1\}^K$$ target vectors", which has an "exact solution in closed form". It is used "for fast on-the-fly evaluation where fine-tuning would be too costly".
>
> **What it means:**
>
> Fine-tuning accuracy is the real exam: adapt the whole model, then test it. It costs a full training run per model and per task. Few-shot accuracy is the quick quiz. Freeze the model, take a handful of labelled pictures per class, and fit one linear layer on their vectors.
>
> The clever part is how the layer is fitted. There are no gradient steps. The paper asks for scores of +1 for the right class and −1 for the others, and solves for the best layer in one formula. No learning rate, no epochs, just one solve of a linear system. That is why the paper can use it "on the fly", for the many models of Figure 4 and the appendix.

> [!DEFINITION] Few-shot
> Learning a task from very few labelled examples. "5-shot" means five labelled pictures per class. With 10 classes, a 5-shot task has 50 training pictures in total.

> [!DEFINITION] Frozen
> Not changed by training. A frozen model is only run forward; its weights stay as they were after pre-training.

> [!DEFINITION] Least squares and regularization (ridge regression)
> Least squares picks the weights that make the squared errors between predictions and targets as small as possible. Adding $$\lambda$$ times the squared size of the weights keeps the weights small ("regularized", also called ridge regression). That matters when there are fewer examples than features.

**The closed form.** The intuition: draw the best straight line through the training points. "Best" means the smallest total squared miss, plus a small fine for steep lines. For the few-shot layer:

$$
W^{*} = \arg\min_{W}\ \lVert XW - Y \rVert^2 + \lambda \lVert W \rVert^2 = (X^\top X + \lambda I)^{-1} X^\top Y
$$

where:

- $$X$$ has one row per training picture: its frozen representation of $$d$$ numbers, plus a 1 at the end for a bias. So it is $$n \times (d + 1)$$;
- $$Y$$ is $$n \times K$$: row $$i$$ holds +1 in the column of picture $$i$$'s class and −1 in the other $$K - 1$$ columns;
- $$W$$ is the linear layer, $$(d + 1) \times K$$; $$XW$$ are the predicted scores;
- $$\lambda \geq 0$$ sets the fine for large weights, and $$I$$ is the identity matrix;
- $$\lVert \cdot \rVert^2$$ is the sum of squares of all entries.

A new picture with representation $$x$$ gets the $$K$$ scores $$xW$$, and the largest score names its class. The formula comes from one line of calculus: set the gradient $$2X^\top(XW - Y) + 2\lambda W$$ to zero, and you get $$(X^\top X + \lambda I)\,W = X^\top Y$$. Solving that linear system is all the "training" there is.

**A tiny example by hand** ([`vit_part3_fewshot.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part3_fewshot.py)). Four pictures have two features each, two of class A and two of class B. We use $$\lambda = 1$$ and no bias column, to keep the sums short.

```text
== 1. the closed form by hand: 4 pictures, 2 features, K = 2 classes, lambda = 1 ==
  X (rows = pictures A1, A2, B1, B2) = [[2.0, 1.0], [1.0, 0.0], [0.0, 2.0], [1.0, 3.0]]
  Y (targets in {-1, 1}: class A = [1, -1], class B = [-1, 1]) = [[1.0, -1.0], [1.0, -1.0], [-1.0, 1.0], [-1.0, 1.0]]
  X^T X = [[6.0, 5.0], [5.0, 14.0]]   + lambda I = [[7.0, 5.0], [5.0, 15.0]]   det = 80
  (X^T X + I)^-1 = [[0.1875, -0.0625], [-0.0625, 0.0875]]
  X^T Y = [[2.0, -2.0], [-4.0, 4.0]]
  W = (X^T X + I)^-1 X^T Y = [[0.625, -0.625], [-0.475, 0.475]]
  new picture x = [1.5, 0.5]: scores x W = [0.7, -0.7] -> class A
  training scores X W = [[0.775, -0.775], [0.625, -0.625], [-0.95, 0.95], [-0.8, 0.8]] (all on the right side of 0)
```

Check it with a pencil. The top-left entry of $$X^\top X$$ is $$2^2 + 1^2 + 0^2 + 1^2 = 6$$, and the off-diagonal entry is $$2 \cdot 1 + 1 \cdot 0 + 0 \cdot 2 + 1 \cdot 3 = 5$$. Adding $$\lambda = 1$$ on the diagonal gives $$\begin{bmatrix} 7 & 5 \\ 5 & 15 \end{bmatrix}$$, with determinant $$7 \cdot 15 - 5 \cdot 5 = 80$$, so its inverse is $$\tfrac{1}{80}\begin{bmatrix} 15 & -5 \\ -5 & 7 \end{bmatrix}$$. The first column of $$X^\top Y$$ is $$(2 + 1 - 0 - 1,\ 1 + 0 - 2 - 3) = (2, -4)$$. So the first column of $$W$$ is

$$
\tfrac{1}{80}\,(15 \cdot 2 - 5 \cdot (-4),\ -5 \cdot 2 + 7 \cdot (-4)) = \tfrac{1}{80}\,(50, -38) = (0.625, -0.475)
$$

The new picture $$(1.5, 0.5)$$ scores $$1.5 \times 0.625 + 0.5 \times (-0.475) = 0.7$$ for class A and $$-0.7$$ for class B, so it is class A. With two classes, the second column is just the first with its sign flipped. With $$K$$ classes, each column is its own "this class against all the others" line.

<figure class="fig"><svg viewBox="0 0 800 330" role="img" aria-label="Four training points in two dimensions, two of class A and two of class B, the line where the two class scores of the ridge solution are equal, and a new point that falls on the class A side."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">The few-shot classifier of the tiny example</text><line class="grid" x1="110.0" y1="50" x2="110.0" y2="290.0"/><text class="t-tick" x="110.0" y="306.0" text-anchor="middle">0</text><line class="grid" x1="80" y1="260.0" x2="320.0" y2="260.0"/><text class="t-tick" x="72.0" y="264.0" text-anchor="end">0</text><line class="grid" x1="170.0" y1="50" x2="170.0" y2="290.0"/><text class="t-tick" x="170.0" y="306.0" text-anchor="middle">1</text><line class="grid" x1="80" y1="200.0" x2="320.0" y2="200.0"/><text class="t-tick" x="72.0" y="204.0" text-anchor="end">1</text><line class="grid" x1="230.0" y1="50" x2="230.0" y2="290.0"/><text class="t-tick" x="230.0" y="306.0" text-anchor="middle">2</text><line class="grid" x1="80" y1="140.0" x2="320.0" y2="140.0"/><text class="t-tick" x="72.0" y="144.0" text-anchor="end">2</text><line class="grid" x1="290.0" y1="50" x2="290.0" y2="290.0"/><text class="t-tick" x="290.0" y="306.0" text-anchor="middle">3</text><line class="grid" x1="80" y1="80.0" x2="320.0" y2="80.0"/><text class="t-tick" x="72.0" y="84.0" text-anchor="end">3</text><line class="key-line" x1="87.2" y1="290.0" x2="269.6" y2="50.0"/><circle class="s1 ring" cx="230.0" cy="200.0" r="8"/><text class="t-tick" x="242.0" y="192.0" text-anchor="start">A1</text><circle class="s1 ring" cx="170.0" cy="260.0" r="8"/><text class="t-tick" x="182.0" y="252.0" text-anchor="start">A2</text><circle class="s3 ring" cx="110.0" cy="140.0" r="8"/><text class="t-tick" x="122.0" y="132.0" text-anchor="start">B1</text><circle class="s3 ring" cx="170.0" cy="80.0" r="8"/><text class="t-tick" x="182.0" y="72.0" text-anchor="start">B2</text><circle class="s4 ring" cx="200.0" cy="230.0" r="7"/><text class="t-tick" x="212.0" y="246.0" text-anchor="start">new</text><text class="t-tick" x="400.0" y="70.0" text-anchor="start">features x = (x₁, x₂); class A (blue), class B (green)</text><text class="t-tick" x="400.0" y="89.0" text-anchor="start">W = (XᵀX + I)⁻¹ XᵀY</text><text class="t-tick" x="400.0" y="108.0" text-anchor="start">  = [[0.6250, -0.6250],</text><text class="t-tick" x="400.0" y="127.0" text-anchor="start">     [-0.4750, 0.4750]]</text><text class="t-tick" x="400.0" y="146.0" text-anchor="start"></text><text class="t-tick" x="400.0" y="165.0" text-anchor="start">score A − score B = 0 on the orange line;</text><text class="t-tick" x="400.0" y="184.0" text-anchor="start">class A below it, class B above it.</text><text class="t-tick" x="400.0" y="203.0" text-anchor="start">new picture (1.5, 0.5): scores 0.700 and -0.700</text><text class="t-tick" x="400.0" y="222.0" text-anchor="start">  → class A</text></svg><figcaption>The tiny example as a picture. Two class-A pictures (blue) and two class-B pictures (green) in a plane of two features. The orange line is where the two class scores of W are equal; the new picture (1.5, 0.5) falls on the class-A side with scores 0.7 and −0.7.</figcaption></figure>

**The real thing.** We froze ViT-B/16 with its ImageNet-21k weights, never trained on Imagenette. We computed the representation of 500 training pictures (a pool of 50 per class) and of all 3,925 validation pictures. Then we drew 1, 5 or 10 pictures per class from the pool, five times each, and solved the formula above:

```python
X = np.hstack([features[idx], np.ones((len(idx), 1))])     # n x 769: 768 features and a bias
Y = -np.ones((len(idx), 10)); Y[np.arange(len(idx)), labels[idx]] = 1
W = np.linalg.solve(X.T @ X + lam * np.eye(769), X.T @ Y)  # the closed form
pred = (np.hstack([val_features, ones]) @ W).argmax(1)
```

```text
== 2. frozen ViT-B/16 (ImageNet-21k weights), device mps: features of 500 training pictures (a pool of 50 per class) and 3925 validation pictures ==
  representation "pre-logits": the tanh layer on the [class] token, 768 numbers;  "class token": the final LayerNorm output (Eq. 4)
  accuracy on the 3,925 validation pictures, mean +- std over 5 random draws of the training pictures
  pre-logits  lambda   0.01:  1-shot  98.0 +-  0.6%   5-shot  99.0 +-  0.2%   10-shot  99.0 +-  0.1%
  pre-logits  lambda      1:  1-shot  98.0 +-  0.6%   5-shot  99.1 +-  0.1%   10-shot  99.0 +-  0.1%
  pre-logits  lambda    100:  1-shot  98.2 +-  0.5%   5-shot  99.1 +-  0.2%   10-shot  99.2 +-  0.1%
  class token lambda   0.01:  1-shot  98.1 +-  0.5%   5-shot  99.0 +-  0.1%   10-shot  99.0 +-  0.0%
  class token lambda      1:  1-shot  98.1 +-  0.5%   5-shot  99.1 +-  0.1%   10-shot  99.0 +-  0.1%
  class token lambda    100:  1-shot  98.3 +-  0.3%   5-shot  99.2 +-  0.1%   10-shot  99.2 +-  0.1%
  chance is 10%; 1-shot means 10 training pictures in total, one per class
  5-shot, draw 0, pre-logits, lambda 1: X is (50, 769) (50 pictures x 768 features + bias), Y is (50, 10), W is (769, 10); one solve of a 769x769 system
  per-class accuracy (5-shot, draw 0): tench 100%, springer 100%, cassette player 99%, chain saw 96%, church 99%, French horn 99%, garbage truck 99%, gas pump 99%, golf ball 99%, parachute 99%
  scores of the first validation picture (a tench): +0.92 -0.96 -1.01 -1.04 -1.02 -0.97 -0.97 -0.94 -0.95 -1.02  -> class tench
```

One picture per class, ten pictures in all, already gives 98.0% on 3,925 new pictures. Five per class gives 99.1%. The value of $$\lambda$$ hardly matters here (98.0% to 98.3% at 1-shot), and the two choices of representation agree within 0.2 points. The scores of one tench picture show the targets at work: +0.92 for tench and about −1 for every other class.

<figure class="fig"><svg viewBox="0 0 800 366" role="img" aria-label="Horizontal bars of the ten class scores that the 5-shot ridge layer gives one tench picture: plus 0.92 for tench and about minus 1 for each other class."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">The 10 scores x W of one validation picture (a tench), 5-shot, frozen ViT-B/16</text><line class="grid" x1="348.3" y1="44" x2="348.3" y2="310.0"/><text class="t-tick" x="348.3" y="324.0" text-anchor="middle">-1</text><line class="axis" x1="540.0" y1="44" x2="540.0" y2="310.0"/><text class="t-tick" x="540.0" y="324.0" text-anchor="middle">0</text><line class="grid" x1="731.7" y1="44" x2="731.7" y2="310.0"/><text class="t-tick" x="731.7" y="324.0" text-anchor="middle">+1</text><text class="t-tick t-strong" x="230.0" y="66.0" text-anchor="end">tench</text><g class="mark"><title>tench: +0.92</title><rect class="s3" x="540.0" y="54" width="176.5" height="16" rx="3"/></g><text class="t-tick" x="722.5" y="67.0" text-anchor="start">+0.92</text><text class="t-tick" x="230.0" y="92.0" text-anchor="end">springer</text><g class="mark"><title>springer: -0.96</title><rect class="s2" x="355.9" y="80" width="184.1" height="16" rx="3"/></g><text class="t-tick" x="349.9" y="93.0" text-anchor="end">-0.96</text><text class="t-tick" x="230.0" y="118.0" text-anchor="end">cassette player</text><g class="mark"><title>cassette player: -1.01</title><rect class="s2" x="346.9" y="106" width="193.1" height="16" rx="3"/></g><text class="t-tick" x="340.9" y="119.0" text-anchor="end">-1.01</text><text class="t-tick" x="230.0" y="144.0" text-anchor="end">chain saw</text><g class="mark"><title>chain saw: -1.04</title><rect class="s2" x="340.9" y="132" width="199.1" height="16" rx="3"/></g><text class="t-tick" x="334.9" y="145.0" text-anchor="end">-1.04</text><text class="t-tick" x="230.0" y="170.0" text-anchor="end">church</text><g class="mark"><title>church: -1.02</title><rect class="s2" x="345.1" y="158" width="194.9" height="16" rx="3"/></g><text class="t-tick" x="339.1" y="171.0" text-anchor="end">-1.02</text><text class="t-tick" x="230.0" y="196.0" text-anchor="end">French horn</text><g class="mark"><title>French horn: -0.97</title><rect class="s2" x="354.3" y="184" width="185.7" height="16" rx="3"/></g><text class="t-tick" x="348.3" y="197.0" text-anchor="end">-0.97</text><text class="t-tick" x="230.0" y="222.0" text-anchor="end">garbage truck</text><g class="mark"><title>garbage truck: -0.97</title><rect class="s2" x="353.1" y="210" width="186.9" height="16" rx="3"/></g><text class="t-tick" x="347.1" y="223.0" text-anchor="end">-0.97</text><text class="t-tick" x="230.0" y="248.0" text-anchor="end">gas pump</text><g class="mark"><title>gas pump: -0.94</title><rect class="s2" x="360.1" y="236" width="179.9" height="16" rx="3"/></g><text class="t-tick" x="354.1" y="249.0" text-anchor="end">-0.94</text><text class="t-tick" x="230.0" y="274.0" text-anchor="end">golf ball</text><g class="mark"><title>golf ball: -0.95</title><rect class="s2" x="358.0" y="262" width="182.0" height="16" rx="3"/></g><text class="t-tick" x="352.0" y="275.0" text-anchor="end">-0.95</text><text class="t-tick" x="230.0" y="300.0" text-anchor="end">parachute</text><g class="mark"><title>parachute: -1.02</title><rect class="s2" x="343.9" y="288" width="196.1" height="16" rx="3"/></g><text class="t-tick" x="337.9" y="301.0" text-anchor="end">-1.02</text><text class="t-muted" x="20.0" y="340.0" text-anchor="start">The layer was fitted to targets of +1 for the right class and −1 for the others, and it nearly hits them on a new picture:</text><text class="t-muted" x="20.0" y="356.0" text-anchor="start">tench +0.92, every other class close to −1. The largest score names the class.</text></svg><figcaption>The ten scores that the 5-shot layer gives one new tench picture. It was fitted to targets of +1 for the right class and −1 for the others, and on this unseen picture it gives +0.92 for tench and between −0.94 and −1.04 for the other nine classes.</figcaption></figure>

Do not read these numbers as "ViT learns anything from one picture". Imagenette is easy: ten very different objects, and all ten are ImageNet classes, so the ImageNet-21k pre-training already saw labelled pictures of each of them. The test shows something narrower and still useful.

A frozen representation that already separates the classes needs only a linear layer and a handful of examples. The closed form finds that layer in one solve. The paper uses the same metric on harder tasks, where models differ more. [Part 4](https://ishwarj.com/papers/vit/part-4-results/) runs it across model sizes.

Few-shot and fine-tuning measure different things. Fine-tuning asks how good the model can become on a task; few-shot asks how good its frozen vectors already are. The paper reports fine-tuning in its main tables. It uses few-shot where it must test many models, because one solve of a linear system is far cheaper than thousands of training steps.

> [!TAKEAWAYS] Key takeaways
> - Fine-tuning removes the whole pre-training head (the pre-logits layer and the class layer) and attaches one **zero-initialised** $$D \times K$$ layer. At step 0 every score is 0 and the loss is exactly $$\ln 10$$ for 10 classes. The body gets a gradient of exactly 0, so only the head learns at first.
> - ViT is fine-tuned at a **higher resolution** (usually 384) with the same patch size. For ViT-B/16 the sequence grows from 197 to 577 tokens, and the forward compute from about 34.9 to 110.3 billion operations.
> - The 196 learned position rows are **stretched in 2D** from 14 × 14 to 24 × 24 (the official code: linear interpolation; the library: bicubic). On Imagenette at 384 with no training, the 2D tables kept 84.76% and 84.97%, a 1D stretch fell to 48.38%, and no positions to 13.99%.
> - Patch cutting and this stretch are the **only places** where the paper hands ViT the 2D structure of pictures.
> - Pre-training data: ImageNet (1.3M pictures), ImageNet-21k (14M), JFT-300M (303M), each de-duplicated against the test sets. Downstream: ImageNet and ReaL, CIFAR-10/100, Pets, Flowers-102 and the 19 VTAB tasks.
> - **ViT-L/16** means the Large size with 16 × 16 patches. Halving the patch side gives 4 times the tokens: ViT-B/16 costs 4.06 times ViT-B/32 while having 1.9% fewer parameters.
> - The CNN baseline is **ResNet (BiT)**: Group Norm and Weight Standardization instead of Batch Norm, so a picture's output never depends on its batch. Noisy Student, a semi-supervised EfficientNet, held the ImageNet state of the art.
> - Pre-training: **Adam** (β₁ = 0.9, β₂ = 0.999), batch 4,096, weight decay 0.1 on JFT, 10,000 warm-up steps, linear decay, all at 224 pixels. Fine-tuning: **SGD with momentum** 0.9, batch 512, cosine decay, no weight decay, clipping at global norm 1, a learning rate picked on a held-out slice.
> - **Few-shot accuracy** fits one linear layer to targets in $$\{-1, 1\}^K$$ with the closed form $$W = (X^\top X + \lambda I)^{-1} X^\top Y$$. On frozen ViT-B/16 features, one Imagenette picture per class gave 98.0%, an easy task that the pre-training already knew.

**Next, in Part 4:** the results. Table 2 pits ViT-H/14 and ViT-L/16 against BiT-L and Noisy Student, and Figure 2 breaks down VTAB by group. Figures 3 and 4 answer the central question: how much data does ViT need to beat a ResNet? Continue to [Part 4](https://ishwarj.com/papers/vit/part-4-results/).

<details>
<summary>Run it yourself</summary>

Every number in this part comes from four scripts in [`code/papers/vit/`](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/vit). `vit_part3_setup.py` does the arithmetic of Tables 1, 3 and 4, the optimizer steps and the normalisation example. It also runs ResNet-50 once (about a minute on a CPU). `vit_part3_posemb.py` stretches the position table and scores five versions of it on Imagenette. `vit_part3_fewshot.py` solves the few-shot problem on frozen features.

`vit_part3_finetune.py` fine-tunes ViT-B/16 for 300 steps. The last three use the Imagenette set (imagenette2-160, from the fast.ai datasets) and a GPU if there is one.

```bash
pip install torch transformers safetensors scipy pillow
python vit_part3_setup.py
python vit_part3_posemb.py
python vit_part3_fewshot.py
python vit_part3_finetune.py
```

<figure><img src="/img/papers/vit/part3_setup-run.png" alt="Terminal output of vit_part3_setup.py: sequence lengths and compute for each ViT variant, parameter counts from the weight files, the ResNet-50 stage shapes, the Batch Norm and Group Norm example, the schedules of Tables 3 and 4, and Adam, SGD, clipping and Polyak averaging by hand" loading="lazy" /><figcaption>The real output of vit_part3_setup.py (first 60 lines).</figcaption></figure>

<figure><img src="/img/papers/vit/part3_posemb-run.png" alt="Terminal output of vit_part3_posemb.py: the 2 by 2 to 3 by 3 example, the 14 by 14 to 24 by 24 stretch checked against SciPy and the library, the similarity numbers and the accuracy of five position tables on Imagenette" loading="lazy" /><figcaption>The real output of vit_part3_posemb.py.</figcaption></figure>

<figure><img src="/img/papers/vit/part3_fewshot-run.png" alt="Terminal output of vit_part3_fewshot.py: the closed-form example by hand, and the 1, 5 and 10-shot accuracies of frozen ViT-B/16 for three values of lambda and two representations" loading="lazy" /><figcaption>The real output of vit_part3_fewshot.py.</figcaption></figure>

<figure><img src="/img/papers/vit/part3_finetune-run.png" alt="Terminal output of vit_part3_finetune.py: the zero head at step 0 with the loss ln 10 and no gradient in the body, the random head for comparison, and the 300-step fine-tuning run with loss, learning rate, gradient norm and validation accuracy" loading="lazy" /><figcaption>The real output of vit_part3_finetune.py.</figcaption></figure>

</details>

## The paper and the code

[The paper (arXiv 2010.11929)](https://arxiv.org/abs/2010.11929) · [Official code and checkpoints](https://github.com/google-research/vision_transformer) · [Our scripts for this part](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/vit)

