# The Big Idea: An Image Is Worth 16×16 Words

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/vit/part-1-the-big-idea/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vit/part-1-the-big-idea.md


On 22 October 2020, twelve researchers at Google posted a paper with an odd title: *An Image is Worth 16×16 Words*. It took the Transformer, the network design that had taken over language processing, cut photographs into small squares, and fed the squares to the Transformer as if they were words. On the standard photo benchmark, ImageNet, the model matched or beat the best convolutional networks of the day, once it had been trained on enough pictures. The paper was accepted at ICLR 2021, and the model it introduced, the Vision Transformer (ViT), is today the base of most image models and of the image half of models that read both text and pictures.

This series reads the paper slowly, in the paper's own order. Each piece shows the exact lines of the paper as a highlighted screenshot in a teal box, then explains what they mean. Pictures and real code, with its real output, help where they can.

The small `Paper §1` tag above each heading tells you which section of the paper you are reading. The screenshots come from the paper's second arXiv version (3 June 2021), the one most people read today. We run every experiment on the model weights Google released, through the Hugging Face `transformers` library, on one ordinary computer, an Apple M5 Pro (64 GB).

> [!TIP] Try it
> [Vision Transformer, step by step](https://ishwarj.com/papers/vit/demo): pick a picture or your own photo, and watch a real ViT turn it into a class, one step at a time.

> [!DEFINITION] Definition boxes
> A yellow box like this explains one technical word, the first time it appears. If you already know the word, skip the box.

## The title {§Title}

> [!PAPER] · Title
> [![The title of the ViT paper: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, with 16x16 Words and Image Recognition at Scale highlighted](https://ishwarj.com/img/papers/vit/p1-title.pngrs/vit/p1-title.png)](/img/papers/vit/p1-title.png)
>
> **What it says:** "An Image is Worth **16×16 Words**: Transformers for **Image Recognition at Scale**."
>
> **What it means:**
>
> The title plays on the old saying "a picture is worth a thousand words". The joke is the method. The paper turns a picture into a sequence of words, and each "word" is a square of 16×16 pixels.
>
> Every page of the paper carries the line "Published as a conference paper at ICLR 2021". ICLR is one of the three big machine-learning conferences, so the paper passed peer review there.

Let us take the title apart, one piece at a time.

- **An image.** A digital picture. For this paper, a photograph of one main thing: a cat, a car, a bird.
- **16×16 words.** The picture is cut into squares of 16 pixels by 16 pixels. Each square is treated as one "word". A 224×224 picture gives 14×14 = 196 such squares. The paper calls them **patches**.
- **Transformers.** The kind of neural network used. It was introduced in 2017 for translating sentences, and by 2020 it was the standard network for text. The paper uses it with almost no changes.
- **Image recognition.** The task: look at a picture and say what is in it. The paper's version of the task is **image classification**: choose one label out of a fixed list (for ImageNet, one of 1,000 labels).
- **At scale.** The key words. The method only works well when the model is trained on very many pictures: 14 million to 300 million. On the "ordinary" 1.3 million pictures of ImageNet, it loses to convolutional networks. This is the paper's main finding, and most of the paper is about it.

> [!DEFINITION] Image recognition and image classification
> **Image recognition** is the general job of understanding what a picture shows. **Image classification** is the simplest version: the model gets a picture and must pick one label from a fixed list ("tabby cat", "sports car", "goldfinch"). It does not say *where* the thing is, only *what* it is.

> [!DEFINITION] Pixel
> One dot of a digital picture. A 224×224 picture has 224 rows of 224 dots, 50,176 in all. Each dot stores a colour as numbers.

> [!DEFINITION] Channel (RGB)
> A colour pixel is stored as three numbers: how much red, how much green and how much blue. Each of the three is a **channel**. So a 224×224 colour picture is really 3 × 224 × 224 = 150,528 numbers.

> [!DEFINITION] Patch
> A small square cut out of the picture, here 16 pixels by 16 pixels by 3 channels, that is 768 numbers. ViT treats each patch as one token.

> [!DEFINITION] Token
> One unit of input to a Transformer. In language, a token is a word or a piece of a word. In ViT, a token is a patch. The model never sees words or pictures directly: it sees a list of tokens, each turned into a list of numbers.

> [!DEFINITION] Transformer
> A neural network design built around **attention**: every token computes how much it should "look at" every other token, and mixes in information from the tokens it looks at. If attention is new to you, the [attention series](https://ishwarj.com/writings/attention-1-self-attention/) explains it from zero. For this paper you only need the idea: tokens look at other tokens. ViT uses the same **encoder** half of the Transformer as [BERT](https://ishwarj.com/papers/bert/).

<figure class="fig"><svg viewBox="0 0 760 220" role="img" aria-label="Timeline: AlexNet 2012 and ResNet 2015 made convolutional networks the standard for vision; the Transformer 2017 and BERT 2018 took over language; ViT 2020 applies the language model design to images."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="axis" x1="50.0" y1="96.0" x2="730.0" y2="96.0" marker-end="url(#ah)"/><circle class="s1" cx="90" cy="96" r="8"/><text class="t-title" x="90.0" y="40.0" text-anchor="middle">2012</text><text class="t-note" x="90.0" y="64.0" text-anchor="middle">AlexNet</text><text class="t-muted" x="90.0" y="124.0" text-anchor="middle">deep CNN wins the</text><text class="t-muted" x="90.0" y="141.0" text-anchor="middle">ImageNet contest</text><circle class="s1" cx="240" cy="96" r="8"/><text class="t-title" x="240.0" y="40.0" text-anchor="middle">2015</text><text class="t-note" x="240.0" y="64.0" text-anchor="middle">ResNet</text><text class="t-muted" x="240.0" y="124.0" text-anchor="middle">CNNs 100+ layers</text><text class="t-muted" x="240.0" y="141.0" text-anchor="middle">deep; still the</text><text class="t-muted" x="240.0" y="158.0" text-anchor="middle">vision standard</text><circle class="s2" cx="390" cy="96" r="8"/><text class="t-title" x="390.0" y="40.0" text-anchor="middle">2017</text><text class="t-note" x="390.0" y="64.0" text-anchor="middle">Transformer</text><text class="t-muted" x="390.0" y="124.0" text-anchor="middle">attention only;</text><text class="t-muted" x="390.0" y="141.0" text-anchor="middle">built for</text><text class="t-muted" x="390.0" y="158.0" text-anchor="middle">translation</text><circle class="s2" cx="540" cy="96" r="8"/><text class="t-title" x="540.0" y="40.0" text-anchor="middle">2018</text><text class="t-note" x="540.0" y="64.0" text-anchor="middle">BERT</text><text class="t-muted" x="540.0" y="124.0" text-anchor="middle">pre-train, then</text><text class="t-muted" x="540.0" y="141.0" text-anchor="middle">fine-tune: the</text><text class="t-muted" x="540.0" y="158.0" text-anchor="middle">NLP recipe</text><circle class="s3" cx="690" cy="96" r="8"/><text class="t-title" x="690.0" y="40.0" text-anchor="middle">2020</text><text class="t-note" x="690.0" y="64.0" text-anchor="middle">ViT</text><text class="t-muted" x="690.0" y="124.0" text-anchor="middle">the Transformer,</text><text class="t-muted" x="690.0" y="141.0" text-anchor="middle">unchanged, on</text><text class="t-muted" x="690.0" y="158.0" text-anchor="middle">image patches</text><rect class="s1" x="40" y="196" width="10" height="10" rx="2"/><text class="t-tick" x="56.0" y="205.0" text-anchor="start">computer vision</text><rect class="s2" x="190" y="196" width="10" height="10" rx="2"/><text class="t-tick" x="206.0" y="205.0" text-anchor="start">language (NLP)</text><rect class="s3" x="330" y="196" width="10" height="10" rx="2"/><text class="t-tick" x="346.0" y="205.0" text-anchor="start">this paper: the two lines meet</text></svg><figcaption>Two lines of research meet. In vision, AlexNet (2012) and ResNet (2015) made convolutional networks the standard. In language, the Transformer (2017) and BERT (2018) replaced older designs and introduced the pre-train-then-fine-tune recipe. ViT (2020) takes the language design, unchanged, to pictures.</figcaption></figure>

<details>
<summary>History: the five milestones on this timeline (optional reading)</summary>

If you want the back-story behind each dot on the line, read on. If not, skip to the next section; nothing below is needed for the paper.

**2012: AlexNet.** Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton (University of Toronto) entered a deep convolutional network in the ImageNet contest and won by a large margin: a top-5 error of 15.3%, against 26.2% for the runner-up, which used hand-made features. Three choices made it work: the **ReLU** activation, which is simply $$f(x) = \max(0, x)$$ and trains much faster than the smooth curves used before; training on two GPUs; and a trick called dropout to fight overfitting. The result convinced the field that learning features from data beats designing them by hand, and it started the deep-learning era in computer vision.

**2015: ResNet.** Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun (Microsoft Research Asia) found that very deep networks were *harder* to train than shallow ones, even on the training data. Their fix was the **residual connection**: instead of making a block of layers learn a new output $$H(x)$$, make it learn only the change, so the block computes

$$
y = F(x) + x
$$

where $$x$$ is the block's input, $$F(x)$$ is what the layers compute, and the "$$+\,x$$" is a shortcut that passes the input straight through. If a block has nothing useful to add, it can learn $$F(x) \approx 0$$ and do no harm. This let them train 152-layer networks, win ImageNet 2015 with 3.57% top-5 error, and it is why ViT has a "+" after every block too (Part 2, Equations 2 and 3). The ResNets in this paper are direct descendants.

**2017: the Transformer.** Ashish Vaswani and seven colleagues at Google published *Attention Is All You Need*, a network for translating sentences that used no recurrence and no convolution, only **attention**: every word computes how much to look at every other word,

$$
\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
$$

where $$Q$$, $$K$$ and $$V$$ are three versions of the word vectors (queries, keys and values), $$QK^\top$$ scores every pair of words, $$\sqrt{d_k}$$ keeps the scores from growing too large, and softmax turns each row into weights that add up to 1. Because every word can reach every other word in one step, the design trains fast on GPUs and TPUs, and it scales: bigger models kept getting better. ViT uses this encoder with almost no change.

**2018: BERT.** Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova (Google) took the Transformer encoder, hid 15% of the words in billions of sentences, and trained the model to guess the hidden words from both sides (the **masked language model**). The pre-trained model was then **fine-tuned**, with one small new layer, for eleven different language tasks, and set a new best result on all of them. Two things carry over to ViT directly: the recipe (pre-train once on a huge dataset, fine-tune cheaply per task) and the special `[CLS]` token whose output vector stands for the whole input. ViT's `[class]` token is the same idea, and its model sizes (Base and Large) are copied from BERT. The [BERT breakdown](https://ishwarj.com/papers/bert/) on this site reads that paper in full.

**2020: ViT.** Twelve researchers at Google Research (Brain Team), led by Alexey Dosovitskiy and Neil Houlsby, asked what happens if the two lines meet: cut a picture into 16×16 patches, treat each patch as a word, and feed them to the BERT-style encoder. The answer, as the rest of this part explains, is that it loses on 1.3 million pictures and wins on 300 million. That finding, "large scale training trumps inductive bias", is the paper's thesis, and it made the Transformer the common design for text and images alike.

</details>

The title's analogy is worth drawing out, because the whole paper rests on it. A language model reads a sentence as a sequence of word tokens, each turned into a vector of numbers. ViT reads a picture as a sequence of patch tokens, each turned into a vector of numbers of the same size. After that step, the Transformer cannot tell whether it is reading a sentence or a photo.

<figure class="fig"><svg viewBox="0 0 760 292" role="img" aria-label="The analogy of the title: a sentence becomes a sequence of word tokens and then vectors; a picture cut into patches becomes a sequence of patch tokens and then vectors, read by the same Transformer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">Language: words → tokens → vectors</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">Vision: patches → tokens → vectors</text><line class="edge-dim" x1="380.0" y1="14.0" x2="380.0" y2="300.0"/><rect class="box" x="30.0" y="44.0" width="320.0" height="28.0" rx="6"/><text class="t-tick" x="190.0" y="63.0" text-anchor="middle">"The cat sat on the mat"</text><line class="edge" x1="190.0" y1="76.0" x2="190.0" y2="100.0" marker-end="url(#ah)"/><rect class="box" x="31.0" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="55.0" y="123.5" text-anchor="middle">The</text><rect class="box" x="85.4" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="109.4" y="123.5" text-anchor="middle">cat</text><rect class="box" x="139.8" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="163.8" y="123.5" text-anchor="middle">sat</text><rect class="box" x="194.2" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="218.2" y="123.5" text-anchor="middle">on</text><rect class="box" x="248.6" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="272.6" y="123.5" text-anchor="middle">the</text><rect class="box" x="303.0" y="104.0" width="48.0" height="30.0" rx="6"/><text class="t-tick" x="327.0" y="123.5" text-anchor="middle">mat</text><line class="edge" x1="55.0" y1="140.0" x2="55.0" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="46.0" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="49.0" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="49.0" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="49.0" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="49.0" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="49.0" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="49.0" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="49.0" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><line class="edge" x1="109.4" y1="140.0" x2="109.4" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="100.4" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="103.4" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="103.4" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="103.4" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="103.4" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="103.4" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="103.4" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="103.4" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><line class="edge" x1="163.8" y1="140.0" x2="163.8" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="154.8" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="157.8" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="157.8" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="157.8" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="157.8" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="157.8" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="157.8" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="157.8" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><line class="edge" x1="218.2" y1="140.0" x2="218.2" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="209.2" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="212.2" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="212.2" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="212.2" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="212.2" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="212.2" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="212.2" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="212.2" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><line class="edge" x1="272.6" y1="140.0" x2="272.6" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="263.6" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="266.6" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="266.6" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="266.6" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="266.6" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="266.6" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="266.6" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="266.6" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><line class="edge" x1="327.0" y1="140.0" x2="327.0" y2="164.0" marker-end="url(#ah)"/><rect class="box-ghost" x="318.0" y="165.0" width="17.0" height="95.0" rx="4"/><rect class="s1" x="321.0" y="168.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="321.0" y="181.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="321.0" y="194.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="321.0" y="207.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="321.0" y="220.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="321.0" y="233.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="321.0" y="246.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><text class="t-muted" x="190.0" y="276.0" text-anchor="middle">6 tokens, each a vector of 768 numbers</text><rect class="s1" x="525.5" y="40.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.35"/><rect class="s2" x="545.5" y="40.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.43"/><rect class="s3" x="565.5" y="40.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.51"/><rect class="s4" x="525.5" y="60.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.59"/><rect class="s1" x="545.5" y="60.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.67"/><rect class="s2" x="565.5" y="60.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="525.5" y="80.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.43"/><rect class="s4" x="545.5" y="80.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.51"/><rect class="s1" x="565.5" y="80.5" width="19.0" height="19.0" rx="2" style="fill-opacity:0.59"/><text class="t-tick" x="555.0" y="116.0" text-anchor="middle">a picture, cut into 3 × 3</text><line class="edge" x1="555.0" y1="122.0" x2="555.0" y2="140.0" marker-end="url(#ah)"/><rect class="box-1" x="410.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="425.0" y="162.5" text-anchor="middle">p1</text><rect class="box-2" x="446.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="461.0" y="162.5" text-anchor="middle">p2</text><rect class="box-3" x="482.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="497.0" y="162.5" text-anchor="middle">p3</text><rect class="box-4" x="518.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="533.0" y="162.5" text-anchor="middle">p4</text><rect class="box-1" x="554.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="569.0" y="162.5" text-anchor="middle">p5</text><rect class="box-2" x="590.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="605.0" y="162.5" text-anchor="middle">p6</text><rect class="box-3" x="626.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="641.0" y="162.5" text-anchor="middle">p7</text><rect class="box-4" x="662.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="677.0" y="162.5" text-anchor="middle">p8</text><rect class="box-1" x="698.0" y="144.0" width="30.0" height="28.0" rx="6"/><text class="t-tick" x="713.0" y="162.5" text-anchor="middle">p9</text><line class="edge" x1="425.0" y1="176.0" x2="425.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="416.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s1" x="419.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="419.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="419.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="419.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="419.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="461.0" y1="176.0" x2="461.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="452.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s2" x="455.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s2" x="455.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s2" x="455.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s2" x="455.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s2" x="455.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="497.0" y1="176.0" x2="497.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="488.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s3" x="491.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="491.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="491.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="491.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="491.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="533.0" y1="176.0" x2="533.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="524.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s4" x="527.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s4" x="527.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s4" x="527.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s4" x="527.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s4" x="527.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="569.0" y1="176.0" x2="569.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="560.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s1" x="563.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="563.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="563.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="563.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="563.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="605.0" y1="176.0" x2="605.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="596.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s2" x="599.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s2" x="599.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s2" x="599.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s2" x="599.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s2" x="599.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="641.0" y1="176.0" x2="641.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="632.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s3" x="635.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="635.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="635.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="635.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="635.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="677.0" y1="176.0" x2="677.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="668.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s4" x="671.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s4" x="671.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s4" x="671.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s4" x="671.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s4" x="671.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><line class="edge" x1="713.0" y1="176.0" x2="713.0" y2="192.0" marker-end="url(#ah)"/><rect class="box-ghost" x="704.0" y="193.0" width="17.0" height="69.0" rx="4"/><rect class="s1" x="707.0" y="196.0" width="11" height="11" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="707.0" y="209.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="707.0" y="222.0" width="11" height="11" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="707.0" y="235.0" width="11" height="11" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="707.0" y="248.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/><text class="t-muted" x="570.0" y="276.0" text-anchor="middle">9 patch tokens, each a vector of 768 numbers</text></svg><figcaption>The analogy of the title. Left: a sentence becomes six word tokens, and each token becomes a vector of 768 numbers. Right: a picture is cut into nine patches, each patch becomes a token, and each token becomes a vector of 768 numbers. The same Transformer reads both.</figcaption></figure>

Here is the paper's own drawing of the model, which Part 2 explains block by block. We show it now because it makes the title concrete.

> [!PAPER] · Figure 1
> [![Figure 1 of the ViT paper: a picture is cut into nine patches, the patches are flattened and passed through a linear projection, a class token and position embeddings are added, the sequence goes through a Transformer encoder, and an MLP head on the class token outputs the class. On the right, the Transformer encoder block with layer norm, multi-head attention, an MLP and residual connections](https://ishwarj.com/img/papers/vit/p1-figure1.png/vit/p1-figure1.png)](/img/papers/vit/p1-figure1.png)
>
> **What it says:** "We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder." For classification, an extra learnable "classification token" is added to the sequence, "the standard approach" from BERT.
>
> **What it means:**
>
> Look at the bottom row of the left panel. Nine small photo squares, numbered 1 to 9, go into a box. That row *is* the title: the picture has become a short sentence of patches.
>
> Everything above that box is a Transformer encoder, copied from the language world with almost no change. This is the only drawing of the model in the paper. Part 2 takes it apart block by block.

## The authors and the footnotes {§Title}

> [!PAPER] · Authors
> [![The author list of the ViT paper: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, with footnotes for equal technical contribution and equal advising, and the affiliation Google Research, Brain Team](https://ishwarj.com/img/papers/vit/p1-authors.png/vit/p1-authors.png)](/img/papers/vit/p1-authors.png)
>
> **What it says:** the asterisk marks "equal technical contribution": five people (Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai) plus Houlsby share the credit for building and running the experiments. The dagger marks "equal advising": Dosovitskiy and Houlsby led the project. The affiliation is "Google Research, Brain Team", and the two contact e-mails belong to the two leads.
>
> **What it means:**
>
> This is the work of a large industrial team: twelve authors with one affiliation. Six of them share the equal-contribution mark, so the order of the names says less than usual.
>
> The names also explain the baseline. Kolesnikov, Beyer, Zhai and Houlsby had just written the Big Transfer paper, which scaled up convolutional networks on the same giant datasets. Those Big Transfer models are the main competitors here, so the team tests ViT against its own earlier work. One more author, Jakob Uszkoreit, is also an author of the original Transformer paper.

When this series says "the authors", it means the team.

The last sentence of the abstract carries one footnote.

> [!PAPER] · Footnote 1
> [![Footnote 1 of the ViT paper: Fine-tuning code and pre-trained models are available at https://github.com/google-research/vision_transformer](https://ishwarj.com/img/papers/vit/p1-footnote.pngvit/p1-footnote.png)](/img/papers/vit/p1-footnote.png)
>
> **What it says:** "Fine-tuning code and pre-trained models are available at" the `google-research/vision_transformer` repository on GitHub.
>
> **What it means:**
>
> This one line is the reason this series can run the real model. Google released the trained weights. The Hugging Face checkpoints we use (`google/vit-base-patch16-224` and its relatives) are conversions of those files.
>
> Read the wording with care: it says *fine-tuning* code. The pre-training itself ran on Google's private JFT-300M dataset. So you can download the finished models and adapt them, but you cannot repeat the most expensive step at home.

## The abstract, sentence by sentence {§Abstract}

The abstract is one paragraph of four sentences. We read it in two halves.

> [!PAPER] · Abstract
> [![The first half of the ViT abstract: while the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place](https://ishwarj.com/img/papers/vit/p1-abstract-1.pngt/p1-abstract-1.png)](/img/papers/vit/p1-abstract-1.png)
>
> **What it says:** the Transformer is "the de-facto standard" for language, but "its applications to computer vision remain limited". Where vision does use attention, it is either "in conjunction with convolutional networks" or used to "replace certain components of convolutional networks while keeping their overall structure in place".
>
> **What it means:**
>
> Think of a workshop that buys a new tool but uses it only inside its old machines. Vision in 2020 was like that. It had borrowed attention from language, but always inside a convolutional design that stayed in charge.
>
> The two sentences set up the gap that the paper fills. Nobody had thrown the convolutions away completely and shown that the result works. The last two sentences of the abstract claim to do exactly that.

**Sentence 1** says that in language the question was settled. "De-facto standard" means "the standard in practice, even if nobody voted on it". By 2020 every strong language model was a Transformer. The second half of the sentence says vision had not followed.

**Sentence 2** names the two ways vision had used attention so far. Both keep the **convolutional network** in charge. To see why that matters, we need the two words.

> [!DEFINITION] Convolution
> A small table of numbers (the **filter**) is slid over the picture. At each position, the filter's numbers are multiplied with the pixels under it and added up, giving one output number. Sliding it over the whole picture gives a new, filtered picture. A 3×3 filter looks at 9 neighbouring pixels at a time.

> [!DEFINITION] Filter (kernel)
> The small table of learned numbers that a convolution slides over the picture, for example 3×3 = 9 numbers. One filter might light up on vertical edges, another on a patch of orange. "Kernel" means the same thing.

> [!DEFINITION] Convolutional neural network (CNN)
> A neural network built from layers of convolutions. Early layers find edges and colours, middle layers find textures and parts, late layers find whole objects. From 2012 to 2020, essentially every strong image model was a CNN.

> [!DEFINITION] Self-attention
> The core operation of a Transformer. Each token produces a score for every other token (how relevant is that one to me?), turns the scores into weights that add up to 1, and takes a weighted mix of the other tokens' vectors. "Self" because the tokens attend to each other, within one input.

<figure class="fig"><svg viewBox="0 0 760 314" role="img" aria-label="Left: a convolution slides a 3 by 3 filter over the image, so each output depends on 9 neighbours and the same weights are used everywhere. Right: in ViT one patch attends to all 49 patches at once, with learned weights and no built-in preference for neighbours."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">CNN: a small filter slides over the image</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">ViT: every patch attends to every patch</text><line class="edge-dim" x1="380.0" y1="14.0" x2="380.0" y2="310.0"/><rect class="s1" x="101.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="101.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="127.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="153.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="179.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="205.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="231.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s1" x="257.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="box-ghost" x="100.0" y="64.0" width="78.0" height="78.0" rx="3"/><rect class="box-on" x="178.0" y="116.0" width="78.0" height="78.0" rx="3"/><rect class="box-ghost" x="100.0" y="168.0" width="78.0" height="78.0" rx="3"/><line class="edge" x1="100.0" y1="52.0" x2="282.0" y2="52.0" marker-end="url(#ah)"/><text class="t-muted" x="191.0" y="46.0" text-anchor="middle">the window visits every position</text><text class="t-tick" x="190.0" y="264.0" text-anchor="middle">3 × 3 window: each output sees 9 neighbours (locality)</text><text class="t-tick" x="190.0" y="282.0" text-anchor="middle">the same 9 weights at every position (weight sharing)</text><text class="t-muted" x="190.0" y="300.0" text-anchor="middle">far-away pixels meet only after many layers</text><rect class="s3" x="481.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="65.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="91.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="117.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="143.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="169.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="195.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="481.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="507.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="533.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="559.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="585.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="611.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><rect class="s3" x="637.0" y="221.0" width="24.0" height="24.0" rx="2" style="fill-opacity:0.18"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="77.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="103.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="129.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="155.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="181.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="207.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="493.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="519.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="545.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="571.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="597.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="623.0" y2="233.0"/><line class="edge-dim" x1="571.0" y1="129.0" x2="649.0" y2="233.0"/><rect class="box-on" x="558.0" y="116.0" width="26.0" height="26.0" rx="3"/><text class="t-tick" x="570.0" y="264.0" text-anchor="middle">one patch looks at all 49 patches, in the first layer</text><text class="t-tick" x="570.0" y="282.0" text-anchor="middle">how much to look at each is learned (attention weights)</text><text class="t-muted" x="570.0" y="300.0" text-anchor="middle">no built-in idea that neighbours matter more</text></svg><figcaption>The two operations side by side. Left: a convolution slides a 3×3 window over the picture; each output sees only its 9 neighbours and the same 9 weights are used at every position. Right: in a Transformer one patch attends to all 49 patches at once, in the very first layer, with weights that are learned rather than fixed by position.</figcaption></figure>

The figure shows the real difference between the two operations. A convolution is **local** (each output looks at a small neighbourhood) and **shares weights** (the same filter is used everywhere). Self-attention is **global** (each token can look at every token) and its weights depend on the content, not on the position. Hold on to those two words, local and global; the introduction will come back to them under the name "inductive bias".

> [!PAPER] · Abstract
> [![The second half of the ViT abstract: we show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train](https://ishwarj.com/img/papers/vit/p1-abstract-2.pngt/p1-abstract-2.png)](/img/papers/vit/p1-abstract-2.png)
>
> **What it says:** "this reliance on CNNs is not necessary". A "pure transformer applied directly to sequences of image patches" works well for classification. The recipe: pre-train "on large amounts of data", then transfer to "mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.)". The result: "excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train".
>
> **What it means:**
>
> Two sentences hold three separate claims. First, no convolutions are needed. Second, the model works when it is pre-trained on a lot of data and then transferred. Third, it costs less to train than the best convolutional networks.
>
> The second claim is the condition on the first. The abstract does not say that a Transformer beats a CNN on any data. It says this happens after pre-training "on large amounts of data". Parts 4 and 5 test each of the three claims against the paper's own tables.

**Sentence 3** is the thesis. "Pure transformer" means no convolution layers inside the network (Part 2 will show that the very first step, the patch embedding, can be written as one convolution, but the authors mean no CNN *body*). "Sequences of image patches" is the title again.

**Sentence 4** is the method and the result, and it introduces the two-step recipe that the whole paper follows.

> [!DEFINITION] Pre-training
> Training a model first on a very large, general dataset, before it ever sees the task you care about. For ViT, pre-training means learning to classify the 14 million pictures of ImageNet-21k or the 303 million pictures of JFT-300M. It is the expensive step, done once.

> [!DEFINITION] Fine-tuning and transfer
> Taking the pre-trained model and training it a little more on a smaller target task, with a new output layer for that task's labels. The knowledge "transfers" from the big dataset to the small one. In this paper, every reported number is a pre-trained model that was then fine-tuned on the benchmark in question. Part 3 covers the details.

> [!DEFINITION] Benchmark
> A fixed, public dataset and scoring rule that everyone uses, so results can be compared fairly. The abstract names three: ImageNet, CIFAR-100 and VTAB.

> [!DEFINITION] ImageNet (ImageNet-1k)
> The standard image classification benchmark since 2012: about 1.3 million training photos sorted into 1,000 classes (120 dog breeds among them), with 50,000 test photos. "ImageNet" in this paper means this 1,000-class set, officially the ILSVRC-2012 dataset.

> [!DEFINITION] CIFAR-100
> A small benchmark of 60,000 tiny 32×32 colour pictures in 100 classes (apple, bicycle, whale, ...), 50,000 for training and 10,000 for testing. Models are often tested on it after pre-training elsewhere, because 50,000 tiny pictures are not enough to learn from scratch.

> [!DEFINITION] VTAB
> The Visual Task Adaptation Benchmark: 19 different classification tasks (natural photos, medical and satellite images, and tasks about counting or distance), each with only 1,000 training examples. It tests how well a pre-trained model adapts to new kinds of pictures with very little data. The score is the average over the 19 tasks.

> [!DEFINITION] State of the art (SOTA)
> The best result anyone has published so far on a benchmark. "Compared to state-of-the-art convolutional networks" means "compared to the best CNN results published".

<figure class="fig"><svg viewBox="0 0 810 292" role="img" aria-label="The recipe of the paper: pre-train ViT once on a very large labelled image set (ImageNet-21k or JFT-300M), then make a copy for each small benchmark and fine-tune it with one new output layer."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="26.0" text-anchor="start">Step 1: pre-train once on a very large labelled image set</text><text class="t-title" x="430.0" y="26.0" text-anchor="start">Step 2: transfer to each small benchmark</text><rect class="box" x="30.0" y="52.0" width="140.0" height="90.0" rx="6"/><rect class="box" x="36.0" y="58.0" width="140.0" height="90.0" rx="6"/><rect class="box" x="42.0" y="64.0" width="140.0" height="90.0" rx="6"/><rect class="box" x="48.0" y="70.0" width="140.0" height="90.0" rx="6"/><text class="t-tick" x="106.0" y="92.0" text-anchor="middle">ImageNet-21k: 14M images</text><text class="t-tick" x="106.0" y="110.0" text-anchor="middle">or JFT-300M: 303M images</text><text class="t-muted" x="106.0" y="130.0" text-anchor="middle">with class labels</text><text class="t-tick" x="106.0" y="180.0" text-anchor="middle">thousands of TPU-core-days</text><line class="edge" x1="188.0" y1="100.0" x2="232.0" y2="100.0" marker-end="url(#ah)"/><rect class="box-on" x="236.0" y="70.0" width="130.0" height="64.0" rx="12"/><text class="t-big" x="301.0" y="98.0" text-anchor="middle">ViT</text><text class="t-tick" x="301.0" y="118.0" text-anchor="middle">learns to see</text><text class="t-tick" x="301.0" y="160.0" text-anchor="middle">all the weights</text><text class="t-tick" x="301.0" y="178.0" text-anchor="middle">are learned here</text><line class="edge" x1="368.0" y1="102.0" x2="444.0" y2="74.0" marker-end="url(#ah)"/><rect class="box-on" x="448.0" y="48.0" width="110.0" height="52.0" rx="10"/><text class="t-note" x="503.0" y="71.0" text-anchor="middle">ViT copy</text><text class="t-muted" x="503.0" y="89.0" text-anchor="middle">starts pre-trained</text><rect class="box-1" x="558.0" y="56.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="586.0" y1="74.0" x2="612.0" y2="74.0" marker-end="url(#ah)"/><text class="t-note" x="618.0" y="69.0" text-anchor="start">ImageNet</text><text class="t-tick" x="618.0" y="87.0" text-anchor="start">1,000 classes, 1.3M images</text><line class="edge" x1="368.0" y1="102.0" x2="444.0" y2="148.0" marker-end="url(#ah)"/><rect class="box-on" x="448.0" y="122.0" width="110.0" height="52.0" rx="10"/><text class="t-note" x="503.0" y="145.0" text-anchor="middle">ViT copy</text><text class="t-muted" x="503.0" y="163.0" text-anchor="middle">starts pre-trained</text><rect class="box-2" x="558.0" y="130.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="586.0" y1="148.0" x2="612.0" y2="148.0" marker-end="url(#ah)"/><text class="t-note" x="618.0" y="143.0" text-anchor="start">CIFAR-100</text><text class="t-tick" x="618.0" y="161.0" text-anchor="start">100 classes, 50k tiny images</text><line class="edge" x1="368.0" y1="102.0" x2="444.0" y2="222.0" marker-end="url(#ah)"/><rect class="box-on" x="448.0" y="196.0" width="110.0" height="52.0" rx="10"/><text class="t-note" x="503.0" y="219.0" text-anchor="middle">ViT copy</text><text class="t-muted" x="503.0" y="237.0" text-anchor="middle">starts pre-trained</text><rect class="box-3" x="558.0" y="204.0" width="26.0" height="36.0" rx="6"/><line class="edge" x1="586.0" y1="222.0" x2="612.0" y2="222.0" marker-end="url(#ah)"/><text class="t-note" x="618.0" y="217.0" text-anchor="start">VTAB</text><text class="t-tick" x="618.0" y="235.0" text-anchor="start">19 tasks, 1,000 images each</text><text class="t-tick" x="570.0" y="278.0" text-anchor="middle">+ one new classification layer per benchmark (fine-tuning, Part 3)</text></svg><figcaption>The recipe of the paper. Step 1: pre-train ViT once on a very large labelled picture set (ImageNet-21k or JFT-300M), which costs thousands of TPU-core-days. Step 2: for each small benchmark, start from a copy of the pre-trained model, add one new output layer, and fine-tune.</figcaption></figure>

If you have read the [BERT breakdown](https://ishwarj.com/papers/bert/), this recipe is familiar: BERT pre-trains on text once and fine-tunes a copy per task. ViT does the same with pictures, with one difference. BERT's pre-training needs no labels (it hides words and guesses them). ViT's pre-training in this paper is **supervised**: every pre-training picture comes with a class label.

> [!DEFINITION] Supervised learning
> Learning from examples that come with the right answer attached. For ViT, every training picture has a label such as "tabby cat", and the model is trained to output that label. The opposite is learning from unlabelled data, which the paper only tries in a small side experiment (Part 5).

The abstract also says what the paper does *not* do. It says "image classification tasks", not detection or segmentation.

<figure class="fig"><svg viewBox="0 0 760 236" role="img" aria-label="What the paper does and does not do: it trains ViT for image classification (one label per picture); object detection and segmentation are left to later work."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box-on" x="30.0" y="20.0" width="210.0" height="200.0" rx="12"/><text class="t-title" x="135.0" y="46.0" text-anchor="middle">image classification</text><text class="t-muted" x="135.0" y="66.0" text-anchor="middle">one label for the whole picture</text><rect class="s1" x="75.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="75.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="85.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="95.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="105.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="115.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="125.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="135.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="145.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="155.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="165.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="175.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="185.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="box-3" x="105.0" y="176.0" width="60.0" height="22.0" rx="5"/><text class="t-tick" x="135.0" y="191.0" text-anchor="middle">"cat"</text><text class="t-on" x="135.0" y="208.0" text-anchor="middle">this paper</text><rect class="box-ghost" x="275.0" y="20.0" width="210.0" height="200.0" rx="12"/><text class="t-title" x="380.0" y="46.0" text-anchor="middle">object detection</text><text class="t-muted" x="380.0" y="66.0" text-anchor="middle">a box around every object</text><rect class="s1" x="320.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="320.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="330.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="340.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="350.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="360.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="370.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="380.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="390.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="400.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="410.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="420.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="430.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="box-2" x="330.0" y="99.0" width="45.0" height="50.0" rx="3"/><rect class="box-4" x="385.0" y="109.0" width="50.0" height="45.0" rx="3"/><text class="t-tick" x="380.0" y="191.0" text-anchor="middle">cat, remote</text><text class="t-muted" x="380.0" y="208.0" text-anchor="middle">left to later work (Part 6)</text><rect class="box-ghost" x="520.0" y="20.0" width="210.0" height="200.0" rx="12"/><text class="t-title" x="625.0" y="46.0" text-anchor="middle">segmentation</text><text class="t-muted" x="625.0" y="66.0" text-anchor="middle">a class for every pixel</text><rect class="s1" x="565.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="84.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="144.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="565.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="575.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="585.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="595.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="605.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="615.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="625.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="635.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="645.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="655.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="665.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s1" x="675.5" y="154.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.15"/><rect class="s2" x="575.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="585.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="595.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="605.5" y="94.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="575.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="585.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="595.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="605.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="575.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="585.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="595.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="605.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="575.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="585.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="595.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="605.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="575.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="585.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="595.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="605.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="635.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="645.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="655.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="665.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="675.5" y="104.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="635.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="645.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="655.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="665.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="675.5" y="114.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="635.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="645.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="655.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="665.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="675.5" y="124.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="635.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="645.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="655.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="665.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><rect class="s4" x="675.5" y="134.5" width="9.0" height="9.0" rx="1" style="fill-opacity:0.70"/><text class="t-tick" x="625.0" y="191.0" text-anchor="middle">every pixel labelled</text><text class="t-muted" x="625.0" y="208.0" text-anchor="middle">left to later work (Part 6)</text></svg><figcaption>What the paper does and does not do. It trains ViT for image classification: one label for the whole picture. Object detection (a box around every object) and segmentation (a class for every pixel) are left to later work, which Part 6 covers.</figcaption></figure>

### Let us run it

Before reading the introduction, it helps to see the object the paper is talking about. We loaded the released ViT-B/16 model (`google/vit-base-patch16-224`: pre-trained on ImageNet-21k, fine-tuned on ImageNet) and gave it one picture, the test photo from the `huggingface/cats-image` dataset.

<figure><img src="/img/papers/vit/p1-sample.png" alt="The sample picture: two tabby cats lying on a pink sofa, with a remote control between them" loading="lazy" width="480" /><figcaption>The picture we use throughout this part: two tabby cats on a pink sofa, with two remote controls. It is 640×480 pixels; the model resizes it to 224×224.</figcaption></figure>

The code is in [`vit_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1.py):

```python
proc = AutoImageProcessor.from_pretrained('google/vit-base-patch16-224')
vit = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224').eval()
x = proc(images=image, return_tensors='pt').pixel_values            # resized to 224 x 224, scaled to [-1, 1]
C, H, W = x.shape[1:]
P = vit.config.patch_size
N = (H // P) * (W // P)
```

```text
ViT-B/16 (google/vit-base-patch16-224)
  image tensor: (1, 3, 224, 224)  = batch of 1, 3 channels, 224 x 224 pixels
  numbers in one image: 3 x 224 x 224 = 150,528
  patch size P = 16; patches per side = 224 // 16 = 14; number of patches N = 14 x 14 = 196
  numbers in one patch: 16 x 16 x 3 = 768
  sequence length with the [class] token: 196 + 1 = 197
  hidden size D = 768, layers = 12, heads = 12
  parameters: 86,567,656
  token matrix after the embedding step: (1, 197, 768)  (1 image, 197 tokens, 768 numbers each)
  output scores: (1, 1000)  (one score per ImageNet class)
```

Every number in that output is one line of arithmetic, and it is worth doing the arithmetic once by hand. The picture is a block of numbers of shape channels × height × width:

$$
\mathbf{x} \in \mathbb{R}^{C \times H \times W}, \qquad C \cdot H \cdot W = 3 \times 224 \times 224 = 150{,}528
$$

where:

- $$C = 3$$ is the number of colour channels (red, green, blue);
- $$H = W = 224$$ are the height and width in pixels, after the model's preprocessing resized the photo;
- $$150{,}528$$ is how many numbers the model receives for one picture.

Cutting it into patches of side $$P = 16$$ gives

$$
N = \frac{H}{P} \cdot \frac{W}{P} = \frac{224}{16} \cdot \frac{224}{16} = 14 \times 14 = 196 \ \text{patches},
\qquad
P^2 C = 16 \times 16 \times 3 = 768 \ \text{numbers per patch}
$$

and the sequence the Transformer reads has one extra token at the front, the `[class]` token (explained in Part 2), so its length is

$$
N + 1 = 197
$$

The 196 × 768 = 150,528 numbers of the patches are exactly the 150,528 numbers of the picture: nothing is lost or added by cutting, only rearranged. The name **ViT-B/16** encodes two of these numbers: "B" for Base (the BERT-base size: 12 layers, 768-wide vectors, 12 attention heads) and "/16" for the patch side. The model has 86.6 million parameters, which the paper rounds to 86M in its Table 1.

> [!DEFINITION] Parameter
> One learned number inside a model: a weight. ViT-B/16 has about 86 million of them; they are set by training and then fixed. "Parameters", "weights" and "model size" are used almost interchangeably.

> [!DEFINITION] Sequence
> An ordered list of tokens. A Transformer always reads a sequence. ViT's sequence has 197 tokens: 196 patches plus the class token.

<figure class="fig"><svg viewBox="0 0 760 316" role="img" aria-label="A 224 by 224 image becomes a 14 by 14 grid of 16 by 16 patches: 196 patches of 768 numbers each, plus one class token, so 197 tokens. The arithmetic is printed on the right."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="s1" x="40.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="56.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="72.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="88.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="104.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="120.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="136.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="152.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="168.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="184.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="200.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="216.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="232.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="248.8" y="60.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="40.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="56.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="72.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="88.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="104.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="120.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="136.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="152.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="168.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="184.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="200.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="216.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="232.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="248.8" y="76.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="40.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="56.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="72.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="88.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="104.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="120.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="136.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="152.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="168.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="184.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="200.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="216.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="232.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="248.8" y="92.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="40.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="56.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="72.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="88.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="104.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="120.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="136.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="152.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="168.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="184.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="200.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="216.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="232.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="248.8" y="108.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="40.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="56.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="72.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="88.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="104.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="120.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="136.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="152.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="168.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="184.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="200.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="216.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="232.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="248.8" y="124.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="40.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="56.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="72.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="88.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="104.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="120.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="136.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="152.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="168.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="184.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="200.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="216.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="232.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="248.8" y="140.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="40.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="56.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="72.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="88.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="104.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="120.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="136.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="152.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="168.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="184.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="200.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="216.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="232.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="248.8" y="156.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="40.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="56.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="72.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="88.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="104.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="120.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="136.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="152.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="168.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="184.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="200.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="216.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="232.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="248.8" y="172.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="40.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="56.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="72.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="88.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="104.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="120.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="136.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="152.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="168.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="184.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="200.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="216.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="232.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="248.8" y="188.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="40.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="56.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="72.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="88.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="104.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="120.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="136.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="152.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="168.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="184.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="200.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="216.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="232.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="248.8" y="204.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="40.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="56.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="72.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="88.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="104.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="120.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="136.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="152.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="168.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="184.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="200.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="216.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="232.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="248.8" y="220.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="40.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="56.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="72.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="88.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="104.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="120.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="136.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="152.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="168.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="184.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="200.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="216.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="232.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="248.8" y="236.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="40.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="56.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="72.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="88.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="104.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="120.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="136.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="152.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="168.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="184.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="200.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="216.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="232.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="248.8" y="252.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="40.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="56.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="72.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="88.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="104.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="120.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="136.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="152.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.28"/><rect class="s1" x="168.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.37"/><rect class="s1" x="184.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="200.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.34"/><rect class="s1" x="216.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.22"/><rect class="s1" x="232.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.31"/><rect class="s1" x="248.8" y="268.8" width="14.5" height="14.5" rx="1" style="fill-opacity:0.40"/><rect class="box-on" x="184.0" y="124.0" width="16.0" height="16.0" rx="2"/><path class="edge" d="M40.0,54.0 L40.0,46.0 L264.0,46.0 L264.0,54.0"/><text class="t-tick" x="152.0" y="34.0" text-anchor="middle">224 pixels = 14 patches × 16 pixels</text><path class="edge" d="M40.0,286.0 L40.0,294.0 L264.0,294.0 L264.0,286.0"/><text class="t-tick" x="152.0" y="314.0" text-anchor="middle">14 × 14 = 196 patches</text><line class="edge-dim" x1="200.0" y1="124.0" x2="330.0" y2="60.0"/><line class="edge-dim" x1="200.0" y1="140.0" x2="330.0" y2="172.0"/><rect class="s3" x="330.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="337.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="344.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="351.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="358.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="365.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="372.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="379.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="386.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="393.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="400.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="407.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="414.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="421.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="428.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="435.5" y="60.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="330.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="337.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="344.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="351.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="358.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="365.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="372.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="379.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="386.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="393.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="400.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="407.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="414.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="421.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="428.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="435.5" y="67.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="330.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="337.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="344.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="351.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="358.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="365.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="372.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="379.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="386.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="393.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="400.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="407.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="414.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="421.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="428.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="435.5" y="74.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="330.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="337.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="344.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="351.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="358.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="365.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="372.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="379.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="386.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="393.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="400.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="407.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="414.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="421.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="428.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="435.5" y="81.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="330.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="337.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="344.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="351.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="358.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="365.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="372.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="379.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="386.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="393.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="400.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="407.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="414.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="421.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="428.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="435.5" y="88.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="330.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="337.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="344.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="351.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="358.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="365.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="372.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="379.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="386.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="393.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="400.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="407.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="414.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="421.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="428.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="435.5" y="95.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="330.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="337.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="344.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="351.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="358.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="365.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="372.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="379.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="386.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="393.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="400.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="407.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="414.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="421.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="428.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="435.5" y="102.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="330.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="337.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="344.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="351.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="358.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="365.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="372.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="379.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="386.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="393.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="400.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="407.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="414.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="421.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="428.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="435.5" y="109.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="330.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="337.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="344.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="351.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="358.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="365.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="372.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="379.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="386.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="393.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="400.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="407.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="414.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="421.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="428.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="435.5" y="116.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="330.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="337.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="344.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="351.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="358.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="365.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="372.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="379.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="386.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="393.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="400.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="407.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="414.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="421.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="428.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="435.5" y="123.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="330.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="337.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="344.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="351.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="358.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="365.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="372.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="379.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="386.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="393.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="400.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="407.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="414.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="421.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="428.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="435.5" y="130.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="330.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="337.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="344.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="351.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="358.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="365.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="372.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="379.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="386.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="393.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="400.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="407.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="414.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="421.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="428.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="435.5" y="137.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="330.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="337.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="344.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="351.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="358.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="365.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="372.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="379.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="386.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="393.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="400.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="407.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="414.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="421.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="428.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="435.5" y="144.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="330.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="337.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="344.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="351.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="358.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="365.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="372.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="379.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="386.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="393.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="400.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="407.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="414.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="421.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="428.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="435.5" y="151.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="330.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="337.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="344.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="351.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="358.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="365.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="372.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="379.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="386.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="393.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="400.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="407.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="414.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="421.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="428.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="435.5" y="158.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="330.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="337.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="344.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="351.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="358.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="365.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="372.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="379.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="386.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="393.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="400.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="407.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="414.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="421.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="428.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="435.5" y="165.5" width="6.0" height="6.0" rx="1" style="fill-opacity:0.75"/><rect class="box-on" x="329.0" y="59.0" width="114.0" height="114.0" rx="2"/><path class="edge" d="M330.0,54.0 L330.0,46.0 L442.0,46.0 L442.0,54.0"/><text class="t-tick" x="386.0" y="34.0" text-anchor="middle">one patch: 16 × 16 pixels</text><text class="t-tick" x="386.0" y="196.0" text-anchor="middle">× 3 colour channels</text><text class="t-title" x="480.0" y="60.0" text-anchor="start">the arithmetic of ViT-B/16</text><text class="t-label" x="480.0" y="90.0" text-anchor="start">the image</text><text class="t-val" x="480.0" y="108.0" text-anchor="start">3 × 224 × 224 = 150,528 numbers</text><text class="t-label" x="480.0" y="130.0" text-anchor="start">one patch</text><text class="t-val" x="480.0" y="148.0" text-anchor="start">16 × 16 × 3 = 768 numbers</text><text class="t-label" x="480.0" y="170.0" text-anchor="start">patches</text><text class="t-val" x="480.0" y="188.0" text-anchor="start">(224 / 16)² = 14 × 14 = 196</text><text class="t-label" x="480.0" y="210.0" text-anchor="start">tokens</text><text class="t-val" x="480.0" y="228.0" text-anchor="start">196 + 1 [class] = 197</text><text class="t-label" x="480.0" y="250.0" text-anchor="start">token matrix</text><text class="t-val" x="480.0" y="268.0" text-anchor="start">197 × 768 numbers</text></svg><figcaption>The arithmetic of ViT-B/16. A 224×224 picture is a 14×14 grid of 16×16 patches: 196 patches, each 16 × 16 × 3 = 768 numbers. With one class token the sequence has 197 tokens, each a vector of 768 numbers.</figcaption></figure>

Now the answer. The model outputs 1,000 scores, one per ImageNet class; a softmax turns them into probabilities that add up to 1:

$$
p_k = \frac{e^{s_k}}{\sum_{j=1}^{1000} e^{s_j}}
$$

where $$s_k$$ is the raw score for class $$k$$ and $$p_k$$ is its probability. We printed the five largest, and then did the same with a classic convolutional network, ResNet-50 (`microsoft/resnet-50`), on the same picture:

```python
rproc = AutoImageProcessor.from_pretrained('microsoft/resnet-50')
resnet = ResNetForImageClassification.from_pretrained('microsoft/resnet-50').eval()
xr = rproc(images=image, return_tensors='pt').pixel_values
with torch.no_grad():
    ro = resnet(pixel_values=xr, output_hidden_states=True)
p_res = torch.softmax(ro.logits[0], -1)
```

```text
  top-5 classes:                       (ViT-B/16)
     0.9374  Egyptian cat
     0.0384  tabby
     0.0144  tiger cat
     0.0033  lynx
     0.0007  Siamese cat

ResNet-50 (microsoft/resnet-50), a convolutional network
  image tensor: (1, 3, 224, 224)
  parameters: 25,557,032
  feature map after stage 0: (64, 56, 56)  (channels, height, width)
  feature map after stage 1: (256, 56, 56)  (channels, height, width)
  feature map after stage 2: (512, 28, 28)  (channels, height, width)
  feature map after stage 3: (1024, 14, 14)  (channels, height, width)
  feature map after stage 4: (2048, 7, 7)  (channels, height, width)
  top-5 classes:
     0.9416  tiger cat
     0.0344  tabby
     0.0016  remote control
     0.0014  Egyptian cat
     0.0008  jinrikisha

both models agree on the top class: False  (ViT: Egyptian cat, ResNet-50: tiger cat)
```

<figure class="fig"><svg viewBox="0 0 760 236" role="img" aria-label="Top-5 ImageNet classes for the sample picture: ViT-B/16 says Egyptian cat 0.937, tabby, tiger cat, lynx, Siamese cat; ResNet-50 says tiger cat 0.942, tabby, remote control, Egyptian cat, jinrikisha."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="24.0" text-anchor="middle">the same picture (two tabby cats on a pink sofa) through two models</text><text class="t-title" x="20.0" y="56.0" text-anchor="start">ViT-B/16 (Transformer)</text><text class="t-tick t-strong" x="122.0" y="83.0" text-anchor="end">Egyptian cat</text><g class="mark"><title>Egyptian cat: 0.9374</title><rect class="s3" x="130.0" y="69.0" width="178.1" height="18.0" rx="3"/></g><text class="t-val" x="314.1" y="83.0" text-anchor="start">0.9374</text><text class="t-tick" x="122.0" y="109.0" text-anchor="end">tabby</text><g class="mark"><title>tabby: 0.0384</title><rect class="s3" x="130.0" y="95.0" width="7.3" height="18.0" rx="3"/></g><text class="t-val" x="143.3" y="109.0" text-anchor="start">0.0384</text><text class="t-tick" x="122.0" y="135.0" text-anchor="end">tiger cat</text><g class="mark"><title>tiger cat: 0.0144</title><rect class="s3" x="130.0" y="121.0" width="2.7" height="18.0" rx="3"/></g><text class="t-val" x="138.7" y="135.0" text-anchor="start">0.0144</text><text class="t-tick" x="122.0" y="161.0" text-anchor="end">lynx</text><g class="mark"><title>lynx: 0.0033</title><rect class="s3" x="130.0" y="147.0" width="2.0" height="18.0" rx="3"/></g><text class="t-val" x="138.0" y="161.0" text-anchor="start">0.0033</text><text class="t-tick" x="122.0" y="187.0" text-anchor="end">Siamese cat</text><g class="mark"><title>Siamese cat: 0.0007</title><rect class="s3" x="130.0" y="173.0" width="2.0" height="18.0" rx="3"/></g><text class="t-val" x="138.0" y="187.0" text-anchor="start">0.0007</text><text class="t-title" x="400.0" y="56.0" text-anchor="start">ResNet-50 (CNN)</text><text class="t-tick t-strong" x="502.0" y="83.0" text-anchor="end">tiger cat</text><g class="mark"><title>tiger cat: 0.9416</title><rect class="s1" x="510.0" y="69.0" width="178.9" height="18.0" rx="3"/></g><text class="t-val" x="694.9" y="83.0" text-anchor="start">0.9416</text><text class="t-tick" x="502.0" y="109.0" text-anchor="end">tabby</text><g class="mark"><title>tabby: 0.0344</title><rect class="s1" x="510.0" y="95.0" width="6.5" height="18.0" rx="3"/></g><text class="t-val" x="522.5" y="109.0" text-anchor="start">0.0344</text><text class="t-tick" x="502.0" y="135.0" text-anchor="end">remote control</text><g class="mark"><title>remote control: 0.0016</title><rect class="s1" x="510.0" y="121.0" width="2.0" height="18.0" rx="3"/></g><text class="t-val" x="518.0" y="135.0" text-anchor="start">0.0016</text><text class="t-tick" x="502.0" y="161.0" text-anchor="end">Egyptian cat</text><g class="mark"><title>Egyptian cat: 0.0014</title><rect class="s1" x="510.0" y="147.0" width="2.0" height="18.0" rx="3"/></g><text class="t-val" x="518.0" y="161.0" text-anchor="start">0.0014</text><text class="t-tick" x="502.0" y="187.0" text-anchor="end">jinrikisha</text><g class="mark"><title>jinrikisha: 0.0008</title><rect class="s1" x="510.0" y="173.0" width="2.0" height="18.0" rx="3"/></g><text class="t-val" x="518.0" y="187.0" text-anchor="start">0.0008</text><text class="t-muted" x="380.0" y="220.0" text-anchor="middle">both say "cat" with about 94% confidence; they disagree about the breed (the picture has two cats and a remote control)</text></svg><figcaption>The same picture through two models. ViT-B/16 says Egyptian cat with probability 0.937; ResNet-50 says tiger cat with 0.942. Both are sure it is a cat and both put "tabby" second. They disagree about the breed, and ResNet-50 also noticed the remote control.</figcaption></figure>

Three things to notice:

- **Both models are right**, in the sense that matters: the picture shows cats, and both put about 94% on a cat class. ImageNet has several house-cat classes (Egyptian cat, tabby, tiger cat, Persian, Siamese), and the picture does not clearly belong to one, so the disagreement is about a fine label, not about the object.
- **ResNet-50 is a different shape of machine.** Its printed "feature maps" shrink from 56×56 to 7×7 as the picture passes through five stages, while the number of channels grows from 64 to 2,048. That is what a CNN does: it summarises larger and larger neighbourhoods. ViT keeps its 197 tokens of 768 numbers at every layer; nothing shrinks.
- **ResNet-50 has 25.6 million parameters, ViT-B/16 86.6 million.** The paper's fair comparison is against much bigger CNNs (Part 4). This run is only a first look, not a contest.

> [!DEFINITION] Accuracy (top-1)
> The share of test pictures for which the model's single most probable class is the correct label. If a model gets 885 of 1,000 pictures right, its top-1 accuracy is 88.5%. All the headline numbers of this paper are top-1 accuracies.

## Transformers took over language {§1}

Now the introduction. Its first paragraph is about language, not pictures.

> [!PAPER] · §1
> [![The first paragraph of the ViT introduction: self-attention-based architectures, in particular Transformers, have become the model of choice in natural language processing. The dominant approach is to pre-train on a large text corpus and then fine-tune on a smaller task-specific dataset. Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters. With the models and datasets growing, there is still no sign of saturating performance](https://ishwarj.com/img/papers/vit/p1-intro-nlp.pngit/p1-intro-nlp.png)](/img/papers/vit/p1-intro-nlp.png)
>
> **What it says:** Transformers are "the model of choice" in NLP. The standard recipe is to "pre-train on a large text corpus and then fine-tune on a smaller task-specific dataset". Because Transformers are efficient and scale well, models "with over 100B parameters" exist, and "there is still no sign of saturating performance".
>
> **What it means:**
>
> In language, the story of 2017 to 2020 was short: make the Transformer bigger, give it more text, and it gets better. Nobody had found the ceiling. Four citations back this up. They are the Transformer (Vaswani et al., 2017), BERT (Devlin et al., 2019), GPT-3 (Brown et al., 2020) and GShard (Lepikhin et al., 2020).
>
> The paragraph says nothing about pictures, and that is the point. It is the motivation for everything that follows. If the same design and the same recipe keep getting better with scale in language, do pictures behave the same way?

Each of the four citations stands for one idea, and the paper leans on all four. We screenshot the sentences it points to.

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · Abstract
> [![The first sentences of the Transformer paper's abstract: the dominant sequence transduction models are based on complex recurrent or convolutional neural networks. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely](https://ishwarj.com/img/papers/vit/p1-vaswani-abstract.pngaswani-abstract.png)](/img/papers/vit/p1-vaswani-abstract.png)
>
> **What it says:** earlier models used recurrent or convolutional networks. The Transformer is "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely".
>
> **What it means:**
>
> In 2017 the Transformer was a model for translation between languages. Its pitch was to drop the older building blocks, recurrence and convolution, and keep only attention. "Dispensing with recurrence and convolutions entirely" was a bold claim for text.
>
> ViT makes the same move for pictures. There the claim is bolder, because convolutions had been the base of every strong vision model since 2012.

> [!PAPER] Devlin et al. (2019), BERT · Abstract
> [![The first paragraph of the BERT abstract: BERT is designed to pre-train deep bidirectional representations from unlabeled text, and the pre-trained model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks](https://ishwarj.com/img/papers/vit/p1-bert-abstract.png1-bert-abstract.png)](/img/papers/vit/p1-bert-abstract.png)
>
> **What it says:** BERT is designed to "pre-train deep bidirectional representations from unlabeled text" and then be "fine-tuned with just one additional output layer".
>
> **What it means:**
>
> BERT, from 2018, made "pre-train, then fine-tune" the standard recipe of language processing. It is "the dominant approach" that the ViT paragraph describes. The [BERT breakdown](https://ishwarj.com/papers/bert/) on this site reads it in full.
>
> ViT copies the shape of the recipe: pre-train once, then fine-tune with one new output layer. It also copies BERT's model sizes and its `[class]` token. A reader who knows BERT already knows half of ViT.

> [!PAPER] Brown et al. (2020), GPT-3 · Abstract
> [![A sentence from the GPT-3 abstract: we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model](https://ishwarj.com/img/papers/vit/p1-gpt3-abstract.png1-gpt3-abstract.png)](/img/papers/vit/p1-gpt3-abstract.png)
>
> **What it says:** a language model "with 175 billion parameters, 10x more than any previous non-sparse language model".
>
> **What it means:**
>
> GPT-3 came out five months before ViT. It is one of the two "over 100B parameters" examples of the paragraph.
>
> The number itself is not the point. The point is that the model kept getting better as it got bigger, with no sign of a ceiling. ViT asks whether pictures show the same trend.

> [!PAPER] Lepikhin et al. (2020), GShard · Abstract
> [![A sentence from the GShard abstract: GShard enabled us to scale up a multilingual neural machine translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600 billion parameters using automatic sharding](https://ishwarj.com/img/papers/vit/p1-gshard-abstract.pnggshard-abstract.png)](/img/papers/vit/p1-gshard-abstract.png)
>
> **What it says:** a translation Transformer scaled "beyond 600 billion parameters".
>
> **What it means:**
>
> GShard is a Google paper on how to spread one giant model across many chips. It is the other "over 100B" example of the paragraph.
>
> It also tells us something the paper does not write. The authors worked next to the people who trained such giant models, so they had the tools to train very large models. The open question was whether vision would reward those tools.

<figure class="fig"><svg viewBox="0 0 760 244" role="img" aria-label="Parameter counts of the language Transformers the introduction points to, on a log scale: 213 million for the 2017 Transformer, 340 million for BERT-large, 175 billion for GPT-3 and 600 billion for GShard."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="28.0" text-anchor="start">parameters of NLP Transformers named in the first paragraph (log scale)</text><text class="t-tick" x="230.0" y="70.0" text-anchor="end">Transformer "big", 2017</text><g class="mark"><title>Transformer "big", 2017: 213,000,000 parameters</title><rect class="s2" x="240" y="54" width="34.5" height="24" rx="3"/></g><text class="t-val" x="280.5" y="70.0" text-anchor="start">213M</text><text class="t-tick" x="230.0" y="104.0" text-anchor="end">BERT-large, 2018</text><g class="mark"><title>BERT-large, 2018: 340,000,000 parameters</title><rect class="s2" x="240" y="88" width="55.8" height="24" rx="3"/></g><text class="t-val" x="301.8" y="104.0" text-anchor="start">340M</text><text class="t-tick" x="230.0" y="138.0" text-anchor="end">GPT-3, 2020</text><g class="mark"><title>GPT-3, 2020: 175,000,000,000 parameters</title><rect class="s2" x="240" y="122" width="340.5" height="24" rx="3"/></g><text class="t-val" x="586.5" y="138.0" text-anchor="start">175B</text><text class="t-tick" x="230.0" y="172.0" text-anchor="end">GShard, 2020</text><g class="mark"><title>GShard, 2020: 600,000,000,000 parameters</title><rect class="s2" x="240" y="156" width="396.7" height="24" rx="3"/></g><text class="t-val" x="642.7" y="172.0" text-anchor="start">600B</text><line class="grid" x1="240.0" y1="46.0" x2="240.0" y2="186.0"/><text class="t-muted" x="240.0" y="202.0" text-anchor="middle">100M</text><line class="grid" x1="345.0" y1="46.0" x2="345.0" y2="186.0"/><text class="t-muted" x="345.0" y="202.0" text-anchor="middle">1B</text><line class="grid" x1="450.0" y1="46.0" x2="450.0" y2="186.0"/><text class="t-muted" x="450.0" y="202.0" text-anchor="middle">10B</text><line class="grid" x1="555.0" y1="46.0" x2="555.0" y2="186.0"/><text class="t-muted" x="555.0" y="202.0" text-anchor="middle">100B</text><line class="grid" x1="660.0" y1="46.0" x2="660.0" y2="186.0"/><text class="t-muted" x="660.0" y="202.0" text-anchor="middle">1T</text><text class="t-muted" x="20.0" y="228.0" text-anchor="start">ViT-B/16 has 86M parameters, ViT-L/16 307M, ViT-H/14 632M: the sizes of BERT, not of GPT-3</text></svg><figcaption>How big the language Transformers in that paragraph are, on a log scale: 213 million parameters for the 2017 Transformer's large version, 340 million for BERT-large, 175 billion for GPT-3 and 600 billion for GShard. The ViT models of this paper (86 million to 632 million) are BERT-sized, not GPT-3-sized.</figcaption></figure>

The paragraph's last sentence is the one to remember: "there is still no sign of saturating performance". In language, the curve of accuracy against model size and data size had not flattened. The rest of the introduction asks whether the same curve exists for pictures, and answers yes, with a catch.

> [!DEFINITION] Embedding
> The vector of numbers that stands for one token inside the model. A word embedding is the vector for a word; a patch embedding is the vector for a patch. ViT-B uses embeddings of 768 numbers, the same width as BERT-base.

## In vision, convolutions still ruled {§1}

> [!PAPER] · §1
> [![The second paragraph of the ViT introduction: in computer vision, convolutional architectures remain dominant. Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention, some replacing the convolutions entirely. The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns. Therefore, in large-scale image recognition, classic ResNet-like architectures are still state of the art](https://ishwarj.com/img/papers/vit/p1-intro-cnn.pngit/p1-intro-cnn.png)](/img/papers/vit/p1-intro-cnn.png)
>
> **What it says:** "convolutional architectures remain dominant" (LeCun 1989, AlexNet 2012, ResNet 2016). Several works tried "combining CNN-like architectures with self-attention" (Non-local networks, DETR), some "replacing the convolutions entirely" (stand-alone self-attention, Axial-DeepLab). The latter are "theoretically efficient" but "have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns". So "classic ResNet-like architectures are still state of the art" (Mahajan 2018, Noisy Student, Big Transfer).
>
> **What it means:**
>
> This paragraph explains *why nobody had done the obvious thing*. Plain attention over every pixel is far too expensive. So people invented clever local attention patterns that look only at nearby pixels.
>
> Those patterns work on paper ("theoretically efficient"). But the chips that train big models are fast only at plain, dense matrix work, and the custom patterns did not run fast on them. So those models never grew large. The paper's way out is to attend over patches, not pixels, so that plain attention becomes cheap enough.

Let us take the three sentences in turn.

**"Convolutional architectures remain dominant."** The three citations are the history of CNNs in three steps. LeCun et al. (1989) trained a convolutional network to read handwritten postcodes, the first practical CNN. Krizhevsky et al. (2012), the AlexNet paper, won the 2012 ImageNet contest by a large margin with a deep CNN trained on GPUs, which started the deep-learning era in vision. He et al. (2016), the ResNet paper, made CNNs of 100+ layers trainable.

> [!PAPER] He et al. (2015), Deep Residual Learning (ResNet) · Abstract
> [![A sentence from the ResNet abstract: on the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers, 8 times deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task](https://ishwarj.com/img/papers/vit/p1-resnet-abstract.pngresnet-abstract.png)](/img/papers/vit/p1-resnet-abstract.png)
>
> **What it says:** residual networks "with a depth of up to 152 layers" reached "3.57% error on the ImageNet test set" and "won the 1st place on the ILSVRC 2015 classification task".
>
> **What it means:**
>
> ResNet, from 2015, is the CNN design that every vision baseline in the ViT paper is built on. The "ResNet-like architectures" of the paragraph are its descendants.
>
> So this is what ViT has to beat. The strongest baseline in the paper, Big Transfer (BiT), is a very large ResNet trained on the same giant datasets as ViT. Same data, different design: that makes the comparison fair.

> [!DEFINITION] ResNet
> A family of CNNs from 2015 built from "residual blocks", in which each block adds its output to its input, so that very deep stacks (50, 101, 152 layers) can be trained. ResNet-50 is the everyday workhorse; the ViT paper's baselines are ResNets with 152 layers made several times wider.

**"Combining CNN-like architectures with self-attention."** The second sentence lists what people had tried. Non-local networks (Wang et al., 2018) added an attention layer inside a CNN for video. DETR (Carion et al., 2020) put a Transformer on top of a CNN's output to detect objects. Stand-alone self-attention (Ramachandran et al., 2019) and Axial-DeepLab (Wang et al., 2020a) went further and replaced every convolution with a *local* form of attention. The abstract's two categories are these: attention *with* a CNN, and attention *replacing parts of* a CNN while "keeping their overall structure in place".

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="The three earlier ways of using attention in vision (attention added on top of a CNN, attention replacing some CNN layers, and pure local attention with special patterns) next to ViT, which is a standard Transformer with global attention over patch tokens."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="105.0" y="26.0" text-anchor="middle">1. CNN + attention</text><text class="t-muted" x="105.0" y="46.0" text-anchor="middle">Non-local nets (2018)</text><text class="t-muted" x="105.0" y="62.0" text-anchor="middle">DETR (2020)</text><rect class="box-1" x="45.0" y="220.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="236.0" text-anchor="middle">conv layer</text><rect class="box-1" x="45.0" y="190.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="206.0" text-anchor="middle">conv layer</text><rect class="box-1" x="45.0" y="160.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="176.0" text-anchor="middle">conv layer</text><rect class="box-1" x="45.0" y="130.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="146.0" text-anchor="middle">conv layer</text><rect class="box-2" x="45.0" y="100.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="116.0" text-anchor="middle">attention on top</text><rect class="box" x="45.0" y="260.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="105.0" y="276.0" text-anchor="middle">pixels</text><line class="edge" x1="105.0" y1="258.0" x2="105.0" y2="248.0" marker-end="url(#ah)"/><text class="t-tick" x="105.0" y="310.0" text-anchor="middle">the CNN still does the work</text><text class="t-title" x="290.0" y="26.0" text-anchor="middle">2. attention inside a CNN</text><text class="t-muted" x="290.0" y="46.0" text-anchor="middle">Stand-alone (2019)</text><text class="t-muted" x="290.0" y="62.0" text-anchor="middle">Axial-DeepLab (2020)</text><rect class="box-1" x="230.0" y="220.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="236.0" text-anchor="middle">conv layer</text><rect class="box-2" x="230.0" y="190.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="206.0" text-anchor="middle">local attention</text><rect class="box-1" x="230.0" y="160.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="176.0" text-anchor="middle">conv layer</text><rect class="box-2" x="230.0" y="130.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="146.0" text-anchor="middle">local attention</text><rect class="box-1" x="230.0" y="100.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="116.0" text-anchor="middle">conv layer</text><rect class="box" x="230.0" y="260.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="290.0" y="276.0" text-anchor="middle">pixels</text><line class="edge" x1="290.0" y1="258.0" x2="290.0" y2="248.0" marker-end="url(#ah)"/><text class="t-tick" x="290.0" y="310.0" text-anchor="middle">the CNN skeleton stays</text><text class="t-title" x="475.0" y="26.0" text-anchor="middle">3. pure local attention</text><text class="t-muted" x="475.0" y="46.0" text-anchor="middle">"specialized attention</text><text class="t-muted" x="475.0" y="62.0" text-anchor="middle">patterns", hard on TPUs</text><rect class="box-2" x="415.0" y="220.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="236.0" text-anchor="middle">local attention</text><rect class="box-2" x="415.0" y="190.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="206.0" text-anchor="middle">local attention</text><rect class="box-2" x="415.0" y="160.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="176.0" text-anchor="middle">local attention</text><rect class="box-2" x="415.0" y="130.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="146.0" text-anchor="middle">local attention</text><rect class="box-2" x="415.0" y="100.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="116.0" text-anchor="middle">local attention</text><rect class="box" x="415.0" y="260.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="475.0" y="276.0" text-anchor="middle">pixels</text><line class="edge" x1="475.0" y1="258.0" x2="475.0" y2="248.0" marker-end="url(#ah)"/><text class="t-tick" x="475.0" y="310.0" text-anchor="middle">efficient in theory, slow in practice</text><text class="t-on" x="660.0" y="26.0" text-anchor="middle">ViT: a standard Transformer</text><text class="t-muted" x="660.0" y="46.0" text-anchor="middle">global attention,</text><text class="t-muted" x="660.0" y="62.0" text-anchor="middle">patches as tokens</text><rect class="box-on" x="600.0" y="220.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="236.0" text-anchor="middle">Transformer layer</text><rect class="box-on" x="600.0" y="190.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="206.0" text-anchor="middle">Transformer layer</text><rect class="box-on" x="600.0" y="160.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="176.0" text-anchor="middle">Transformer layer</text><rect class="box-on" x="600.0" y="130.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="146.0" text-anchor="middle">Transformer layer</text><rect class="box-on" x="600.0" y="100.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="116.0" text-anchor="middle">Transformer layer</text><rect class="box" x="600.0" y="260.0" width="120.0" height="24.0" rx="5"/><text class="t-tick" x="660.0" y="276.0" text-anchor="middle">16 × 16 patches</text><line class="edge" x1="660.0" y1="258.0" x2="660.0" y2="248.0" marker-end="url(#ah)"/><text class="t-tick" x="660.0" y="310.0" text-anchor="middle">"fewest possible modifications"</text></svg><figcaption>The three earlier ways of using attention in vision, next to ViT. (1) A CNN does the work and attention is added on top (Non-local networks, DETR). (2) Some CNN layers are swapped for local attention, but the CNN skeleton stays (stand-alone self-attention, Axial-DeepLab). (3) Every layer is local attention with a special pattern, efficient in theory but hard to run fast. ViT is a standard Transformer with global attention over patch tokens: the "fewest possible modifications".</figcaption></figure>

**"Specialized attention patterns."** The third sentence is the technical heart of the paragraph. Why did the convolution-free models need special patterns at all? Because plain attention over pixels is impossibly expensive. The paper spells this out in its related work (Part 2): "each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes." Let us compute the cost, because the whole design of ViT follows from it.

> [!DEFINITION] Hardware accelerator (GPU, TPU)
> A chip built to do many multiplications at once, which is what neural networks need. GPUs (graphics processing units) came from video games; TPUs (tensor processing units) are Google's own chips. Both are fastest on big, regular blocks of arithmetic, such as multiplying two large matrices. Irregular "patterns" of attention, where each pixel looks at a hand-picked subset of others, do not map well onto them.

### The quadratic cost of attention over pixels

Self-attention computes, for every token, a score against every token. With $$n$$ tokens that is an $$n \times n$$ table of scores per attention head per layer. The [attention series](https://ishwarj.com/writings/attention-1-self-attention/) derives the formula; here it is with the shapes written in:

$$
\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right) V
$$

where:

- $$Q$$, $$K$$, $$V$$ are the queries, keys and values, three $$n \times d$$ tables made from the $$n$$ token vectors (for ViT-B, $$d = 64$$ per head);
- $$QK^\top$$ is the $$n \times n$$ table of scores: entry $$(i, j)$$ says how much token $$i$$ should look at token $$j$$;
- the softmax turns every row into weights that add up to 1;
- the result is $$n \times d$$: one new vector per token.

The expensive part is the $$n \times n$$ table. Its size is

$$
\text{attention weights per head per layer} = n^2, \qquad \text{memory in float32} = 4\,n^2 \ \text{bytes}
$$

> [!DEFINITION] Quadratic cost
> A cost that grows with the square of the input size. Doubling the number of tokens makes a quadratic cost four times larger; multiplying the tokens by 255 makes it about 65,000 times larger. Attention has a quadratic cost in the number of tokens.

We computed this table for several choices of what a token is ([`vit_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1_math.py), part (a)):

```python
for name, n in [('32 x 32 CIFAR pixels', 32 * 32), ('224 x 224 ImageNet pixels', 224 * 224),
                ('ViT-B/32 patches + [class] (7x7+1)', 7 * 7 + 1), ('ViT-B/16 patches + [class] (14x14+1)', 14 * 14 + 1),
                ('ViT-B/16 at 384 px (24x24+1)', 24 * 24 + 1)]:
    nn_ = n * n
    bytes_ = nn_ * 4
```

```text
    tokens are                                n            n x n  float32 memory
    32 x 32 CIFAR pixels                  1,024        1,048,576         4.19 MB
    224 x 224 ImageNet pixels            50,176    2,517,630,976        10.07 GB
    ViT-B/32 patches + [class] (7x7+1)       50            2,500         10.0 kB
    ViT-B/16 patches + [class] (14x14+1)      197           38,809        155.2 kB
    ViT-B/16 at 384 px (24x24+1)            577          332,929         1.33 MB
    pixels / patch tokens = 50,176 / 197 = 254.7 x more tokens, so 64,872 x more attention weights
    ViT-B/16 has 12 heads x 12 layers = 144 attention tables of 197 x 197 = 5,588,496 weights for one image
```

<figure class="fig"><svg viewBox="0 0 760 268" role="img" aria-label="Attention weights per head per layer on a log scale: 2,500 for ViT-B/32, 38,809 for ViT-B/16, about one million for 32 by 32 CIFAR pixels and 2.5 billion (10 GB in float32) for the 50,176 pixels of a 224 by 224 image."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="38.0" text-anchor="start">attention weights per head per layer = n × n (log scale)</text><text class="t-tick" x="262.0" y="70.0" text-anchor="end">ViT-B/32 patches (7x7+1)</text><g class="mark"><title>ViT-B/32 patches (7x7+1): 2,500 weights</title><rect class="s3" x="270.0" y="54.0" width="18.1" height="22.0" rx="3"/></g><text class="t-val" x="294.1" y="70.0" text-anchor="start">2,500   (10.0 kB)</text><text class="t-tick" x="262.0" y="102.0" text-anchor="end">ViT-B/16 patches (14x14+1)</text><g class="mark"><title>ViT-B/16 patches (14x14+1): 38,809 weights</title><rect class="s3" x="270.0" y="86.0" width="72.2" height="22.0" rx="3"/></g><text class="t-val" x="348.2" y="102.0" text-anchor="start">38,809   (155.2 kB)</text><text class="t-tick" x="262.0" y="134.0" text-anchor="end">ViT-B/16 at 384 px (24x24+1)</text><g class="mark"><title>ViT-B/16 at 384 px (24x24+1): 332,929 weights</title><rect class="s3" x="270.0" y="118.0" width="114.6" height="22.0" rx="3"/></g><text class="t-val" x="390.6" y="134.0" text-anchor="start">332,929   (1.33 MB)</text><text class="t-tick" x="262.0" y="166.0" text-anchor="end">32 x 32 CIFAR pixels</text><g class="mark"><title>32 x 32 CIFAR pixels: 1,048,576 weights</title><rect class="s1" x="270.0" y="150.0" width="137.3" height="22.0" rx="3"/></g><text class="t-val" x="413.3" y="166.0" text-anchor="start">1,048,576   (4.19 MB)</text><text class="t-tick" x="262.0" y="198.0" text-anchor="end">224 x 224 ImageNet pixels</text><g class="mark"><title>224 x 224 ImageNet pixels: 2,517,630,976 weights</title><rect class="s1" x="270.0" y="182.0" width="290.9" height="22.0" rx="3"/></g><text class="t-val" x="566.9" y="198.0" text-anchor="start">2,517,630,976   (10.07 GB)</text><line class="grid" x1="270.0" y1="48.0" x2="270.0" y2="210.0"/><text class="t-muted" x="270.0" y="226.0" text-anchor="middle">10^3</text><line class="grid" x1="315.4" y1="48.0" x2="315.4" y2="210.0"/><text class="t-muted" x="315.4" y="226.0" text-anchor="middle">10^4</text><line class="grid" x1="360.9" y1="48.0" x2="360.9" y2="210.0"/><text class="t-muted" x="360.9" y="226.0" text-anchor="middle">10^5</text><line class="grid" x1="406.3" y1="48.0" x2="406.3" y2="210.0"/><text class="t-muted" x="406.3" y="226.0" text-anchor="middle">10^6</text><line class="grid" x1="451.8" y1="48.0" x2="451.8" y2="210.0"/><text class="t-muted" x="451.8" y="226.0" text-anchor="middle">10^7</text><line class="grid" x1="497.2" y1="48.0" x2="497.2" y2="210.0"/><text class="t-muted" x="497.2" y="226.0" text-anchor="middle">10^8</text><line class="grid" x1="542.7" y1="48.0" x2="542.7" y2="210.0"/><text class="t-muted" x="542.7" y="226.0" text-anchor="middle">10^9</text><text class="t-note" x="20.0" y="252.0" text-anchor="start">pixels of a 224 × 224 image: 254.7 × more tokens than ViT-B/16, so 64,872 × more attention weights</text></svg><figcaption>Attention weights per head per layer, on a log scale. Over the 50,176 pixels of a 224×224 picture, one attention table has 2.5 billion entries (10 GB in float32). Over ViT-B/16's 197 patch tokens it has 38,809 entries (155 kB). Even the 1,024 pixels of a tiny CIFAR picture need a million entries.</figcaption></figure>

Read the second row. If every pixel of a 224×224 picture attended to every other pixel, one attention table of one head of one layer would hold $$50{,}176^2 \approx 2.5$$ billion numbers, about 10 GB. ViT-B has 144 such tables (12 heads × 12 layers), and training needs several copies of each. It cannot be done. Even for the tiny 32×32 pictures of CIFAR, one table has a million entries. That is why the earlier convolution-free models restricted each pixel to a local neighbourhood, and why those restrictions needed "specialized attention patterns" that ran poorly on accelerators.

Now read the fourth row. With 16×16 patches as tokens, $$n = 197$$ and one table has 38,809 entries. The ratio between the two is

$$
\frac{50{,}176^2}{197^2} = \left(\frac{50{,}176}{197}\right)^2 \approx 254.7^2 \approx 64{,}872
$$

Patches make attention about 65,000 times cheaper than pixels. That single fact is the engineering reason the title says "16×16 words" rather than "50,176 words". And 197 tokens is a very ordinary sequence length for a Transformer (BERT was trained with 512), so the standard, dense, accelerator-friendly attention can be used unchanged. The paper's related work (Part 2) credits Cordonnier et al. (2020) with the patch idea, using 2×2 patches on small pictures; the ViT authors use bigger patches, bigger pictures and far more data.

**"Classic ResNet-like architectures are still state of the art."** The paragraph closes with the three results that defined the top of the ImageNet leaderboard in 2020: Mahajan et al. (2018) pre-trained ResNeXt CNNs on 3.5 billion Instagram pictures; Xie et al. (2020), Noisy Student, used 300 million unlabelled pictures with a self-training loop; and Kolesnikov et al. (2020), Big Transfer (BiT), pre-trained very large ResNets on the same ImageNet-21k and JFT-300M datasets that ViT will use. All three are CNNs made better with *more data*, which is the same lever ViT pulls.

## The fewest possible modifications {§1}

> [!PAPER] · §1
> [![The third paragraph of the ViT introduction: inspired by the Transformer scaling successes in NLP, we experiment with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these patches as an input to a Transformer. Image patches are treated the same way as tokens (words) in an NLP application. We train the model on image classification in supervised fashion](https://ishwarj.com/img/papers/vit/p1-intro-ours.pngt/p1-intro-ours.png)](/img/papers/vit/p1-intro-ours.png)
>
> **What it says:** apply "a standard Transformer directly to images, with the fewest possible modifications". Split the image into patches, feed "the sequence of linear embeddings of these patches" to the Transformer. "Image patches are treated the same way as tokens (words) in an NLP application." Train "on image classification in supervised fashion".
>
> **What it means:**
>
> Imagine a test of whether a new fuel helps a car. You change only the fuel and keep the engine the same, so any change in speed comes from the fuel. "Fewest possible modifications" is the same kind of choice.
>
> By changing nothing but the input, the authors can link everything they find to the data and the scale, not to a clever new design. The choice also has a practical side. Every tool built for language Transformers, from training code to scaling tricks, works without change.

The four sentences are the whole method. We illustrate each.

**"Split an image into patches."** The picture is cut into a grid of equal squares and the squares are read off in order, row by row, like text.

<figure class="fig"><svg viewBox="0 0 760 200" role="img" aria-label="An image cut into a 3 by 3 grid of patches, then flattened row by row into a sequence of 9 patch tokens, numbered 1 to 9 in reading order."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-note" x="105.0" y="30.0" text-anchor="middle">an image, cut into a 3 × 3 grid</text><rect class="s1" x="31.5" y="51.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.30"/><rect class="s2" x="81.5" y="51.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.37"/><rect class="s3" x="131.5" y="51.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.44"/><rect class="s4" x="31.5" y="101.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.51"/><rect class="s1" x="81.5" y="101.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.58"/><rect class="s2" x="131.5" y="101.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.30"/><rect class="s3" x="31.5" y="151.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.37"/><rect class="s4" x="81.5" y="151.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.44"/><rect class="s1" x="131.5" y="151.5" width="47.0" height="47.0" rx="4" style="fill-opacity:0.51"/><text class="t-note" x="55.0" y="80.0" text-anchor="middle">1</text><text class="t-note" x="105.0" y="80.0" text-anchor="middle">2</text><text class="t-note" x="155.0" y="80.0" text-anchor="middle">3</text><text class="t-note" x="55.0" y="130.0" text-anchor="middle">4</text><text class="t-note" x="105.0" y="130.0" text-anchor="middle">5</text><text class="t-note" x="155.0" y="130.0" text-anchor="middle">6</text><text class="t-note" x="55.0" y="180.0" text-anchor="middle">7</text><text class="t-note" x="105.0" y="180.0" text-anchor="middle">8</text><text class="t-note" x="155.0" y="180.0" text-anchor="middle">9</text><line class="edge" x1="190.0" y1="125.0" x2="240.0" y2="125.0" marker-end="url(#ah)"/><text class="t-muted" x="215.0" y="112.0" text-anchor="middle">flatten</text><text class="t-muted" x="215.0" y="150.0" text-anchor="middle">row by row</text><rect class="box-1" x="250.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="274.0" y="129.5" text-anchor="middle">1</text><rect class="box-2" x="304.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="328.0" y="129.5" text-anchor="middle">2</text><rect class="box-3" x="358.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="382.0" y="129.5" text-anchor="middle">3</text><rect class="box-4" x="412.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="436.0" y="129.5" text-anchor="middle">4</text><rect class="box-1" x="466.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="490.0" y="129.5" text-anchor="middle">5</text><rect class="box-2" x="520.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="544.0" y="129.5" text-anchor="middle">6</text><rect class="box-3" x="574.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="598.0" y="129.5" text-anchor="middle">7</text><rect class="box-4" x="628.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="652.0" y="129.5" text-anchor="middle">8</text><rect class="box-1" x="682.0" y="100.0" width="48.0" height="50.0" rx="6"/><text class="t-note" x="706.0" y="129.5" text-anchor="middle">9</text><path class="edge" d="M250.0,154.0 L250.0,162.0 L730.0,162.0 L730.0,154.0"/><text class="t-tick" x="490.0" y="182.0" text-anchor="middle">a sequence of 9 patch tokens, in reading order (left to right, top to bottom)</text><text class="t-muted" x="250.0" y="44.0" text-anchor="start">patch 1 is the top-left corner, patch 9 the bottom-right; the order is fixed</text><text class="t-muted" x="250.0" y="62.0" text-anchor="start">a position embedding (Part 2) tells the model where each patch came from</text></svg><figcaption>An image cut into a 3×3 grid of patches and flattened, row by row, into a sequence of nine patch tokens. Patch 1 is the top-left corner and patch 9 the bottom-right. The order is fixed; in Part 2 a position embedding tells the model where each patch came from.</figcaption></figure>

**"The sequence of linear embeddings of these patches."** Each patch is 16 × 16 × 3 = 768 raw pixel numbers. A "linear embedding" multiplies those 768 numbers by a learned matrix to give the patch's token vector. In the paper's notation (Equation 1, which Part 2 covers in full) a flattened patch $$\mathbf{x}_p^i$$ becomes

$$
\mathbf{x}_p^i \mathbf{E}, \qquad \mathbf{x}_p^i \in \mathbb{R}^{1 \times 768}, \quad \mathbf{E} \in \mathbb{R}^{768 \times D}
$$

where $$D$$ is the Transformer's width ($$D = 768$$ for ViT-B, so $$\mathbf{E}$$ happens to be square, 768 × 768). "Linear" means no non-linearity: it is one matrix multiplication, the simplest possible way to turn a patch into a vector. The output has the same shape as a word embedding in BERT, which is the point.

<figure class="fig"><svg viewBox="0 0 760 250" role="img" aria-label="One 16 by 16 patch with 3 colour channels is flattened into 768 numbers and multiplied by a learned matrix E to give a 768-number patch token, the same shape as a word vector in BERT."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">one patch, 16 × 16 pixels × 3 channels</text><rect class="s1" x="30.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="36.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="42.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="48.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="54.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="60.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="66.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="72.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="78.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="84.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="90.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="96.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="102.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="108.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="114.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="120.5" y="44.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="30.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="36.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="42.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="48.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="54.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="60.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="66.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="72.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="78.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="84.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="90.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="96.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="102.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="108.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="114.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="120.5" y="50.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="30.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="36.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="42.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="48.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="54.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="60.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="66.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="72.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="78.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="84.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="90.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="96.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="102.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="108.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="114.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="120.5" y="56.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="30.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="36.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="42.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="48.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="54.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="60.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="66.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="72.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="78.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="84.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="90.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="96.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="102.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="108.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="114.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="120.5" y="62.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="30.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="36.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="42.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="48.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="54.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="60.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="66.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="72.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="78.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="84.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="90.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="96.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="102.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="108.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="114.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="120.5" y="68.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="30.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="36.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="42.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="48.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="54.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="60.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="66.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="72.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="78.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="84.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="90.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="96.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="102.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="108.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="114.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="120.5" y="74.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="30.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="36.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="42.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="48.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="54.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="60.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="66.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="72.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="78.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="84.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="90.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="96.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="102.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="108.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="114.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="120.5" y="80.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="30.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="36.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="42.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="48.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="54.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="60.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="66.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="72.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="78.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="84.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="90.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="96.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="102.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="108.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="114.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="120.5" y="86.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="30.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="36.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="42.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="48.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="54.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="60.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="66.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="72.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="78.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="84.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="90.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="96.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="102.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="108.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="114.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="120.5" y="92.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="30.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="36.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="42.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="48.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="54.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="60.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="66.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="72.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="78.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="84.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="90.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="96.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="102.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="108.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="114.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="120.5" y="98.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="30.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="36.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="42.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="48.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="54.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="60.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="66.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="72.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="78.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="84.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="90.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="96.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="102.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="108.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="114.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="120.5" y="104.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="30.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="36.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="42.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="48.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="54.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="60.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="66.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="72.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="78.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="84.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="90.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="96.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="102.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="108.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="114.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="120.5" y="110.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="30.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="36.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="42.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="48.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="54.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="60.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="66.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="72.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="78.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="84.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="90.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="96.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="102.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="108.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="114.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="120.5" y="116.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="30.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="36.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="42.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="48.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="54.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="60.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="66.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="72.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="78.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="84.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="90.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="96.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="102.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="108.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="114.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="120.5" y="122.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="30.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="36.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="42.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="48.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="54.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="60.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="66.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="72.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="78.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="84.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="90.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="96.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="102.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="108.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="114.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="120.5" y="128.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="30.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="36.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="42.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="48.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="54.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s1" x="60.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s1" x="66.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s1" x="72.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s1" x="78.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s1" x="84.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s1" x="90.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s1" x="96.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s1" x="102.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s1" x="108.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s1" x="114.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s1" x="120.5" y="134.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="40.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="46.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="52.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="58.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="64.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="70.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="76.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="82.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="88.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="94.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="100.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="106.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="112.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="118.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="124.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="130.5" y="54.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="40.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="46.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="52.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="58.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="64.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="70.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="76.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="82.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="88.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="94.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="100.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="106.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="112.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="118.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="124.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="130.5" y="60.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="40.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="46.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="52.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="58.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="64.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="70.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="76.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="82.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="88.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="94.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="100.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="106.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="112.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="118.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="124.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="130.5" y="66.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="40.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="46.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="52.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="58.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="64.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="70.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="76.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="82.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="88.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="94.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="100.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="106.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="112.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="118.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="124.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="130.5" y="72.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="40.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="46.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="52.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="58.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="64.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="70.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="76.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="82.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="88.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="94.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="100.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="106.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="112.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="118.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="124.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="130.5" y="78.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="40.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="46.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="52.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="58.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="64.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="70.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="76.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="82.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="88.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="94.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="100.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="106.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="112.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="118.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="124.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="130.5" y="84.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="40.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="46.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="52.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="58.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="64.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="70.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="76.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="82.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="88.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="94.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="100.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="106.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="112.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="118.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="124.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="130.5" y="90.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="40.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="46.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="52.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="58.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="64.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="70.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="76.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="82.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="88.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="94.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="100.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="106.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="112.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="118.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="124.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="130.5" y="96.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="40.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="46.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="52.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="58.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="64.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="70.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="76.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="82.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="88.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="94.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="100.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="106.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="112.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="118.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="124.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="130.5" y="102.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="40.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="46.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="52.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="58.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="64.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="70.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="76.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="82.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="88.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="94.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="100.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="106.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="112.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="118.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="124.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="130.5" y="108.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="40.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="46.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="52.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="58.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="64.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="70.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="76.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="82.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="88.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="94.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="100.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="106.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="112.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="118.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="124.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="130.5" y="114.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="40.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="46.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="52.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="58.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="64.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="70.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="76.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="82.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="88.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="94.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="100.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="106.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="112.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="118.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="124.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="130.5" y="120.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="40.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="46.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="52.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="58.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="64.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="70.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="76.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="82.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="88.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="94.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="100.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="106.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="112.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="118.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="124.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="130.5" y="126.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="40.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="46.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="52.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="58.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="64.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="70.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="76.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="82.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="88.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="94.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="100.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="106.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="112.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="118.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="124.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="130.5" y="132.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="40.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="46.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="52.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="58.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="64.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="70.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="76.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="82.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="88.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="94.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="100.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="106.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="112.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="118.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="124.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="130.5" y="138.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="40.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="46.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="52.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="58.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="64.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s2" x="70.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s2" x="76.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s2" x="82.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s2" x="88.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s2" x="94.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s2" x="100.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s2" x="106.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s2" x="112.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s2" x="118.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s2" x="124.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s2" x="130.5" y="144.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="50.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="56.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="62.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="68.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="74.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="80.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="86.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="92.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="98.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="104.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="110.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="116.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="122.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="128.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="134.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="140.5" y="64.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="50.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="56.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="62.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="68.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="74.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="80.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="86.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="92.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="98.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="104.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="110.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="116.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="122.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="128.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="134.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="140.5" y="70.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="50.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="56.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="62.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="68.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="74.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="80.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="86.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="92.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="98.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="104.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="110.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="116.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="122.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="128.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="134.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="140.5" y="76.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="50.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="56.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="62.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="68.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="74.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="80.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="86.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="92.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="98.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="104.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="110.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="116.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="122.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="128.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="134.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="140.5" y="82.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="50.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="56.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="62.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="68.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="74.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="80.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="86.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="92.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="98.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="104.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="110.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="116.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="122.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="128.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="134.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="140.5" y="88.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="50.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="56.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="62.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="68.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="74.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="80.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="86.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="92.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="98.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="104.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="110.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="116.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="122.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="128.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="134.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="140.5" y="94.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="50.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="56.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="62.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="68.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="74.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="80.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="86.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="92.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="98.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="104.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="110.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="116.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="122.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="128.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="134.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="140.5" y="100.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="50.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="56.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="62.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="68.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="74.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="80.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="86.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="92.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="98.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="104.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="110.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="116.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="122.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="128.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="134.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="140.5" y="106.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="50.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="56.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="62.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="68.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="74.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="80.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="86.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="92.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="98.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="104.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="110.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="116.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="122.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="128.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="134.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="140.5" y="112.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="50.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="56.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="62.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="68.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="74.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="80.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="86.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="92.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="98.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="104.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="110.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="116.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="122.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="128.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="134.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="140.5" y="118.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="50.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="56.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="62.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="68.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="74.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="80.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="86.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="92.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="98.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="104.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="110.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="116.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="122.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="128.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="134.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="140.5" y="124.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="50.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="56.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="62.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="68.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="74.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="80.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="86.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="92.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="98.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="104.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="110.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="116.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="122.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="128.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="134.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="140.5" y="130.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="50.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="56.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="62.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="68.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="74.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="80.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="86.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="92.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="98.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="104.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="110.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="116.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="122.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="128.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="134.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="140.5" y="136.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="50.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="56.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="62.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="68.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="74.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="80.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="86.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="92.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="98.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="104.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="110.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="116.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="122.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="128.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="134.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="140.5" y="142.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="50.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="56.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="62.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="68.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="74.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="80.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="86.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="92.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="98.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="104.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="110.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="116.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="122.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="128.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="134.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="140.5" y="148.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="50.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="56.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="62.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="68.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="74.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><rect class="s3" x="80.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.45"/><rect class="s3" x="86.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.70"/><rect class="s3" x="92.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.40"/><rect class="s3" x="98.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.65"/><rect class="s3" x="104.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.35"/><rect class="s3" x="110.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.60"/><rect class="s3" x="116.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.30"/><rect class="s3" x="122.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.55"/><rect class="s3" x="128.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.25"/><rect class="s3" x="134.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.50"/><rect class="s3" x="140.5" y="154.5" width="5.0" height="5.0" rx="1" style="fill-opacity:0.75"/><text class="t-muted" x="88.0" y="184.0" text-anchor="middle">red, green, blue</text><line class="edge" x1="158.0" y1="104.0" x2="190.0" y2="104.0" marker-end="url(#ah)"/><text class="t-muted" x="174.0" y="92.0" text-anchor="middle">flatten</text><rect class="box-ghost" x="217.0" y="95.0" width="356.0" height="15.0" rx="4"/><rect class="s1" x="220.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="231.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="242.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="253.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="264.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="275.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="286.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="297.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="308.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="319.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="330.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="341.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="352.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="363.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="374.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="385.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="396.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="407.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="418.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="429.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="440.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="451.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="462.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="473.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="484.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="495.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="506.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="517.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s1" x="528.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s1" x="539.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s1" x="550.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="561.0" y="98.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><text class="t-note" x="578.0" y="106.0" text-anchor="start">…</text><path class="edge" d="M220.0,112.0 L220.0,120.0 L590.0,120.0 L590.0,112.0"/><text class="t-tick" x="405.0" y="140.0" text-anchor="middle">768 numbers in a row (16 × 16 × 3)</text><line class="edge" x1="395.0" y1="154.0" x2="395.0" y2="180.0" marker-end="url(#ah)"/><text class="t-tick" x="405.0" y="172.0" text-anchor="start">multiply by a learned matrix E (768 × 768): the "linear embedding"</text><rect class="box-ghost" x="217.0" y="183.0" width="356.0" height="15.0" rx="4"/><rect class="s3" x="220.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="231.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="242.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="253.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="264.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="275.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="286.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="297.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="308.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="319.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="330.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="341.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="352.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="363.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="374.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="385.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="396.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="407.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="418.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="429.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="440.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="451.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="462.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="473.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="484.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="495.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="506.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><rect class="s3" x="517.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.85"/><rect class="s3" x="528.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.47"/><rect class="s3" x="539.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.72"/><rect class="s3" x="550.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.35"/><rect class="s3" x="561.0" y="186.0" width="9" height="9" rx="2" style="fill-opacity:0.60"/><text class="t-note" x="578.0" y="194.0" text-anchor="start">…</text><path class="edge" d="M220.0,200.0 L220.0,208.0 L590.0,208.0 L590.0,200.0"/><text class="t-tick" x="405.0" y="228.0" text-anchor="middle">one patch token: a vector of D = 768 numbers, the same shape as a word vector in BERT</text></svg><figcaption>One patch becomes one token. The 16×16×3 block of pixels is flattened into a row of 768 numbers, multiplied by a learned 768×768 matrix E (the "linear embedding"), and comes out as a vector of 768 numbers, the same shape as a word vector in BERT.</figcaption></figure>

Our code found this layer in the released model. In the implementation it is stored as a convolution with a 16×16 filter and a stride of 16, which is exactly the same computation as "cut into 16×16 patches and multiply each by $$\mathbf{E}$$", because a stride-16 filter of size 16 touches each pixel exactly once:

```text
    patch embedding layer: Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))
    patch vectors: (196, 768) = 196 patches x 768 numbers, arranged on a 14 x 14 grid
```

**"Treated the same way as tokens (words)."** After the embedding, the Transformer sees 196 vectors of 768 numbers. It would see the same thing for a 196-word sentence in BERT. Nothing in the attention layers, the feed-forward layers or the normalisation knows about pixels, rows or columns.

**"In supervised fashion."** Pre-training is ordinary classification with labels, on a very large labelled set. BERT's label-free pre-training was its big trick; ViT does not need a trick, because Google had hundreds of millions of labelled pictures. (Part 5 shows a first attempt at label-free pre-training for ViT, and Part 6 shows how later papers made it work.)

## Mid-sized data, and the inductive-bias explanation {§1}

The fourth paragraph reports the first, disappointing result, and explains it.

> [!PAPER] · §1
> [![The fourth paragraph of the ViT introduction, first part: when trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases](https://ishwarj.com/img/papers/vit/p1-intro-midsize.png1-intro-midsize.png)](/img/papers/vit/p1-intro-midsize.png)
> [![The paragraph continues on page 2: inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data](https://ishwarj.com/img/papers/vit/p1-intro-midsize2.png-intro-midsize2.png)](/img/papers/vit/p1-intro-midsize2.png)
>
> **What it says:** trained on "mid-sized datasets such as ImageNet without strong regularization", ViT models reach "modest accuracies of a few percentage points below ResNets of comparable size". The authors call this "seemingly discouraging" but expected: "Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data."
>
> **What it means:**
>
> This is the first honest result of the paper, and it is a loss. On ImageNet alone, with no strong tricks against memorising, ViT is a few points worse than a ResNet of the same size.
>
> The authors expected it, and their reason is the centre of the paper. A CNN comes with built-in assumptions about pictures; a Transformer does not. Think of two students: one gets a summary sheet before the test, the other must work everything out from examples. With few examples, the summary sheet wins. With many examples, the story changes.

Four technical words appear here, and the rest of the series uses all of them, so we take them slowly.

> [!DEFINITION] Regularization
> Any technique that stops a model from memorising its training set instead of learning general rules: adding noise, randomly dropping parts of the network during training (dropout), shrinking the weights (weight decay), or distorting the training pictures. "Without strong regularization" means the authors did not lean hard on such tricks in this first comparison. Part 4 shows that heavy regularization helps ViT on small data.

> [!DEFINITION] Inductive bias
> The built-in assumptions of a model design, the things it "believes" about the data before seeing any. A CNN assumes that nearby pixels belong together and that an object looks the same wherever it appears. These assumptions let it learn from fewer examples, but they also limit what it can learn. A Transformer has far fewer such assumptions: it must learn from data even that neighbouring patches are related.

> [!DEFINITION] Locality
> The assumption that what matters for a pixel is mostly its neighbours. A convolution with a 3×3 filter builds this in: each output depends on 9 nearby pixels and nothing else. Far-away parts of the picture only meet after many layers. Attention has no such rule: in layer 1, any patch can look at any patch.

The ResNet-50 run above showed locality in action: its feature maps shrink from 56×56 to 7×7 stage by stage, which is how a CNN gradually lets far-away pixels meet. ViT's 197 tokens all meet in the first layer.

> [!DEFINITION] Translation equivariance
> "Translation" here means *shifting*: moving the picture (or an object in it) left, right, up or down. A function is **equivariant** to shifts if shifting the input shifts the output by the same amount, and changes nothing else. A convolution is equivariant: if the cat moves 2 pixels right, the "cat features" move 2 pixels right. This is different from **invariance**, which means the output does not change at all when the input shifts. A classifier should be *invariant* (a cat is a cat wherever it sits); the layers inside a CNN are *equivariant*, and a final pooling step turns equivariant features into an invariant answer.

Let us write these two properties down, because they are often confused. A picture is a function $$x[i, j]$$ of its row $$i$$ and column $$j$$. A two-dimensional convolution with a filter $$w$$ (here 3×3) computes

$$
(x * w)[i, j] = \sum_{u=0}^{2} \sum_{v=0}^{2} w[u, v] \, x[i + u, \, j + v]
$$

where:

- $$x[i + u, j + v]$$ are the 9 pixels under the filter when its top-left corner sits at $$(i, j)$$;
- $$w[u, v]$$ are the 9 filter weights, the same at every position (that is **weight sharing**);
- the sum is one output number; sliding over all $$(i, j)$$ gives the filtered picture.

Define the **shift** operator $$T_{t}$$ that moves a picture $$t$$ pixels to the right:

$$
(T_{t}\, x)[i, j] = x[i, \, j - t]
$$

Then equivariance and invariance of a function $$f$$ are

$$
\text{equivariance:}\quad f(T_{t}\, x) = T_{t}\, f(x)
\qquad\qquad
\text{invariance:}\quad f(T_{t}\, x) = f(x)
$$

In words: an equivariant $$f$$ lets the shift pass through (shift then filter equals filter then shift); an invariant $$f$$ swallows the shift. A convolution is equivariant because the same weights $$w$$ are used at every position: the sum at output position $$(i, j + t)$$ of the shifted picture sees exactly the pixels that the sum at $$(i, j)$$ saw in the original.

**Equivariance with real numbers.** We built a 12×12 toy picture with a bright 5×5 square in it, applied a 3×3 vertical-edge filter, then shifted the picture 2 pixels right and applied the same filter ([`vit_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1_math.py), part (b)):

```python
img = torch.zeros(1, 1, 12, 12)
img[0, 0, 3:8, 2:7] = 1.0                                           # a bright 5 x 5 square at rows 3..7, columns 2..6
k = torch.tensor([[-1., 0., 1.], [-2., 0., 2.], [-1., 0., 1.]]).view(1, 1, 3, 3)   # a 3 x 3 vertical-edge filter (Sobel)
shift = 2
shifted = torch.zeros_like(img)
shifted[..., shift:] = img[..., :-shift]                             # the same square, 2 pixels to the right
out = Fn.conv2d(img, k)                                              # 10 x 10 output (no padding)
out_s = Fn.conv2d(shifted, k)
diff = (out_s[..., shift:] - out[..., :-shift]).abs().max().item()   # move the second output back by 2 and compare
```

```text
    toy image (12, 12), bright square at rows 3-7, columns 2-6; filter (3, 3) (vertical edges)
    filter weights: [[-1.0, 0.0, 1.0], [-2.0, 0.0, 2.0], [-1.0, 0.0, 1.0]]
    output (10, 10); strongest response in the original at column 0, in the shifted at column 2
    max |shifted output moved back by 2 - original output| = 0.0
    row 5 of the output, original:    4    4    0    0    0   -4   -4    0    0    0
    row 5 of the output, shifted:     0    0    4    4    0    0    0   -4   -4    0
    the same filter weights (9 numbers) are used at every one of the 100 output positions (weight sharing)
```

Read the two printed rows. The filter fires +4 where the square's left edge is (dark to bright) and -4 where its right edge is (bright to dark). In the shifted picture the very same values appear, two columns to the right. Moving the second output back by two columns and subtracting the first gives a maximum difference of exactly 0.0. That is equivariance: *the feature moved with the object, nothing else changed.* The filter never had to learn anything about "the square at column 4" versus "the square at column 6"; one set of 9 weights handles every position.

<figure class="fig"><svg viewBox="0 0 760 280" role="img" aria-label="Translation equivariance made concrete: a vertical-edge filter on a bright square finds its two edges; on the same square moved two pixels to the right it finds the same two edges, moved two pixels to the right. The maximum difference after shifting back is exactly zero."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="380.0" y="22.0" text-anchor="middle">the same 3 × 3 edge filter, applied to a square and to the square moved 2 pixels right</text><rect class="box-ghost" x="28.0" y="58.0" width="148.0" height="148.0" rx="3"/><text class="t-tick" x="102.0" y="50.0" text-anchor="middle">input (12 × 12)</text><g class="mark"><title>row 3, column 2: 1</title><rect class="cell" x="54.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 3: 1</title><rect class="cell" x="66.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 4: 1</title><rect class="cell" x="78.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 5: 1</title><rect class="cell" x="90.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 6: 1</title><rect class="cell" x="102.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 2: 1</title><rect class="cell" x="54.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 3: 1</title><rect class="cell" x="66.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 4: 1</title><rect class="cell" x="78.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 5: 1</title><rect class="cell" x="90.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 6: 1</title><rect class="cell" x="102.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 2: 1</title><rect class="cell" x="54.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 3: 1</title><rect class="cell" x="66.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 4: 1</title><rect class="cell" x="78.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 5: 1</title><rect class="cell" x="90.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 6: 1</title><rect class="cell" x="102.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 2: 1</title><rect class="cell" x="54.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 3: 1</title><rect class="cell" x="66.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 4: 1</title><rect class="cell" x="78.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 5: 1</title><rect class="cell" x="90.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 6: 1</title><rect class="cell" x="102.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 2: 1</title><rect class="cell" x="54.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 3: 1</title><rect class="cell" x="66.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 4: 1</title><rect class="cell" x="78.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 5: 1</title><rect class="cell" x="90.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 6: 1</title><rect class="cell" x="102.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><line class="edge" x1="180.0" y1="132.0" x2="208.0" y2="132.0" marker-end="url(#ah)"/><text class="t-muted" x="194.0" y="122.0" text-anchor="middle">conv</text><rect class="box-ghost" x="212.0" y="70.0" width="124.0" height="124.0" rx="3"/><text class="t-tick" x="274.0" y="62.0" text-anchor="middle">output (10 × 10)</text><g class="mark"><title>row 1, column 0: 1</title><rect class="cell" x="214.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 1: 1</title><rect class="cell" x="226.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 5: -1</title><rect class="s2" x="274.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 6: -1</title><rect class="s2" x="286.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 2, column 0: 3</title><rect class="cell" x="214.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 1: 3</title><rect class="cell" x="226.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 5: -3</title><rect class="s2" x="274.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 6: -3</title><rect class="s2" x="286.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 3, column 0: 4</title><rect class="cell" x="214.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 1: 4</title><rect class="cell" x="226.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 5: -4</title><rect class="s2" x="274.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 6: -4</title><rect class="s2" x="286.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 0: 4</title><rect class="cell" x="214.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 1: 4</title><rect class="cell" x="226.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 5: -4</title><rect class="s2" x="274.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 6: -4</title><rect class="s2" x="286.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 0: 4</title><rect class="cell" x="214.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 1: 4</title><rect class="cell" x="226.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 5: -4</title><rect class="s2" x="274.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 6: -4</title><rect class="s2" x="286.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 0: 3</title><rect class="cell" x="214.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 1: 3</title><rect class="cell" x="226.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 5: -3</title><rect class="s2" x="274.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 6: -3</title><rect class="s2" x="286.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 7, column 0: 1</title><rect class="cell" x="214.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 1: 1</title><rect class="cell" x="226.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 5: -1</title><rect class="s2" x="274.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 6: -1</title><rect class="s2" x="286.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><rect class="box-ghost" x="376.0" y="58.0" width="148.0" height="148.0" rx="3"/><text class="t-tick" x="450.0" y="50.0" text-anchor="middle">input shifted 2 px →</text><g class="mark"><title>row 3, column 4: 1</title><rect class="cell" x="426.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 5: 1</title><rect class="cell" x="438.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 6: 1</title><rect class="cell" x="450.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 7: 1</title><rect class="cell" x="462.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 8: 1</title><rect class="cell" x="474.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 4: 1</title><rect class="cell" x="426.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 5: 1</title><rect class="cell" x="438.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 6: 1</title><rect class="cell" x="450.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 7: 1</title><rect class="cell" x="462.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 8: 1</title><rect class="cell" x="474.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 4: 1</title><rect class="cell" x="426.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 5: 1</title><rect class="cell" x="438.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 6: 1</title><rect class="cell" x="450.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 7: 1</title><rect class="cell" x="462.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 8: 1</title><rect class="cell" x="474.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 4: 1</title><rect class="cell" x="426.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 5: 1</title><rect class="cell" x="438.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 6: 1</title><rect class="cell" x="450.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 7: 1</title><rect class="cell" x="462.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 8: 1</title><rect class="cell" x="474.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 4: 1</title><rect class="cell" x="426.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 5: 1</title><rect class="cell" x="438.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 6: 1</title><rect class="cell" x="450.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 7: 1</title><rect class="cell" x="462.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 7, column 8: 1</title><rect class="cell" x="474.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><line class="edge" x1="528.0" y1="132.0" x2="556.0" y2="132.0" marker-end="url(#ah)"/><text class="t-muted" x="542.0" y="122.0" text-anchor="middle">conv</text><rect class="box-ghost" x="560.0" y="70.0" width="124.0" height="124.0" rx="3"/><text class="t-tick" x="622.0" y="62.0" text-anchor="middle">output shifted 2 px →</text><g class="mark"><title>row 1, column 2: 1</title><rect class="cell" x="586.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 3: 1</title><rect class="cell" x="598.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 7: -1</title><rect class="s2" x="646.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 1, column 8: -1</title><rect class="s2" x="658.5" y="84.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 2, column 2: 3</title><rect class="cell" x="586.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 3: 3</title><rect class="cell" x="598.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 7: -3</title><rect class="s2" x="646.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 2, column 8: -3</title><rect class="s2" x="658.5" y="96.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 3, column 2: 4</title><rect class="cell" x="586.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 3: 4</title><rect class="cell" x="598.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 7: -4</title><rect class="s2" x="646.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 3, column 8: -4</title><rect class="s2" x="658.5" y="108.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 2: 4</title><rect class="cell" x="586.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 3: 4</title><rect class="cell" x="598.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 7: -4</title><rect class="s2" x="646.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 4, column 8: -4</title><rect class="s2" x="658.5" y="120.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 2: 4</title><rect class="cell" x="586.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 3: 4</title><rect class="cell" x="598.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 7: -4</title><rect class="s2" x="646.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 5, column 8: -4</title><rect class="s2" x="658.5" y="132.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.95"/></g><g class="mark"><title>row 6, column 2: 3</title><rect class="cell" x="586.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 3: 3</title><rect class="cell" x="598.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 7: -3</title><rect class="s2" x="646.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 6, column 8: -3</title><rect class="s2" x="658.5" y="144.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.77"/></g><g class="mark"><title>row 7, column 2: 1</title><rect class="cell" x="586.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 3: 1</title><rect class="cell" x="598.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 7: -1</title><rect class="s2" x="646.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><g class="mark"><title>row 7, column 8: -1</title><rect class="s2" x="658.5" y="156.5" width="11.0" height="11.0" rx="1.5" style="fill-opacity:0.42"/></g><rect class="cell" x="30" y="232" width="10" height="10" rx="2" style="fill-opacity:0.9"/><text class="t-tick" x="46.0" y="241.0" text-anchor="start">positive response (dark-to-bright edge)</text><rect class="s2" x="330" y="232" width="10" height="10" rx="2" style="fill-opacity:0.9"/><text class="t-tick" x="346.0" y="241.0" text-anchor="start">negative response (bright-to-dark edge)</text><text class="t-note" x="30.0" y="266.0" text-anchor="start">max |shifted output moved back by 2 - original output| = 0.0: the feature moved with the object, nothing else changed</text></svg><figcaption>Translation equivariance made concrete. Left pair: the toy picture and the edge filter's output, which marks the square's left edge (positive) and right edge (negative). Right pair: the same picture moved 2 pixels to the right, and the output, which has moved by exactly 2 pixels and is otherwise identical. The maximum difference after shifting back is 0.0.</figcaption></figure>

This is why a CNN can learn from fewer examples. A cat in the top-left corner teaches the same filters as a cat in the bottom-right. The network does not need to see cats everywhere; the design guarantees that what it learns in one place applies in every place.

### Patch tokens are not shift-equivariant

Now the other side. ViT's first step cuts the picture along a fixed 16-pixel grid. Shift the picture by less than 16 pixels and every patch now contains a different mix of pixels, so every patch vector changes. Shift by exactly 16 and each patch's content moves, whole, into the neighbouring slot. We measured this on the real cat picture and the real ViT-B/16 patch-embedding layer, using cosine similarity to compare vectors:

$$
\cos(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}
$$

where $$\mathbf{a}$$ and $$\mathbf{b}$$ are two 768-number patch vectors, the dot is their dot product and $$\lVert \cdot \rVert$$ is the length. The cosine is 1 for identical directions, 0 for unrelated ones, and negative for opposite ones. For each shift $$k$$ we compared patch $$(i, j)$$ of the original with patch $$(i, j + \lfloor k/16 \rfloor)$$ of the shifted picture, that is, we move the token grid back by the whole number of patches the picture moved (0 for shifts under 16, 1 for a shift of 16):

```python
embed = vit.vit.embeddings.patch_embeddings                          # one 16 x 16 convolution with stride 16 = the linear patch projection
with torch.no_grad():
    ek = embed(shift_right(x, k))[0].view(14, 14, -1)
back = k // P                                                    # whole patches the picture moved
a, b = grid[:, :14 - back], ek[:, back:]                        # compare patch (i, j) of the original with patch (i, j + back) of the shifted
cos = Fn.cosine_similarity(a.reshape(-1, 768), b.reshape(-1, 768), dim=-1)
```

```text
     shift k  grid moved back  mean cosine  min cosine   ViT-B/16 top-1             ResNet-50 top-1
           0                0        1.000       1.000   Egyptian cat 0.937         tiger cat 0.942
           1                0        0.935       0.643   Egyptian cat 0.888         tiger cat 0.939
           4                0        0.649      -0.219   Egyptian cat 0.861         tiger cat 0.847
           8                0        0.507      -0.353   Egyptian cat 0.939         tiger cat 0.826
          12                0        0.457      -0.446   Egyptian cat 0.847         tiger cat 0.607
          16                1        1.000       1.000   Egyptian cat 0.908         tiger cat 0.737
    for k = 16 compared at the SAME grid position (no moving back): mean cosine = 0.428  (each slot now holds its left neighbour)
```

<figure class="fig"><svg viewBox="0 0 760 320" role="img" aria-label="Shifting the cat picture right by 1, 4, 8, 12 and 16 pixels: the mean cosine similarity between the original and shifted ViT-B/16 patch vectors drops to 0.457 at 12 pixels and returns to 1.000 at 16 pixels, a whole patch; meanwhile both ViT-B/16 and ResNet-50 keep their top-1 class, with probabilities that move only a little."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="64" y1="270.0" x2="520" y2="270.0"/><text class="t-tick" x="56.0" y="274.0" text-anchor="end">0.400</text><line class="grid" x1="64" y1="230.0" x2="520" y2="230.0"/><text class="t-tick" x="56.0" y="234.0" text-anchor="end">0.500</text><line class="grid" x1="64" y1="190.0" x2="520" y2="190.0"/><text class="t-tick" x="56.0" y="194.0" text-anchor="end">0.600</text><line class="grid" x1="64" y1="150.0" x2="520" y2="150.0"/><text class="t-tick" x="56.0" y="154.0" text-anchor="end">0.700</text><line class="grid" x1="64" y1="110.0" x2="520" y2="110.0"/><text class="t-tick" x="56.0" y="114.0" text-anchor="end">0.800</text><line class="grid" x1="64" y1="70.0" x2="520" y2="70.0"/><text class="t-tick" x="56.0" y="74.0" text-anchor="end">0.900</text><line class="grid" x1="64" y1="30.0" x2="520" y2="30.0"/><text class="t-tick" x="56.0" y="34.0" text-anchor="end">1.000</text><text class="t-tick" x="64.0" y="290.0" text-anchor="middle">0</text><text class="t-tick" x="92.5" y="290.0" text-anchor="middle">1</text><text class="t-tick" x="178.0" y="290.0" text-anchor="middle">4</text><text class="t-tick" x="292.0" y="290.0" text-anchor="middle">8</text><text class="t-tick" x="406.0" y="290.0" text-anchor="middle">12</text><text class="t-tick" x="520.0" y="290.0" text-anchor="middle">16</text><line class="axis" x1="64" y1="270" x2="520" y2="270"/><polyline class="l3" points="64.0,30.0 92.5,56.0 178.0,170.3 292.0,227.2 406.0,247.4 520.0,30.0"/><g class="mark"><title>patch cosine, ViT, 0: 1.000</title><circle class="s3 ring" cx="64.0" cy="30.0" r="5"/></g><g class="mark"><title>patch cosine, ViT, 1: 0.935</title><circle class="s3 ring" cx="92.5" cy="56.0" r="5"/></g><g class="mark"><title>patch cosine, ViT, 4: 0.649</title><circle class="s3 ring" cx="178.0" cy="170.3" r="5"/></g><g class="mark"><title>patch cosine, ViT, 8: 0.507</title><circle class="s3 ring" cx="292.0" cy="227.2" r="5"/></g><g class="mark"><title>patch cosine, ViT, 12: 0.457</title><circle class="s3 ring" cx="406.0" cy="247.4" r="5"/></g><g class="mark"><title>patch cosine, ViT, 16: 1.000</title><circle class="s3 ring" cx="520.0" cy="30.0" r="5"/></g><polyline class="l1" points="64.0,55.0 92.5,74.9 178.0,85.6 292.0,54.3 406.0,91.3 520.0,66.8"/><g class="mark"><title>P(Egyptian cat), ViT, 0: 0.937</title><circle class="s1 ring" cx="64.0" cy="55.0" r="5"/></g><g class="mark"><title>P(Egyptian cat), ViT, 1: 0.888</title><circle class="s1 ring" cx="92.5" cy="74.9" r="5"/></g><g class="mark"><title>P(Egyptian cat), ViT, 4: 0.861</title><circle class="s1 ring" cx="178.0" cy="85.6" r="5"/></g><g class="mark"><title>P(Egyptian cat), ViT, 8: 0.939</title><circle class="s1 ring" cx="292.0" cy="54.3" r="5"/></g><g class="mark"><title>P(Egyptian cat), ViT, 12: 0.847</title><circle class="s1 ring" cx="406.0" cy="91.3" r="5"/></g><g class="mark"><title>P(Egyptian cat), ViT, 16: 0.908</title><circle class="s1 ring" cx="520.0" cy="66.8" r="5"/></g><polyline class="l2" points="64.0,53.4 92.5,54.5 178.0,91.1 292.0,99.4 406.0,187.0 520.0,135.2"/><g class="mark"><title>P(tiger cat), ResNet, 0: 0.942</title><circle class="s2 ring" cx="64.0" cy="53.4" r="5"/></g><g class="mark"><title>P(tiger cat), ResNet, 1: 0.939</title><circle class="s2 ring" cx="92.5" cy="54.5" r="5"/></g><g class="mark"><title>P(tiger cat), ResNet, 4: 0.847</title><circle class="s2 ring" cx="178.0" cy="91.1" r="5"/></g><g class="mark"><title>P(tiger cat), ResNet, 8: 0.826</title><circle class="s2 ring" cx="292.0" cy="99.4" r="5"/></g><g class="mark"><title>P(tiger cat), ResNet, 12: 0.607</title><circle class="s2 ring" cx="406.0" cy="187.0" r="5"/></g><g class="mark"><title>P(tiger cat), ResNet, 16: 0.737</title><circle class="s2 ring" cx="520.0" cy="135.2" r="5"/></g><line class="grid" x1="526.0" y1="30.0" x2="536.0" y2="30.0"/><rect class="s3" x="540.0" y="25.0" width="10" height="10" rx="2"/><text class="t-note" x="556.0" y="34.0" text-anchor="start">patch cosine, ViT: 1.000</text><line class="grid" x1="526.0" y1="66.8" x2="536.0" y2="66.8"/><rect class="s1" x="540.0" y="61.8" width="10" height="10" rx="2"/><text class="t-note" x="556.0" y="70.8" text-anchor="start">P(Egyptian cat), ViT: 0.908</text><line class="grid" x1="526.0" y1="135.2" x2="536.0" y2="135.2"/><rect class="s2" x="540.0" y="130.2" width="10" height="10" rx="2"/><text class="t-note" x="556.0" y="139.2" text-anchor="start">P(tiger cat), ResNet: 0.737</text><text class="t-tick" x="292.0" y="310.0" text-anchor="middle">shift of the picture to the right, in pixels</text><text class="t-tick" x="14.0" y="18.0" text-anchor="start">cosine similarity / probability</text></svg><figcaption>What a shift does to ViT's patch tokens, and to both models' answers. Green: the mean cosine similarity between the original and shifted patch vectors falls from 1.000 to 0.457 as the shift grows to 12 pixels, then jumps back to 1.000 at 16 pixels, when whole patches move one slot. Blue and orange: the probability of each model's top class moves around but the class never changes.</figcaption></figure>

Two lessons sit in this table.

- **The patch tokens are not equivariant.** A 1-pixel shift, invisible to a person, already changes the average patch vector's cosine to 0.935 and the worst one to 0.643. By 12 pixels the mean is 0.457 and some vectors point in the opposite direction. Only a shift of exactly one patch (16 pixels) restores a cosine of 1.000, and then only because we moved the token grid back by one slot. Compared at the same slot, the 16-pixel shift gives 0.428. A convolution has none of this: its output moved by exactly 2 pixels and changed by exactly 0.0.
- **ViT still gets the answer right.** At every shift its top class stays "Egyptian cat" with probability between 0.847 and 0.939. The model was never told that shifts do not matter; it learned it, from millions of pictures in which cats sat in different places. That is the paper's argument in one table: what the CNN has by design, the Transformer can learn from data, *if there is enough data*. ResNet-50 also keeps "tiger cat" throughout, as expected for a CNN.

An honesty note on the ResNet column: its confidence drops to 0.607 at a 12-pixel shift and 0.737 at 16. A real ResNet is not perfectly equivariant either, because its strided layers and pooling sample the picture on coarse grids, and our shift fills the strip that enters on the left by repeating the edge column, which adds a small artificial band. The toy filter above is exactly equivariant; a whole ResNet is only approximately so. The qualitative point stands: both networks keep their answer under shifts that scramble every ViT patch vector.

## Large scale training trumps inductive bias {§1}

> [!PAPER] · §1
> [![The fifth paragraph of the ViT introduction: however, the picture changes if the models are trained on larger datasets (14M-300M images). We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the art on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks](https://ishwarj.com/img/papers/vit/p1-intro-scale.png/p1-intro-scale.png)](/img/papers/vit/p1-intro-scale.png)
>
> **What it says:** "the picture changes if the models are trained on larger datasets (14M-300M images). We find that large scale training trumps inductive bias." Pre-trained on "the public ImageNet-21k dataset or the in-house JFT-300M dataset", ViT "approaches or beats state of the art on multiple image recognition benchmarks". The best model reaches "88.55% on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks".
>
> **What it means:**
>
> This is the turn of the story. "Large scale training trumps inductive bias" is the sentence the paper is remembered for.
>
> Go back to the two students. The one with the summary sheet starts faster, but the sheet also limits them: it holds only the rules a teacher thought of. Give the other student millions of examples, and they can learn better rules than the sheet holds. With enough data, the model with fewer built-in assumptions wins.

The two datasets named here are the paper's fuel.

> [!DEFINITION] ImageNet-21k
> The full ImageNet collection, of which the usual 1,000-class benchmark is a subset: about 14 million pictures labelled with about 21,000 classes (the paper writes "21k classes and 14M images"). It is public, so anyone with enough compute can repeat the ImageNet-21k experiments, and the released ViT weights were pre-trained on it.

> [!DEFINITION] JFT-300M
> Google's internal dataset of about 303 million pictures with about 18,000 noisy labels, described in Sun et al. (2017). It is not public. The paper's very best numbers come from models pre-trained on it; its ImageNet-21k models are the ones the rest of us can download.

"14M-300M images" refers to these two: 14 million for ImageNet-21k, 303 million for JFT-300M. Against them, the 1.3 million pictures of ImageNet are "mid-sized", which is a startling thing to call the dataset that defined the field for a decade.

<figure class="fig"><svg viewBox="0 0 760 280" role="img" aria-label="A sketch of the paper's claim: with little pre-training data a convolutional network beats ViT because its built-in assumptions help; with a lot of data ViT overtakes it. This is a sketch of the idea, not measured data."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="axis" x1="70.0" y1="230.0" x2="590.0" y2="230.0" marker-end="url(#ah)"/><line class="axis" x1="70.0" y1="230.0" x2="70.0" y2="40.0" marker-end="url(#ah)"/><text class="t-tick" x="330.0" y="260.0" text-anchor="middle">amount of pre-training data (more →)</text><text class="t-tick" x="60.0" y="28.0" text-anchor="start">accuracy on the target task</text><polyline class="l1" points="70.0,112.2 200.0,93.2 330.0,81.8 460.0,76.1 590.0,72.3"/><polyline class="l3" points="70.0,144.5 200.0,112.2 330.0,85.6 460.0,66.6 590.0,55.2"/><circle class="s1 ring" cx="70.0" cy="112.2" r="4.5"/><circle class="s1 ring" cx="200.0" cy="93.2" r="4.5"/><circle class="s1 ring" cx="330.0" cy="81.8" r="4.5"/><circle class="s1 ring" cx="460.0" cy="76.1" r="4.5"/><circle class="s1 ring" cx="590.0" cy="72.3" r="4.5"/><circle class="s3 ring" cx="70.0" cy="144.5" r="4.5"/><circle class="s3 ring" cx="200.0" cy="112.2" r="4.5"/><circle class="s3 ring" cx="330.0" cy="85.6" r="4.5"/><circle class="s3 ring" cx="460.0" cy="66.6" r="4.5"/><circle class="s3 ring" cx="590.0" cy="55.2" r="4.5"/><text class="t-s1" x="602.0" y="76.3" text-anchor="start">ResNet (CNN)</text><text class="t-s3" x="602.0" y="59.2" text-anchor="start">ViT</text><text class="t-muted" x="122.0" y="154.0" text-anchor="start">small data: the CNN's built-in</text><text class="t-muted" x="122.0" y="170.0" text-anchor="start">assumptions help; ViT lags</text><text class="t-muted" x="392.4" y="156.2" text-anchor="start">large data: ViT learns the</text><text class="t-muted" x="392.4" y="172.2" text-anchor="start">assumptions, and more, from data</text><text class="t-muted" x="330.0" y="52.0" text-anchor="middle">a sketch of the paper's claim, not measured data (the real numbers follow)</text></svg><figcaption>A sketch of the paper's claim. With little pre-training data, a convolutional network's built-in assumptions help, and it beats ViT. With a lot of data, ViT learns those assumptions, and more, from the pictures themselves, and overtakes. This is a drawing of the idea, not measured data; the measured numbers follow.</figcaption></figure>

The paper proves the sketch in Section 4.3, which Part 4 reads in full. Two sentences from it are worth seeing now, because they are the evidence behind "the picture changes".

> [!PAPER] · §4.3
> [![A passage from Section 4.3: first, we pre-train ViT models on datasets of increasing size: ImageNet, ImageNet-21k, and JFT-300M. When pre-trained on the smallest dataset, ImageNet, ViT-Large models underperform compared to ViT-Base models](https://ishwarj.com/img/papers/vit/p1-sec43-small.png/p1-sec43-small.png)](/img/papers/vit/p1-sec43-small.png)
> [![A sentence from Section 4.3 on page 7: the BiT CNNs outperform ViT on ImageNet, but with the larger datasets, ViT overtakes](https://ishwarj.com/img/papers/vit/p1-sec43-overtake.png-sec43-overtake.png)](/img/papers/vit/p1-sec43-overtake.png)
>
> **What it says:** ViT models were pre-trained "on datasets of increasing size: ImageNet, ImageNet-21k, and JFT-300M". On the smallest, "ViT-Large models underperform compared to ViT-Base models". And: "The BiT CNNs outperform ViT on ImageNet, but with the larger datasets, ViT overtakes."
>
> **What it means:**
>
> These sentences come from Section 4.3, "Pre-training Data Requirements", the experiment that tests the claim of the introduction. They report two crossovers, and both go the way the inductive-bias story predicts.
>
> First, a bigger Transformer is worse than a smaller one on small data. More freedom means more ways to learn the wrong thing from too few pictures. Second, a CNN beats the Transformer on small data and loses on large data. Its built-in assumptions help while data is short, and hold it back once there is plenty.

The exact numbers are in the paper's Table 5 (appendix), which Part 4 redraws in full. Here is the table, and the ImageNet rows for the two 16-pixel-patch models as a chart.

> [!PAPER] · Table 5
> [![Table 5 of the ViT paper: top-1 accuracy of ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32 and ViT-H/14 on CIFAR-10, CIFAR-100, ImageNet, ImageNet ReaL, Oxford Flowers-102 and Oxford-IIIT Pets, after pre-training on ImageNet, ImageNet-21k or JFT-300M. The ImageNet rows are highlighted: 77.91, 73.38, 76.53, 71.16 after ImageNet pre-training; 83.97, 81.28, 85.15, 80.99, 85.13 after ImageNet-21k; 84.15, 80.73, 87.12, 84.37, 88.04 after JFT-300M](https://ishwarj.com/img/papers/vit/p1-table5.pngs/vit/p1-table5.png)](/img/papers/vit/p1-table5.png)
>
> **What it says:** every model gets better as the pre-training set grows. Read the highlighted ImageNet rows for ViT-B/16 and ViT-L/16: 77.91 and 76.53 after ImageNet, 83.97 and 85.15 after ImageNet-21k, 84.15 and 87.12 after JFT-300M.
>
> **What it means:**
>
> Table 5 sits in Appendix C and holds the numbers behind Figure 3 of the main text. All models here were fine-tuned on pictures of 384 pixels.
>
> Compare the two highlighted model rows. With 1.3 million pictures, the larger model (L/16) is *worse* than the smaller one (B/16). With 14 million or 303 million pictures, it is *better*. That is "large scale training trumps inductive bias" in five numbers: a bigger, freer model needs more data before its freedom pays.

<figure class="fig"><svg viewBox="0 0 760 300" role="img" aria-label="ImageNet top-1 accuracy of ViT-B/16 and ViT-L/16 after pre-training on ImageNet (1.3 million images), ImageNet-21k (14 million) and JFT-300M (303 million), from Table 5 of the paper. The larger model is worse with the smallest dataset and best with the largest."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="64" y1="250.0" x2="590" y2="250.0"/><text class="t-tick" x="56.0" y="254.0" text-anchor="end">74%</text><line class="grid" x1="64" y1="206.0" x2="590" y2="206.0"/><text class="t-tick" x="56.0" y="210.0" text-anchor="end">77%</text><line class="grid" x1="64" y1="162.0" x2="590" y2="162.0"/><text class="t-tick" x="56.0" y="166.0" text-anchor="end">80%</text><line class="grid" x1="64" y1="118.0" x2="590" y2="118.0"/><text class="t-tick" x="56.0" y="122.0" text-anchor="end">83%</text><line class="grid" x1="64" y1="74.0" x2="590" y2="74.0"/><text class="t-tick" x="56.0" y="78.0" text-anchor="end">86%</text><line class="grid" x1="64" y1="30.0" x2="590" y2="30.0"/><text class="t-tick" x="56.0" y="34.0" text-anchor="end">89%</text><text class="t-tick" x="64.0" y="270.0" text-anchor="middle">1.3</text><text class="t-tick" x="293.3" y="270.0" text-anchor="middle">14</text><text class="t-tick" x="590.0" y="270.0" text-anchor="middle">303</text><line class="axis" x1="64" y1="250" x2="590" y2="250"/><polyline class="l1" points="64.0,192.7 293.3,103.8 590.0,101.1"/><g class="mark"><title>ViT-B/16, 1.3: 77.91%</title><circle class="s1 ring" cx="64.0" cy="192.7" r="5"/></g><g class="mark"><title>ViT-B/16, 14: 83.97%</title><circle class="s1 ring" cx="293.3" cy="103.8" r="5"/></g><g class="mark"><title>ViT-B/16, 303: 84.15%</title><circle class="s1 ring" cx="590.0" cy="101.1" r="5"/></g><polyline class="l3" points="64.0,212.9 293.3,86.5 590.0,57.6"/><g class="mark"><title>ViT-L/16, 1.3: 76.53%</title><circle class="s3 ring" cx="64.0" cy="212.9" r="5"/></g><g class="mark"><title>ViT-L/16, 14: 85.15%</title><circle class="s3 ring" cx="293.3" cy="86.5" r="5"/></g><g class="mark"><title>ViT-L/16, 303: 87.12%</title><circle class="s3 ring" cx="590.0" cy="57.6" r="5"/></g><line class="grid" x1="596.0" y1="57.6" x2="606.0" y2="57.6"/><rect class="s3" x="610.0" y="52.6" width="10" height="10" rx="2"/><text class="t-note" x="626.0" y="61.6" text-anchor="start">ViT-L/16: 87.12%</text><line class="grid" x1="596.0" y1="101.1" x2="606.0" y2="101.1"/><rect class="s1" x="610.0" y="96.1" width="10" height="10" rx="2"/><text class="t-note" x="626.0" y="105.1" text-anchor="start">ViT-B/16: 84.15%</text><text class="t-tick" x="327.0" y="290.0" text-anchor="middle">pre-training images (millions, log scale): ImageNet 1.3M, ImageNet-21k 14M, JFT-300M 303M</text><text class="t-tick" x="14.0" y="18.0" text-anchor="start">ImageNet top-1 accuracy (%)</text></svg><figcaption>The ImageNet rows of Table 5 for ViT-B/16 and ViT-L/16, against the size of the pre-training set on a log scale. With 1.3 million pictures the smaller model wins (77.91 against 76.53). With 14 million the larger model is ahead (85.15 against 83.97), and with 303 million it is far ahead (87.12 against 84.15).</figcaption></figure>

Now the four headline numbers. They all belong to the paper's biggest model, ViT-H/14 (632 million parameters, 14-pixel patches), pre-trained on JFT-300M.

<figure class="fig"><svg viewBox="0 0 760 222" role="img" aria-label="The four headline numbers of the introduction: 88.55 percent on ImageNet, 90.72 percent on ImageNet-ReaL, 94.55 percent on CIFAR-100 and 77.63 percent on the 19-task VTAB suite."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="40.0" text-anchor="start">top-1 accuracy of the best model (ViT-H/14 pre-trained on JFT-300M)</text><text class="t-tick" x="162.0" y="71.0" text-anchor="end">ImageNet</text><g class="mark"><title>ImageNet: 88.55%</title><rect class="s3" x="170.0" y="53.0" width="371.9" height="26.0" rx="3"/></g><text class="t-val" x="547.9" y="71.0" text-anchor="start">88.55%</text><text class="t-tick" x="162.0" y="105.0" text-anchor="end">ImageNet-ReaL</text><g class="mark"><title>ImageNet-ReaL: 90.72%</title><rect class="s3" x="170.0" y="87.0" width="381.0" height="26.0" rx="3"/></g><text class="t-val" x="557.0" y="105.0" text-anchor="start">90.72%</text><text class="t-tick" x="162.0" y="139.0" text-anchor="end">CIFAR-100</text><g class="mark"><title>CIFAR-100: 94.55%</title><rect class="s3" x="170.0" y="121.0" width="397.1" height="26.0" rx="3"/></g><text class="t-val" x="573.1" y="139.0" text-anchor="start">94.55%</text><text class="t-tick" x="162.0" y="173.0" text-anchor="end">VTAB (19 tasks)</text><g class="mark"><title>VTAB (19 tasks): 77.63%</title><rect class="s3" x="170.0" y="155.0" width="326.0" height="26.0" rx="3"/></g><text class="t-val" x="502.0" y="173.0" text-anchor="start">77.63%</text><text class="t-muted" x="20.0" y="206.0" text-anchor="start">ImageNet-ReaL: ImageNet with cleaner labels. VTAB: the average over 19 small tasks with 1,000 training images each.</text></svg><figcaption>The four headline results of the introduction: 88.55% top-1 on ImageNet, 90.72% on ImageNet-ReaL (ImageNet with cleaner labels), 94.55% on CIFAR-100 and 77.63% averaged over the 19 tasks of VTAB. All from ViT-H/14 pre-trained on JFT-300M.</figcaption></figure>

| Benchmark | What it tests | ViT-H/14 (JFT) | Note |
|---|---|---|---|
| ImageNet | 1,000-class photo classification, 50,000 test pictures | 88.55% | the standard number everyone compares |
| ImageNet-ReaL | the same pictures with corrected labels (Beyer et al., 2020) | 90.72% | higher because fewer "wrong" labels in the test set |
| CIFAR-100 | 100 classes of tiny 32×32 pictures | 94.55% | shows transfer to a very different kind of picture |
| VTAB (19 tasks) | 19 varied tasks with 1,000 training examples each | 77.63% | shows transfer with very little data |

These numbers are compared with the best CNNs in Table 2 of the paper (Part 4). The short version: ViT-H/14 beat the Big Transfer ResNet and Noisy Student on every one of these benchmarks, and did so with less pre-training compute. Part 4 goes through every row and every error bar.

## Who's who: the papers the introduction is talking to

The introduction cites sixteen earlier works in five paragraphs. Here they are in one place, so the rest of the series can refer back to them. Each is listed in full in the references at the end of this part.

| Short name | Paper | Year | What it is, in one line | Role in the ViT paper |
|---|---|---|---|---|
| Transformer | Vaswani et al., [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) | 2017 | the attention-only network for translation | ViT is its encoder, unchanged |
| BERT | Devlin et al., [*BERT*](https://arxiv.org/abs/1810.04805) | 2018 | pre-train a Transformer encoder, fine-tune per task | the recipe, the model sizes and the class token |
| GPT-3 | Brown et al., [*Language Models are Few-Shot Learners*](https://arxiv.org/abs/2005.14165) | 2020 | a 175-billion-parameter language model | "over 100B parameters" |
| GShard | Lepikhin et al., [*GShard*](https://arxiv.org/abs/2006.16668) | 2020 | a 600-billion-parameter translation model | "over 100B parameters" |
| LeNet | LeCun et al., [*Backpropagation Applied to Handwritten Zip Code Recognition*](https://doi.org/10.1162/neco.1989.1.4.541) | 1989 | the first practical CNN | "convolutional architectures remain dominant" |
| AlexNet | Krizhevsky et al., [*ImageNet Classification with Deep CNNs*](https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) | 2012 | the deep CNN that won ImageNet 2012 | start of the deep-learning era in vision |
| ResNet | He et al., [*Deep Residual Learning*](https://arxiv.org/abs/1512.03385) | 2015 | CNNs of 100+ layers | the "ResNet-like architectures" ViT must beat |
| Non-local | Wang et al., [*Non-local Neural Networks*](https://arxiv.org/abs/1711.07971) | 2018 | attention blocks inside a CNN, for video | CNN + attention |
| DETR | Carion et al., [*End-to-End Object Detection with Transformers*](https://arxiv.org/abs/2005.12872) | 2020 | a Transformer on top of a CNN, for detection | CNN + attention |
| Stand-alone | Ramachandran et al., [*Stand-Alone Self-Attention in Vision Models*](https://arxiv.org/abs/1906.05909) | 2019 | local attention replacing every convolution | "replacing the convolutions entirely" |
| Axial-DeepLab | Wang et al. (2020a), [*Axial-DeepLab*](https://arxiv.org/abs/2003.07853) | 2020 | attention along rows, then along columns | "replacing the convolutions entirely" |
| Instagram pre-training | Mahajan et al., [*Exploring the Limits of Weakly Supervised Pretraining*](https://arxiv.org/abs/1805.00932) | 2018 | CNNs pre-trained on 3.5 billion Instagram pictures | ResNets are still state of the art |
| Noisy Student | Xie et al., [*Self-training with Noisy Student*](https://arxiv.org/abs/1911.04252) | 2020 | CNN self-training with 300M unlabelled pictures | ResNets are still state of the art |
| BiT | Kolesnikov et al., [*Big Transfer*](https://arxiv.org/abs/1912.11370) | 2020 | very large ResNets pre-trained on ImageNet-21k and JFT | the main baseline, by the same team |
| ImageNet | Deng et al., [*ImageNet: A Large-Scale Hierarchical Image Database*](https://doi.org/10.1109/CVPR.2009.5206848) | 2009 | the 14-million-picture collection behind ImageNet-1k and 21k | the public pre-training set |
| JFT | Sun et al., [*Revisiting Unreasonable Effectiveness of Data*](https://arxiv.org/abs/1707.02968) | 2017 | Google's 300-million-picture internal dataset | the private pre-training set |

> [!TAKEAWAYS] Key takeaways
> - **The title is the method.** A picture is cut into 16×16-pixel patches, each patch is treated as a word token, and a standard Transformer encoder reads the sequence. A 224×224 picture gives 14 × 14 = 196 patches of 16 × 16 × 3 = 768 numbers; with the class token the sequence has 197 tokens.
> - **"At scale" is the finding.** With only ImageNet's 1.3 million pictures ViT is a few points below ResNets; with 14 million (ImageNet-21k) or 303 million (JFT-300M) it matches or beats the best CNNs. In the paper's words, "large scale training trumps inductive bias".
> - **Why pixels would not work:** attention costs $$n^2$$ per head per layer. Over 50,176 pixels that is 2.5 billion weights (10 GB); over 197 patch tokens it is 38,809 (155 kB), about 65,000 times less. Patches, not clever attention patterns, make a plain Transformer affordable.
> - **Inductive bias** means the assumptions a design builds in. CNNs build in **locality** (look at neighbours) and **translation equivariance** (shift the input, the output shifts the same way). We checked equivariance on a real convolution: output shifted by exactly 2 pixels, maximum difference 0.0.
> - **ViT has neither built in.** Shifting the cat picture by 1 to 12 pixels changed every patch vector (mean cosine 0.935 down to 0.457); only a 16-pixel shift, one whole patch, restored them. Yet ViT kept answering "Egyptian cat" (probability 0.847 to 0.939): it had learned shift-robustness from data.
> - **Two steps:** pre-train once on a huge labelled set (supervised), then fine-tune a copy with one new output layer per benchmark. The same recipe as BERT, but with labels.
> - **The crossover in numbers (Table 5):** ViT-B/16 vs ViT-L/16 on ImageNet after pre-training on 1.3M / 14M / 303M pictures: 77.91 / 83.97 / 84.15 against 76.53 / 85.15 / 87.12. The bigger model loses on small data and wins on large data.
> - **Headline results** of ViT-H/14 pre-trained on JFT-300M: **88.55%** ImageNet, **90.72%** ImageNet-ReaL, **94.55%** CIFAR-100, **77.63%** VTAB.
> - On our sample picture, ViT-B/16 (86.6M parameters) and ResNet-50 (25.6M) both said "cat" with about 94% confidence and disagreed only on the breed (Egyptian cat 0.937 vs tiger cat 0.942).
> - The paper does classification only. Detection, segmentation and label-free pre-training came later (Part 6).

**Next, in [Part 2](https://ishwarj.com/papers/vit/part-2-the-model/):** the related work the paper builds on (local attention, Sparse Transformers, iGPT, and the 2×2-patch model of Cordonnier et al.), and then the model itself: Figure 1 block by block, Equations 1 to 4 with the real numbers of ViT-B/16 (the patch embedding, the class token, the position embeddings, one encoder layer), the Appendix A formulas, and where the 86 million parameters live.

<details>
<summary>Run it yourself</summary>

The two scripts behind this part are [`code/papers/vit/vit_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1.py) (the cat picture through ViT-B/16 and ResNet-50) and [`code/papers/vit/vit_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1_math.py) (the quadratic cost table, the equivariance check on a toy convolution, and the patch-shift experiment). Both run on a CPU in under a minute each and download `google/vit-base-patch16-224` (about 350 MB) and `microsoft/resnet-50` (about 100 MB) the first time.

```bash
pip install torch transformers datasets pillow
python vit_part1.py         # shapes, patch arithmetic, top-5 of both models; writes results/part1.json
python vit_part1_math.py    # quadratic cost, equivariance, patch shifts; writes results/part1_math.json
```

<figure><img src="/img/papers/vit/part1-run.png" alt="Terminal output of vit_part1.py: the picture size, the ViT-B/16 tensor shapes and patch arithmetic, its top-5 classes led by Egyptian cat 0.9374, the ResNet-50 feature-map shapes and its top-5 classes led by tiger cat 0.9416" loading="lazy" /><figcaption>The real output of vit_part1.py.</figcaption></figure>

<figure><img src="/img/papers/vit/part1_math-run.png" alt="Terminal output of vit_part1_math.py: the table of attention weights for pixels versus patches, the toy convolution equivariance check with a maximum difference of 0.0, and the patch-shift table with mean cosine similarities and both models' top-1 classes for shifts of 0, 1, 4, 8, 12 and 16 pixels" loading="lazy" /><figcaption>The real output of vit_part1_math.py.</figcaption></figure>

</details>

## References

**The ViT paper**

1. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. [*An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*](https://arxiv.org/abs/2010.11929) (ViT). ICLR 2021 ([OpenReview](https://openreview.net/forum?id=YicbFdNTTy)). arXiv:2010.11929, version 2 (June 2021), which is the version shown in the screenshots.
2. Google Research. [Vision Transformer code and pre-trained models](https://github.com/google-research/vision_transformer). GitHub, 2020.

**Papers the ViT paper cites in this part**

3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin. [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) (the Transformer). NeurIPS 2017.
4. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805) (BERT). NAACL 2019.
5. T. B. Brown et al. [*Language Models are Few-Shot Learners*](https://arxiv.org/abs/2005.14165) (GPT-3). NeurIPS 2020.
6. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, Z. Chen. [*GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding*](https://arxiv.org/abs/2006.16668) (GShard). ICLR 2021.
7. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel. [*Backpropagation Applied to Handwritten Zip Code Recognition*](https://doi.org/10.1162/neco.1989.1.4.541). Neural Computation 1(4), 1989.
8. A. Krizhevsky, I. Sutskever, G. E. Hinton. [*ImageNet Classification with Deep Convolutional Neural Networks*](https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html) (AlexNet). NeurIPS 2012.
9. K. He, X. Zhang, S. Ren, J. Sun. [*Deep Residual Learning for Image Recognition*](https://arxiv.org/abs/1512.03385) (ResNet). CVPR 2016.
10. X. Wang, R. Girshick, A. Gupta, K. He. [*Non-local Neural Networks*](https://arxiv.org/abs/1711.07971). CVPR 2018.
11. N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko. [*End-to-End Object Detection with Transformers*](https://arxiv.org/abs/2005.12872) (DETR). ECCV 2020.
12. P. Ramachandran, N. Parmar, A. Vaswani, I. Bello, A. Levskaya, J. Shlens. [*Stand-Alone Self-Attention in Vision Models*](https://arxiv.org/abs/1906.05909). NeurIPS 2019.
13. H. Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, L.-C. Chen. [*Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation*](https://arxiv.org/abs/2003.07853) ("Wang et al., 2020a" in the paper). ECCV 2020.
14. D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, L. van der Maaten. [*Exploring the Limits of Weakly Supervised Pretraining*](https://arxiv.org/abs/1805.00932). ECCV 2018.
15. Q. Xie, M.-T. Luong, E. Hovy, Q. V. Le. [*Self-training with Noisy Student improves ImageNet classification*](https://arxiv.org/abs/1911.04252) (Noisy Student). CVPR 2020.
16. A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, N. Houlsby. [*Big Transfer (BiT): General Visual Representation Learning*](https://arxiv.org/abs/1912.11370) (BiT). ECCV 2020.
17. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei. [*ImageNet: A Large-Scale Hierarchical Image Database*](https://doi.org/10.1109/CVPR.2009.5206848). CVPR 2009.
18. C. Sun, A. Shrivastava, S. Singh, A. Gupta. [*Revisiting Unreasonable Effectiveness of Data in Deep Learning Era*](https://arxiv.org/abs/1707.02968) (JFT-300M). ICCV 2017.

**Other sources used in this part**

19. X. Zhai, J. Puigcerver, A. Kolesnikov, et al. [*A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark*](https://arxiv.org/abs/1910.04867) (VTAB). arXiv 2019. The benchmark behind the 77.63% number.
20. L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, A. van den Oord. [*Are we done with ImageNet?*](https://arxiv.org/abs/2006.07159) (ImageNet-ReaL). arXiv 2020. The cleaned-up labels behind the 90.72% number.
21. A. Krizhevsky. [*The CIFAR-10 and CIFAR-100 datasets*](https://www.cs.toronto.edu/~kriz/cifar.html). University of Toronto, 2009.
22. Hugging Face model cards: [`google/vit-base-patch16-224`](https://huggingface.co/google/vit-base-patch16-224) and [`microsoft/resnet-50`](https://huggingface.co/microsoft/resnet-50); the sample picture is the [`huggingface/cats-image`](https://huggingface.co/datasets/huggingface/cats-image) dataset.
23. Code for this part: [`vit_part1.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1.py) and [`vit_part1_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part1_math.py).

