# Inside ViT: Patches, Embeddings and the Encoder

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/vit/part-2-the-model/
Updated: 2026-10-17
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vit/part-2-the-model.md


In [Part 1](https://ishwarj.com/papers/vit/part-1-the-big-idea/) we read the paper's claim: cut a picture into 16×16 patches, feed them to an unchanged Transformer, and with enough data it beats the best convolutional networks. This part opens the model. First we read Section 2, the earlier work the paper builds on and argues against. Then Figure 1 and Section 3.1, which describe the model in one picture, four equations and a few paragraphs. Last we read Appendix A, the attention equations the paper hands to its appendix.

Everything that can be checked, we check on a released model, `google/vit-base-patch16-224` (ViT-B/16, pre-trained on ImageNet-21k and fine-tuned on ImageNet at 224 pixels), with one real picture of two cats. Every step of the model is recomputed in plain PyTorch and compared with the library's own output.

## Related work: attention on images before ViT {§2}

> [!PAPER] · §2
> [![The first paragraph of Section 2: Transformers were proposed by Vaswani et al. (2017) for machine translation and have since become the state of the art method in many NLP tasks. Large Transformer-based models are often pre-trained on large corpora and then fine-tuned: BERT uses a denoising self-supervised pre-training task, while the GPT line of work uses language modeling as its pre-training task](https://ishwarj.com/img/papers/vit/p2-related-transformers.pnged-transformers.png)](/img/papers/vit/p2-related-transformers.png)
>
> **What it says:** Transformers "were proposed by Vaswani et al. (2017) for machine translation" and are now the standard in language. Big ones are pre-trained and then fine-tuned: BERT with "a denoising self-supervised pre-training task" (hide words, predict them), the GPT line with "language modeling" (predict the next word).
>
> **What it means:**
>
> Related Work opens with language, not pictures, because language is where the recipe comes from. A plain Transformer is pre-trained on a lot of data, then fine-tuned for each task. BERT and GPT differ only in the pre-training game: fill in hidden words, or guess the next one.
>
> This is the recipe ViT wants to copy for images. The whole paper asks one question: does the recipe survive the move from words to pixels?

> [!DEFINITION] Transformer encoder
> A stack of identical layers that take a sequence of vectors in and give a sequence of vectors of the same size out. Each layer lets every position look at every other position (self-attention) and then processes each position on its own (an MLP). It is the half of the 2017 Transformer that reads; BERT is one, and so is ViT. The [attention series](https://ishwarj.com/writings/attention-1-self-attention/) builds it up from scratch.

> [!DEFINITION] Self-attention
> The step in which every token in a sequence compares itself with every other token, decides how much to look at each one, and takes a weighted mix of their information. The weights are computed from the data, not fixed in advance. Appendix A of the paper, covered at the end of this part, writes it out.

> [!PAPER] · §2
> [![The second paragraph of Section 2: naive application of self-attention to images would require that each pixel attends to every other pixel, with quadratic cost in the number of pixels. Parmar et al. applied self-attention only in local neighborhoods for each query pixel; such local blocks can completely replace convolutions (Hu, Ramachandran, Zhao); Sparse Transformers (Child et al.) employ scalable approximations; attention in blocks of varying sizes (Weissenborn) or only along individual axes (Ho, Wang); these require complex engineering to be implemented efficiently on hardware accelerators](https://ishwarj.com/img/papers/vit/p2-related-approx.png-related-approx.png)](/img/papers/vit/p2-related-approx.png)
>
> **What it says:** attention over pixels means "each pixel attends to every other pixel", which has "quadratic cost in the number of pixels" and "does not scale to realistic input sizes". So people approximated: attention "only in local neighborhoods for each query pixel" (Parmar et al., 2018), local attention blocks that "can completely replace convolutions" (Hu et al., 2019; Ramachandran et al., 2019; Zhao et al., 2020), "Sparse Transformers" (Child et al., 2019), attention "in blocks of varying sizes" (Weissenborn et al., 2019) or "only along individual axes" (Ho et al., 2019; Wang et al., 2020a). All of them work, but "require complex engineering to be implemented efficiently on hardware accelerators".
>
> **What it means:**
>
> Picture a meeting where every person must talk to every other person. With 10 people that is 100 talks; with 50,176 people (the pixels of a 224×224 picture) it is about 2.5 billion. Every method in this paragraph cuts the talks: neighbours only, rows and columns only, or fixed patterns.
>
> The last sentence is the real argument. Each shortcut is a custom attention pattern, and custom patterns need custom code to run fast on GPUs and TPUs. ViT solves the problem another way. It does not make attention cheaper; it makes the sequence shorter (196 patches instead of 50,176 pixels), so plain attention is affordable.

> [!DEFINITION] Quadratic cost
> If a computation over $$n$$ items has to look at every pair of items, its cost grows with $$n^2$$. Double the items and the work is four times larger. Self-attention over $$n$$ tokens has this cost, because every token compares itself with every token. Part 1 works out the numbers for pixels.

> [!DEFINITION] Locality
> The assumption that what matters for a pixel is mostly the pixels near it. Convolutions build it in: each output only sees a small window of the input. Local attention borrows the same assumption to cut the cost.

> [!DEFINITION] Global versus local
> A **global** operation lets every position see every other position at once (plain self-attention). A **local** one lets each position see only a neighbourhood (a convolution, or local attention). Global is more flexible; local is cheaper and carries a built-in assumption.

> [!DEFINITION] Sparse attention
> Self-attention in which each token looks at only a chosen subset of the tokens rather than all of them, following a fixed pattern such as "my row, plus every k-th token". The cost drops from quadratic towards linear, at the price of a pattern that has to be designed and implemented by hand.

> [!DEFINITION] Axial attention
> Sparse attention for grids: a pixel attends first to the other pixels in its row, then (in another layer) to those in its column. Two cheap passes cover the whole image indirectly. Ho et al. (2019) and Wang et al. (2020a) use it.

Two of these ideas are worth seeing in their own papers. Here is Parmar et al. (2018), the Image Transformer, saying in its abstract why it went local:

> [!PAPER] Parmar et al. (2018), Image Transformer · Abstract
> [![The abstract of the Image Transformer: by restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks](https://ishwarj.com/img/papers/vit/p2-ext-parmar.pngt/p2-ext-parmar.png)](/img/papers/vit/p2-ext-parmar.png)
>
> **What it says:** "By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice", while still seeing more than a convolution does.
>
> **What it means:**
>
> The Image Transformer is the paper that ViT cites for "self-attention only in local neighborhoods". Its goal was different: it generated images pixel by pixel, and did not classify them.
>
> Its abstract shows the trade that every method in the paragraph makes. Give up global attention, and you can afford larger images. ViT makes the opposite trade: it keeps global attention and gives up single pixels as tokens.

And here are the patterns of the Sparse Transformer, which the ViT paper names explicitly:

> [!PAPER] Child et al. (2019), Generating Long Sequences with Sparse Transformers · Figure 3
> [![Figure 3 of the Sparse Transformer paper: three connectivity patterns for a 6 by 6 image. (a) full attention of a standard Transformer, (b) strided sparse attention where a position attends to its row and to every k-th position, (c) fixed sparse attention where positions attend within blocks and to summary positions; the bottom row shows each pattern as a connectivity matrix](https://ishwarj.com/img/papers/vit/p2-ext-child-fig3.png-ext-child-fig3.png)](/img/papers/vit/p2-ext-child-fig3.png)
>
> **What it says:** in (a) every output can see every input (the full lower triangle of the matrix). In (b) and (c) only the coloured cells are computed: a row, a column stride, a block. The patterns are chosen so that two heads in sequence still connect everything.
>
> **What it means:**
>
> The figure compares full attention with "two 2d factorized attention schemes" on a tiny 6×6 image. Each grid in the bottom row is an attention matrix: a coloured cell means "this output may look at this input".
>
> These pictures make "scalable approximations to global self-attention" concrete. It means deleting most cells of the attention matrix by hand, with a pattern that a person designs. ViT's attention matrix is the full one, (a), but only 197×197 in size.

> [!PAPER] · §2
> [![The third paragraph of Section 2: most related to ours is the model of Cordonnier et al. (2020), which extracts patches of size 2 by 2 from the input image and applies full self-attention on top. This model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with state-of-the-art CNNs. Cordonnier et al. use a small patch size of 2 by 2 pixels, which makes the model applicable only to small-resolution images](https://ishwarj.com/img/papers/vit/p2-related-cordonnier.pngated-cordonnier.png)](/img/papers/vit/p2-related-cordonnier.png)
>
> **What it says:** Cordonnier et al. (2020) "extracts patches of size 2 × 2 from the input image and applies full self-attention on top". "This model is very similar to ViT." The differences the authors claim: ViT shows that "large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs", and 2×2 patches make the earlier model "applicable only to small-resolution images", while 16×16 patches handle "medium-resolution images as well".
>
> **What it means:**
>
> This is the honest answer to "was the idea new?". No: cutting a picture into patches and running full attention on top already existed.
>
> What was new is the scale, of both the patches and the data. Patches of 2×2 pixels keep the sequence long, so the earlier model could only handle small pictures. Patches of 16×16 make medium-sized pictures affordable, and huge pre-training sets make the plain Transformer competitive. The result that followed is the real contribution.

> [!DEFINITION] Patch
> A small square cut out of the picture, here 16×16 pixels with 3 colour channels. ViT treats each patch as one token, the way BERT treats each word piece. The picture is cut into a grid of patches with no overlap.

The Cordonnier paper is about something else: it proves that attention *can* do what a convolution does, and shows that trained attention layers often learn to. Its 2×2 step appears in the experiments as a down-sampling detail:

> [!PAPER] Cordonnier, Loukas and Jaggi (2020), On the Relationship between Self-Attention and Convolutional Layers · Abstract
> [![The abstract of Cordonnier et al.: this work provides evidence that attention layers can perform convolution and indeed they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer](https://ishwarj.com/img/papers/vit/p2-ext-cordonnier-abs.png-cordonnier-abs.png)](/img/papers/vit/p2-ext-cordonnier-abs.png)
> [![The experimental setup of Cordonnier et al.: in all experiments, we use a 2 by 2 invertible down-sampling on the input to reduce the size of the image, since the attention coefficient tensors scale quadratically with the size of the input image and full attention cannot be applied to bigger images](https://ishwarj.com/img/papers/vit/p2-ext-cordonnier-2x2.png-cordonnier-2x2.png)](/img/papers/vit/p2-ext-cordonnier-2x2.png)
>
> **What it says:** they "prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer". In the experiments "we use a 2 × 2 invertible down-sampling on the input", because the attention tensors "scale quadratically with the size of the input image" and "full attention cannot be applied to bigger images".
>
> **What it means:**
>
> The two crops come from the abstract and from the experiments on CIFAR-10, whose pictures are only 32×32 pixels. The 2×2 groups of the experiments are the "patches" that the ViT paper refers to.
>
> Count the tokens. A 32×32 picture in 2×2 groups gives 256 tokens. A 224×224 picture would give 12,544, far too many for full attention. ViT's 16×16 patches give 196 at 224×224. It is the same idea with patches eight times larger. It also serves a different question: what attention *can* do, against what it does at scale.

> [!PAPER] · §2
> [![The fourth paragraph of Section 2: there has also been a lot of interest in combining convolutional neural networks with forms of self-attention, by augmenting feature maps for image classification (Bello et al.) or by further processing the output of a CNN using self-attention, for object detection (Hu et al., Carion et al.), video processing (Wang et al., Sun et al.), image classification (Wu et al.), unsupervised object discovery (Locatello et al.) or unified text-vision tasks (Chen et al., Lu et al., Li et al.)](https://ishwarj.com/img/papers/vit/p2-related-cnn-attn.pngelated-cnn-attn.png)](/img/papers/vit/p2-related-cnn-attn.png)
>
> **What it says:** people have combined CNNs "with forms of self-attention", either "by augmenting feature maps" inside the CNN (Bello et al., 2019) or "by further processing the output of a CNN using self-attention". The second kind covers object detection (relation networks; DETR), video (non-local networks; VideoBERT), classification (Visual Transformers), object discovery (slot attention) and joint text-and-image models (UNITER, ViLBERT, VisualBERT).
>
> **What it means:**
>
> In all these models the CNN does the seeing and attention does the reasoning on top. Attention is an add-on to the CNN, not a replacement for it.
>
> ViT's "hybrid" model, defined at the end of Section 3.1, is exactly this kind of design. The paper builds it so that it can test the add-on design against the pure Transformer, with everything else the same.

> [!DEFINITION] Feature map
> The output of a convolutional layer: a grid of positions, each holding a vector of numbers. A ResNet turns a 224×224×3 picture into, for example, a 14×14×1024 feature map: 196 positions, each described by 1,024 numbers. The hybrid ViT reads these positions as tokens.

> [!PAPER] · §2
> [![The fifth paragraph of Section 2: another recent related model is image GPT (iGPT), which applies Transformers to image pixels after reducing image resolution and color space. The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy of 72% on ImageNet](https://ishwarj.com/img/papers/vit/p2-related-igpt.pngp2-related-igpt.png)](/img/papers/vit/p2-related-igpt.png)
>
> **What it says:** iGPT "applies Transformers to image pixels after reducing image resolution and color space". It is trained "as a generative model" (predict the next pixel), and its representation "can then be fine-tuned or probed linearly", reaching "a maximal accuracy of 72% on ImageNet".
>
> **What it means:**
>
> iGPT is the one earlier model that ran a Transformer on raw pixels at scale. It shows the other way to shorten the sequence. It shrinks the picture (to 32×32 or 64×64) and squashes the colours into 512 values, so that one pixel is one token.
>
> The price is the 72% on ImageNet, far below the 85% to 88% of good CNNs. Too much of the picture is lost before the model sees it. ViT keeps the resolution and shortens the sequence by cutting it into patches.

> [!DEFINITION] Generative model
> A model trained to produce the data itself, for example to predict the next pixel given the previous ones (iGPT) or the next word given the previous ones (GPT). Nothing is labelled; the data is its own target.

> [!DEFINITION] iGPT
> Image GPT (Chen et al., 2020a): a GPT-2-style Transformer run over the pixels of small images, one pixel per token, trained to predict the next pixel.

> [!DEFINITION] Linear probe
> A test of a representation: freeze the model, take its output vectors, and train only one linear layer on top to predict the classes. If the probe scores well, the frozen vectors already contain the information. Part 4 uses a related "few-shot linear" measure.

> [!PAPER] · §2
> [![The last paragraph of Section 2: our work adds to the increasing collection of papers that explore image recognition at larger scales than the standard ImageNet dataset. Additional data sources allow state-of-the-art results (Mahajan et al., Touvron et al., Xie et al.). Sun et al. (2017) study how CNN performance scales with dataset size, and Kolesnikov et al. (2020) and Djolonga et al. (2020) explore CNN transfer learning from ImageNet-21k and JFT-300M. We focus on these two latter datasets as well, but train Transformers instead of ResNet-based models](https://ishwarj.com/img/papers/vit/p2-related-scale.png2-related-scale.png)](/img/papers/vit/p2-related-scale.png)
>
> **What it says:** the paper "adds to the increasing collection of papers that explore image recognition at larger scales than the standard ImageNet dataset": Mahajan et al. (2018, Instagram hashtags), Touvron et al. (2019), Xie et al. (2020, Noisy Student). Sun et al. (2017) studied "how CNN performance scales with dataset size"; Kolesnikov et al. (2020, Big Transfer) and Djolonga et al. (2020) studied transfer from "ImageNet-21k and JFT-300M". ViT uses "these two latter datasets as well, but train[s] Transformers instead of ResNet-based models".
>
> **What it means:**
>
> The closing paragraph places ViT in the "more data" line of work, not in the "new attention" line. Its idea is not a clever new layer. Its idea is a plain design fed with far more pictures.
>
> This tells you how to read the results in Part 4. The comparison that matters to the authors is with Big Transfer, the same team's ResNets trained on the same data. Same data, same recipe, different design: any difference comes from the design.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="Map of the related work in Section 2: five earlier ways of putting attention on images (local attention, sparse and axial attention, small 2 by 2 patches, CNN plus attention, iGPT on pixels) and ViT below them, which uses 16 by 16 patches, full attention and the standard encoder with large-scale pre-training."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="150.0" y="14.0" width="460.0" height="40.0" rx="10"/><text class="t-note" x="380.0" y="39.0" text-anchor="middle">Full self-attention over pixels: cost grows with (number of pixels)²</text><rect class="box" x="4.0" y="90.0" width="144.0" height="128.0" rx="10"/><text class="t-note" x="76.0" y="110.0" text-anchor="middle">Local attention</text><text class="t-tick" x="12.0" y="130.0" text-anchor="start">Parmar 2018 (Image</text><text class="t-tick" x="12.0" y="145.0" text-anchor="start">Transformer): each</text><text class="t-tick" x="12.0" y="160.0" text-anchor="start">pixel attends to a</text><text class="t-tick" x="12.0" y="175.0" text-anchor="start">neighbourhood.</text><text class="t-tick" x="12.0" y="190.0" text-anchor="start">Hu, Ramachandran,</text><text class="t-tick" x="12.0" y="205.0" text-anchor="start">Zhao: replace convs</text><line class="edge" x1="380.0" y1="54.0" x2="76.0" y2="88.0" marker-end="url(#ah)"/><rect class="box" x="156.0" y="90.0" width="144.0" height="128.0" rx="10"/><text class="t-note" x="228.0" y="110.0" text-anchor="middle">Sparse and axial</text><text class="t-tick" x="164.0" y="130.0" text-anchor="start">Child 2019 (Sparse</text><text class="t-tick" x="164.0" y="145.0" text-anchor="start">Transformers);</text><text class="t-tick" x="164.0" y="160.0" text-anchor="start">Weissenborn 2019</text><text class="t-tick" x="164.0" y="175.0" text-anchor="start">(blocks); Ho 2019,</text><text class="t-tick" x="164.0" y="190.0" text-anchor="start">Wang 2020a: one</text><text class="t-tick" x="164.0" y="205.0" text-anchor="start">axis at a time</text><line class="edge" x1="380.0" y1="54.0" x2="228.0" y2="88.0" marker-end="url(#ah)"/><rect class="box-on" x="308.0" y="90.0" width="144.0" height="128.0" rx="10"/><text class="t-note" x="380.0" y="110.0" text-anchor="middle">Small patches</text><text class="t-tick" x="316.0" y="130.0" text-anchor="start">Cordonnier 2020:</text><text class="t-tick" x="316.0" y="145.0" text-anchor="start">2×2 patches, full</text><text class="t-tick" x="316.0" y="160.0" text-anchor="start">attention on top.</text><text class="t-tick" x="316.0" y="175.0" text-anchor="start">Small images only.</text><text class="t-tick" x="316.0" y="190.0" text-anchor="start">Closest to ViT.</text><text class="t-tick" x="316.0" y="205.0" text-anchor="start"></text><line class="edge" x1="380.0" y1="54.0" x2="380.0" y2="88.0" marker-end="url(#ah)"/><rect class="box" x="460.0" y="90.0" width="144.0" height="128.0" rx="10"/><text class="t-note" x="532.0" y="110.0" text-anchor="middle">CNN + attention</text><text class="t-tick" x="468.0" y="130.0" text-anchor="start">Bello 2019; Hu 2018;</text><text class="t-tick" x="468.0" y="145.0" text-anchor="start">Carion 2020 (DETR);</text><text class="t-tick" x="468.0" y="160.0" text-anchor="start">Wang 2018; Sun 2019;</text><text class="t-tick" x="468.0" y="175.0" text-anchor="start">Wu 2020; Locatello</text><text class="t-tick" x="468.0" y="190.0" text-anchor="start">2020; UNITER,</text><text class="t-tick" x="468.0" y="205.0" text-anchor="start">ViLBERT, VisualBERT</text><line class="edge" x1="380.0" y1="54.0" x2="532.0" y2="88.0" marker-end="url(#ah)"/><rect class="box" x="612.0" y="90.0" width="144.0" height="128.0" rx="10"/><text class="t-note" x="684.0" y="110.0" text-anchor="middle">Pixels as tokens</text><text class="t-tick" x="620.0" y="130.0" text-anchor="start">iGPT (Chen 2020a):</text><text class="t-tick" x="620.0" y="145.0" text-anchor="start">generative model</text><text class="t-tick" x="620.0" y="160.0" text-anchor="start">on down-scaled</text><text class="t-tick" x="620.0" y="175.0" text-anchor="start">pixels; linear</text><text class="t-tick" x="620.0" y="190.0" text-anchor="start">probe reaches 72%</text><text class="t-tick" x="620.0" y="205.0" text-anchor="start">on ImageNet</text><line class="edge" x1="380.0" y1="54.0" x2="684.0" y2="88.0" marker-end="url(#ah)"/><rect class="box-1" x="100.0" y="252.0" width="560.0" height="50.0" rx="10"/><text class="t-note" x="380.0" y="272.0" text-anchor="middle">ViT: 16×16 patches, full (global) attention, the standard encoder unchanged,</text><text class="t-note" x="380.0" y="290.0" text-anchor="middle">and pre-training on 14M to 300M images</text><line class="path" x1="380.0" y1="218.0" x2="380.0" y2="250.0" marker-end="url(#ah-on)"/><text class="t-muted" x="380.0" y="326.0" text-anchor="middle">the large-data line it joins: Mahajan 2018, Touvron 2019, Xie 2020, Sun 2017, Kolesnikov 2020 (BiT), Djolonga 2020</text></svg><figcaption>The related work on one map. Five earlier ways of putting attention on images, each a different answer to the quadratic cost: local attention, sparse and axial patterns, 2×2 patches with full attention, attention bolted on to a CNN, and iGPT on shrunken pixels. ViT, below, keeps full attention and the standard encoder, uses 16×16 patches, and joins the large-data line of work.</figcaption></figure>

## Figure 1: the whole model in one picture {§3}

Section 3 is short. It opens with one sentence of design philosophy and one picture, Figure 1. Here is the picture first, because everything else in this part explains one piece of it.

> [!PAPER] · Figure 1
> [![Figure 1 of the ViT paper. Left: a picture is split into nine patches, each goes through a linear projection of flattened patches, position embeddings 1 to 9 are added along with an extra learnable class embedding at position 0, the sequence enters a Transformer encoder, and an MLP head on the class token outputs Bird, Ball, Car. Right: the Transformer encoder block, repeated L times: Norm, Multi-Head Attention, a residual addition, Norm, MLP, another residual addition. Caption: we split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder; to perform classification we use the standard approach of adding an extra learnable classification token](https://ishwarj.com/img/papers/vit/p2-figure1.png/vit/p2-figure1.png)](/img/papers/vit/p2-figure1.png)
>
> **What it says:** "We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to a standard Transformer encoder. In order to perform classification, we use the standard approach of adding an extra learnable 'classification token' to the sequence."
>
> **What it means:**
>
> This is the one figure of the method section, and its caption is the whole method in two sentences. Learn to read the picture, and the four equations of Section 3.1 are just its labels.
>
> Note the word "standard" twice: a "standard Transformer encoder" and "the standard approach" of a classification token. The authors want you to see that nothing in the picture is new except the bottom row, where the photo becomes patches.

How to read Figure 1, left to right and bottom to top:

- **Bottom left, the picture:** split into a grid of patches (nine in the drawing; 196 for a real 224×224 picture with 16×16 patches).
- **"Linear Projection of Flattened Patches" (pink bar):** each patch is flattened into one long row of numbers and multiplied by one matrix, $$\mathbf{E}$$. Every patch uses the same matrix.
- **"Patch + Position Embedding" (the circles 0 to 9):** to each projected patch a learned position vector is added, so the model can tell patch 3 from patch 7. The starred circle 0 is the "extra learnable [class] embedding": a token that belongs to no patch.
- **"Transformer Encoder" (grey box):** the standard encoder, $$L$$ layers of it. Every token can look at every token.
- **"MLP Head" and "Class":** only the output at position 0, the class token, is read. A small head turns it into a label.
- **Right half, the encoder block:** "Norm", "Multi-Head Attention", an addition; "Norm", "MLP", an addition. Note that Norm comes *before* each block, with the plus signs *after*. That is the "pre-norm" arrangement, and it is Equations 2 and 3.

<figure class="fig"><svg viewBox="0 0 760 426" role="img" aria-label="Figure 1 of the paper redrawn as nine numbered frames: the picture, cutting into 16 by 16 patches, flattening, the linear projection E, prepending the class token, adding position embeddings, the 12-layer encoder, reading the class token output, and the classification head that answers Egyptian cat."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="6.0" y="10.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="24.0" cy="28.0" r="11"/><text class="t-note" x="24.0" y="32.5" text-anchor="middle">1</text><text class="t-title" x="42.0" y="33.0" text-anchor="start">The picture</text><rect class="s1" x="30.0" y="50.0" width="23" height="23" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="54.0" y="50.0" width="23" height="23" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="78.0" y="50.0" width="23" height="23" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="30.0" y="74.0" width="23" height="23" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="54.0" y="74.0" width="23" height="23" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="78.0" y="74.0" width="23" height="23" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="30.0" y="98.0" width="23" height="23" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="54.0" y="98.0" width="23" height="23" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="78.0" y="98.0" width="23" height="23" rx="1.5" style="fill-opacity:0.45"/><text class="t-tick" x="116.0" y="65.0" text-anchor="start">224 × 224 pixels,</text><text class="t-tick" x="116.0" y="81.0" text-anchor="start">3 colour channels</text><text class="t-tick" x="116.0" y="97.0" text-anchor="start">x: H × W × C</text><rect class="box" x="258.0" y="10.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="276.0" cy="28.0" r="11"/><text class="t-note" x="276.0" y="32.5" text-anchor="middle">2</text><text class="t-title" x="294.0" y="33.0" text-anchor="start">Cut into patches</text><rect class="s1" x="282" y="50" width="22" height="22" rx="2" style="fill-opacity:0.35"/><rect class="s1" x="309" y="50" width="22" height="22" rx="2" style="fill-opacity:0.50"/><rect class="s1" x="336" y="50" width="22" height="22" rx="2" style="fill-opacity:0.65"/><rect class="s1" x="282" y="77" width="22" height="22" rx="2" style="fill-opacity:0.60"/><rect class="s1" x="309" y="77" width="22" height="22" rx="2" style="fill-opacity:0.40"/><rect class="s1" x="336" y="77" width="22" height="22" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="282" y="104" width="22" height="22" rx="2" style="fill-opacity:0.50"/><rect class="s1" x="309" y="104" width="22" height="22" rx="2" style="fill-opacity:0.65"/><rect class="s1" x="336" y="104" width="22" height="22" rx="2" style="fill-opacity:0.45"/><text class="t-tick" x="373.0" y="65.0" text-anchor="start">16 × 16 × 3 each</text><text class="t-tick" x="373.0" y="81.0" text-anchor="start">224/16 = 14 per side</text><text class="t-tick" x="373.0" y="97.0" text-anchor="start">N = 14 × 14 = 196</text><rect class="box" x="510.0" y="10.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="528.0" cy="28.0" r="11"/><text class="t-note" x="528.0" y="32.5" text-anchor="middle">3</text><text class="t-title" x="546.0" y="33.0" text-anchor="start">Flatten each patch</text><rect class="box-ghost" x="531.0" y="51.0" width="160.0" height="17.0" rx="4"/><g class="mark"><rect class="s1" x="534.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.49"/></g><g class="mark"><rect class="s1" x="547.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><rect class="s1" x="560.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><rect class="s1" x="573.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><rect class="s1" x="586.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.74"/></g><g class="mark"><rect class="s1" x="599.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s1" x="612.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.80"/></g><g class="mark"><rect class="s1" x="625.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><rect class="s1" x="638.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="651.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.82"/></g><g class="mark"><rect class="s1" x="664.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.87"/></g><g class="mark"><rect class="s1" x="677.0" y="54.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><rect class="box-ghost" x="531.0" y="73.0" width="160.0" height="17.0" rx="4"/><g class="mark"><rect class="s1" x="534.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s1" x="547.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="560.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="573.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.93"/></g><g class="mark"><rect class="s1" x="586.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><rect class="s1" x="599.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><rect class="s1" x="612.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><rect class="s1" x="625.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.75"/></g><g class="mark"><rect class="s1" x="638.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.71"/></g><g class="mark"><rect class="s1" x="651.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><rect class="s1" x="664.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.81"/></g><g class="mark"><rect class="s1" x="677.0" y="76.0" width="11" height="11" rx="2" style="fill-opacity:0.75"/></g><rect class="box-ghost" x="531.0" y="95.0" width="160.0" height="17.0" rx="4"/><g class="mark"><rect class="s1" x="534.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><rect class="s1" x="547.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><rect class="s1" x="560.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><rect class="s1" x="573.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><rect class="s1" x="586.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><rect class="s1" x="599.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s1" x="612.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="625.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><rect class="s1" x="638.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><rect class="s1" x="651.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><rect class="s1" x="664.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><rect class="s1" x="677.0" y="98.0" width="11" height="11" rx="2" style="fill-opacity:0.58"/></g><text class="t-tick" x="700.0" y="65.0" text-anchor="start">768</text><text class="t-tick" x="700.0" y="81.0" text-anchor="start">numbers</text><text class="t-tick" x="700.0" y="97.0" text-anchor="start">per patch</text><rect class="box" x="6.0" y="148.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="24.0" cy="166.0" r="11"/><text class="t-note" x="24.0" y="170.5" text-anchor="middle">4</text><text class="t-title" x="42.0" y="171.0" text-anchor="start">Linear projection E</text><rect class="box-ghost" x="17.0" y="215.0" width="114.0" height="15.0" rx="4"/><g class="mark"><rect class="s1" x="20.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="31.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="42.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="53.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="64.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="75.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="86.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="97.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="108.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s1" x="119.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><line class="edge" x1="134.0" y1="222.0" x2="148.0" y2="222.0" marker-end="url(#ah)"/><rect class="box-4" x="150.0" y="204.0" width="54.0" height="36.0" rx="6"/><text class="t-math" x="177.0" y="220.0" text-anchor="middle">E</text><text class="t-cell" x="177.0" y="234.0" text-anchor="middle">768×768</text><line class="edge" x1="206.0" y1="222.0" x2="220.0" y2="222.0" marker-end="url(#ah)"/><g class="mark"><rect class="s4" x="222.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="233.0" y="218.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><text class="t-tick" x="20.0" y="258.0" text-anchor="start">x_p E: 196 patch embeddings of 768</text><rect class="box" x="258.0" y="148.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="276.0" cy="166.0" r="11"/><text class="t-note" x="276.0" y="170.5" text-anchor="middle">5</text><text class="t-title" x="294.0" y="171.0" text-anchor="start">Prepend [class]</text><rect class="box-2" x="272.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="289.0" y="218.5" text-anchor="middle">[class]</text><rect class="box-1" x="310.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="327.0" y="218.5" text-anchor="middle">p1</text><rect class="box-1" x="348.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="365.0" y="218.5" text-anchor="middle">p2</text><rect class="box-1" x="386.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="403.0" y="218.5" text-anchor="middle">p3</text><rect class="box-1" x="424.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="441.0" y="218.5" text-anchor="middle">…</text><rect class="box-1" x="462.0" y="200.0" width="34.0" height="28.0" rx="6"/><text class="t-cell" x="479.0" y="218.5" text-anchor="middle">p196</text><text class="t-tick" x="272.0" y="252.0" text-anchor="start">196 + 1 = 197 tokens; x_class is learned</text><rect class="box" x="510.0" y="148.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="528.0" cy="166.0" r="11"/><text class="t-note" x="528.0" y="170.5" text-anchor="middle">6</text><text class="t-title" x="546.0" y="171.0" text-anchor="start">Add position embeddings</text><rect class="box-2" x="524.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="541.0" y="208.5" text-anchor="middle">[class]</text><rect class="box-1" x="562.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="579.0" y="208.5" text-anchor="middle">p1</text><rect class="box-1" x="600.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="617.0" y="208.5" text-anchor="middle">p2</text><rect class="box-1" x="638.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="655.0" y="208.5" text-anchor="middle">p3</text><rect class="box-1" x="676.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="693.0" y="208.5" text-anchor="middle">…</text><rect class="box-1" x="714.0" y="192.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="731.0" y="208.5" text-anchor="middle">p196</text><rect class="box-3" x="524.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="541.0" y="244.5" text-anchor="middle">pos 0</text><rect class="box-3" x="562.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="579.0" y="244.5" text-anchor="middle">pos 1</text><rect class="box-3" x="600.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="617.0" y="244.5" text-anchor="middle">pos 2</text><rect class="box-3" x="638.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="655.0" y="244.5" text-anchor="middle">pos 3</text><rect class="box-3" x="676.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="693.0" y="244.5" text-anchor="middle">…</text><rect class="box-3" x="714.0" y="228.0" width="34.0" height="24.0" rx="6"/><text class="t-cell" x="731.0" y="244.5" text-anchor="middle">pos 196</text><text class="t-note" x="517.0" y="221.0" text-anchor="middle">+</text><text class="t-tick" x="524.0" y="268.0" text-anchor="start">E_pos: 197 learned rows (Eq. 1)</text><rect class="box" x="6.0" y="286.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="24.0" cy="304.0" r="11"/><text class="t-note" x="24.0" y="308.5" text-anchor="middle">7</text><text class="t-title" x="42.0" y="309.0" text-anchor="start">Transformer encoder</text><rect class="box-1" x="20.0" y="326.0" width="216.0" height="50.0" rx="8"/><text class="t-tick" x="128.0" y="346.0" text-anchor="middle">LN → MSA → + → LN → MLP → +</text><text class="t-tick" x="128.0" y="366.0" text-anchor="middle">repeated L = 12 times (Eq. 2, 3)</text><text class="t-tick" x="20.0" y="398.0" text-anchor="start">197 × 768 in, 197 × 768 out</text><rect class="box" x="258.0" y="286.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="276.0" cy="304.0" r="11"/><text class="t-note" x="276.0" y="308.5" text-anchor="middle">8</text><text class="t-title" x="294.0" y="309.0" text-anchor="start">Read the [class] output</text><rect class="box-on" x="272.0" y="338.0" width="40.0" height="28.0" rx="6"/><text class="t-cell" x="292.0" y="356.5" text-anchor="middle">z_L⁰</text><rect class="box-ghost" x="316.0" y="338.0" width="40.0" height="28.0" rx="6"/><text class="t-cell" x="336.0" y="356.5" text-anchor="middle">z_L¹</text><rect class="box-ghost" x="360.0" y="338.0" width="40.0" height="28.0" rx="6"/><text class="t-cell" x="380.0" y="356.5" text-anchor="middle">z_L²</text><rect class="box-ghost" x="404.0" y="338.0" width="40.0" height="28.0" rx="6"/><text class="t-cell" x="424.0" y="356.5" text-anchor="middle">…</text><rect class="box-ghost" x="448.0" y="338.0" width="40.0" height="28.0" rx="6"/><text class="t-cell" x="468.0" y="356.5" text-anchor="middle">z_L¹⁹⁶</text><text class="t-tick" x="272.0" y="390.0" text-anchor="start">y = LN(z_L⁰): 768 numbers (Eq. 4)</text><rect class="box" x="510.0" y="286.0" width="244.0" height="128.0" rx="12"/><circle class="node on" cx="528.0" cy="304.0" r="11"/><text class="t-note" x="528.0" y="308.5" text-anchor="middle">9</text><text class="t-title" x="546.0" y="309.0" text-anchor="start">Classification head</text><rect class="box-4" x="524.0" y="330.0" width="110.0" height="34.0" rx="6"/><text class="t-cell" x="579.0" y="351.0" text-anchor="middle">Linear 768→1000</text><line class="edge" x1="636.0" y1="347.0" x2="650.0" y2="347.0" marker-end="url(#ah)"/><text class="t-note" x="654.0" y="344.0" text-anchor="start">Egyptian cat</text><text class="t-tick" x="654.0" y="360.0" text-anchor="start">0.937 (softmax)</text><text class="t-tick" x="524.0" y="398.0" text-anchor="start">one linear layer when fine-tuned</text></svg><figcaption>Figure 1 redrawn as nine numbered frames with the real sizes. The 224×224×3 picture is cut into 196 patches of 16×16×3; each is flattened to 768 numbers and multiplied by E (768×768); the [class] token is prepended and 197 position rows are added; the 12-layer encoder keeps the shape 197×768; the [class] output is normalised and a single linear layer gives 1000 scores, the largest of which is "Egyptian cat" with probability 0.937 on our picture.</figcaption></figure>

> [!PAPER] · §3
> [![The opening of Section 3: in model design we follow the original Transformer (Vaswani et al., 2017) as closely as possible. An advantage of this intentionally simple setup is that scalable NLP Transformer architectures, and their efficient implementations, can be used almost out of the box](https://ishwarj.com/img/papers/vit/p2-method-intro.pngp2-method-intro.png)](/img/papers/vit/p2-method-intro.png)
>
> **What it says:** "we follow the original Transformer (Vaswani et al., 2017) as closely as possible." The reason: "scalable NLP Transformer architectures, and their efficient implementations, can be used almost out of the box."
>
> **What it means:**
>
> To follow the original Transformer "as closely as possible" is a deliberate choice, not laziness. Related Work complained that custom attention needs custom engineering. A standard Transformer needs none.
>
> The code, the hardware kernels and the scaling know-how from language transfer directly. Think of a new car that fits every existing fuel station: nobody has to build anything before it can drive. Later parts show that this is what let the authors train models of 632 million parameters.

The phrase "out of the box" is also why we can check this paper so thoroughly. ViT-B/16 has the same layer count, hidden size, head count and MLP size as BERT-base ([BERT Part 2](https://ishwarj.com/papers/bert/part-2-architecture-and-input/)), and the released code is a few hundred lines on top of a standard Transformer implementation.

<figure class="fig"><svg viewBox="0 0 760 316" role="img" aria-label="BERT and ViT side by side: text tokens with a CLS token versus patch tokens with a class token, a lookup-plus-position embedding versus a linear projection plus position, the same 12-layer encoder of width 768, and the first output token feeding the task head in both."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="190.0" y="24.0" text-anchor="middle">BERT (text)</text><text class="t-title" x="570.0" y="24.0" text-anchor="middle">ViT (pictures)</text><rect class="box-2" x="20.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="50.0" y="62.5" text-anchor="middle">[CLS]</text><rect class="box" x="86.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="116.0" y="62.5" text-anchor="middle">the</text><rect class="box" x="152.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="182.0" y="62.5" text-anchor="middle">kid</text><rect class="box" x="218.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="248.0" y="62.5" text-anchor="middle">smiles</text><rect class="box" x="284.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="314.0" y="62.5" text-anchor="middle">[SEP]</text><text class="t-tick" x="190.0" y="92.0" text-anchor="middle">WordPiece tokens</text><rect class="box-2" x="400.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="430.0" y="62.5" text-anchor="middle">[class]</text><rect class="box-1" x="466.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="496.0" y="62.5" text-anchor="middle">p1</text><rect class="box-1" x="532.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="562.0" y="62.5" text-anchor="middle">p2</text><rect class="box-1" x="598.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="628.0" y="62.5" text-anchor="middle">…</text><rect class="box-1" x="664.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="694.0" y="62.5" text-anchor="middle">p196</text><text class="t-tick" x="570.0" y="92.0" text-anchor="middle">16 × 16 patches</text><line class="edge" x1="184.0" y1="98.0" x2="184.0" y2="112.0" marker-end="url(#ah)"/><rect class="box-3" x="20.0" y="114.0" width="328.0" height="30.0" rx="8"/><text class="t-tick" x="184.0" y="133.0" text-anchor="middle">token table lookup + segment + position (then LayerNorm)</text><line class="edge" x1="184.0" y1="146.0" x2="184.0" y2="160.0" marker-end="url(#ah)"/><line class="edge" x1="564.0" y1="98.0" x2="564.0" y2="112.0" marker-end="url(#ah)"/><rect class="box-3" x="400.0" y="114.0" width="328.0" height="30.0" rx="8"/><text class="t-tick" x="564.0" y="133.0" text-anchor="middle">x_p E (linear projection) + position</text><line class="edge" x1="564.0" y1="146.0" x2="564.0" y2="160.0" marker-end="url(#ah)"/><rect class="box-1" x="20.0" y="162.0" width="328.0" height="50.0" rx="10"/><text class="t-note" x="184.0" y="182.0" text-anchor="middle">Transformer encoder</text><text class="t-tick" x="184.0" y="200.0" text-anchor="middle">12 layers, width 768, 12 heads, MLP 3072</text><line class="edge" x1="184.0" y1="214.0" x2="184.0" y2="228.0" marker-end="url(#ah)"/><rect class="box-1" x="400.0" y="162.0" width="328.0" height="50.0" rx="10"/><text class="t-note" x="564.0" y="182.0" text-anchor="middle">Transformer encoder</text><text class="t-tick" x="564.0" y="200.0" text-anchor="middle">12 layers, width 768, 12 heads, MLP 3072</text><line class="edge" x1="564.0" y1="214.0" x2="564.0" y2="228.0" marker-end="url(#ah)"/><rect class="box-on" x="20.0" y="230.0" width="328.0" height="30.0" rx="8"/><text class="t-tick" x="184.0" y="249.0" text-anchor="middle">C = output of [CLS] → task head (sentence label)</text><rect class="box-on" x="400.0" y="230.0" width="328.0" height="30.0" rx="8"/><text class="t-tick" x="564.0" y="249.0" text-anchor="middle">y = LN(z_L⁰) → head (image class)</text><text class="t-big" x="374.0" y="190.0" text-anchor="middle">=</text><text class="t-big" x="374.0" y="134.0" text-anchor="middle">≈</text><text class="t-muted" x="20.0" y="288.0" text-anchor="start">Differences: BERT normalises after each block (post-norm) and looks tokens up in a table of 30,522 rows;</text><text class="t-muted" x="20.0" y="305.0" text-anchor="start">ViT normalises before each block (pre-norm) and projects every patch with one 768 × 768 matrix E.</text></svg><figcaption>BERT and ViT side by side. BERT reads word pieces with a [CLS] token in front; ViT reads 16×16 patches with a [class] token in front. BERT embeds a token by table lookup plus segment plus position; ViT embeds a patch by a linear projection plus position. The encoder in between has the same dimensions (12 layers, width 768, 12 heads, MLP 3072), and in both the first output token feeds the task head.</figcaption></figure>

## From pixels to a sequence: the reshape {§3.1}

> [!PAPER] · §3.1
> [![The first paragraph of Section 3.1: the standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, we reshape the image x in R H by W by C into a sequence of flattened 2D patches x p in R N by P squared times C, where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image patch, and N equals HW over P squared is the resulting number of patches, which also serves as the effective input sequence length. The Transformer uses constant latent vector size D through all of its layers, so we flatten the patches and map to D dimensions with a trainable linear projection (Eq. 1). We refer to the output of this projection as the patch embeddings](https://ishwarj.com/img/papers/vit/p2-vit-reshape.png/p2-vit-reshape.png)](/img/papers/vit/p2-vit-reshape.png)
>
> **What it says:** a Transformer wants "a 1D sequence of token embeddings". So the picture $$\mathbf{x} \in \mathbb{R}^{H \times W \times C}$$ is reshaped into "a sequence of flattened 2D patches" $$\mathbf{x}_p \in \mathbb{R}^{N \times (P^2 \cdot C)}$$. $$(H, W)$$ is the picture's resolution, $$C$$ its number of channels, $$(P, P)$$ the patch size, and $$N = HW/P^2$$ the number of patches, "which also serves as the effective input sequence length". Because the Transformer "uses constant latent vector size $$D$$ through all of its layers", the flattened patches are mapped "to $$D$$ dimensions with a trainable linear projection". The result is called "the patch embeddings".
>
> **What it means:**
>
> This paragraph does the only real translation work in the paper: it turns a 2D picture into a 1D sequence. Think of cutting a photo into stamps and laying them in one row, left to right and top to bottom. Each stamp is then flattened into a list of numbers and multiplied by one matrix.
>
> Five symbols ($$H, W, C, P, N$$) and that one matrix do the whole job. There is no convolution stack, no feature pyramid, nothing learned about pictures before the Transformer sees them. The paper's "surprising" results rest on how little happens here.

> [!DEFINITION] Tensor shape
> The list of sizes of an array of numbers. A colour picture of height 224 and width 224 has shape $$224 \times 224 \times 3$$ (in PyTorch the channels come first: $$3 \times 224 \times 224$$). A sequence of 197 tokens with 768 numbers each has shape $$197 \times 768$$. Following the shapes is the easiest way to follow a model.

> [!DEFINITION] Reshape
> Rearranging the same numbers into a different shape without changing any of them. Cutting a picture into patches and stacking them as rows is a reshape: no arithmetic, only a new order.

> [!DEFINITION] Flattening
> Reshaping a block of numbers into one long row. A 16×16×3 patch flattened is a row of 768 numbers. The order is fixed (here: channel, then row, then column), so position 0 is always the red value of the top-left pixel.

> [!DEFINITION] Sequence length N
> How many tokens the Transformer reads at once. For text it is the number of word pieces; for ViT it is the number of patches, $$N = HW/P^2$$, plus one for the class token. Attention cost grows with $$N^2$$, so this number decides what the model can afford.

> [!DEFINITION] Hidden size D
> How many numbers describe each token inside the model. ViT-B uses $$D = 768$$, the same as BERT-base. It is the same from the first layer to the last, which is what "constant latent vector size" means.

> [!DEFINITION] Linear projection
> Multiplying a vector by a matrix (and usually adding a bias vector). It maps a row of $$a$$ numbers to a row of $$b$$ numbers with $$a \times b$$ learned weights. "Trainable" means the matrix is learned with the rest of the model.

The arithmetic for our picture, with the paper's symbols:

$$
N = \frac{HW}{P^2} = \frac{224 \times 224}{16^2} = \frac{50{,}176}{256} = 196, \qquad P^2 \cdot C = 16 \times 16 \times 3 = 768
$$

where $$H = W = 224$$ is the resolution the model was trained at, $$P = 16$$ is the patch size (the "/16" in "ViT-B/16"), and $$C = 3$$ for red, green and blue. So the picture becomes $$\mathbf{x}_p$$, a matrix of 196 rows and 768 columns. A happy coincidence of ViT-B/16 at 224: the row length $$P^2 C = 768$$ equals $$D = 768$$, so $$\mathbf{E}$$ happens to be square. For ViT-B/32 the rows are $$32 \cdot 32 \cdot 3 = 3{,}072$$ long and $$\mathbf{E}$$ is $$3{,}072 \times 768$$.

<figure class="fig"><svg viewBox="0 0 760 288" role="img" aria-label="The reshape of Section 3.1: the 224 by 224 by 3 picture, drawn as a 14 by 14 grid of 16-pixel patches, becomes the matrix x_p with 196 rows and 768 columns; N equals 224 times 224 over 16 squared equals 196."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="28.0" text-anchor="start">x ∈ R^(H×W×C): 224 × 224 × 3</text><rect class="s1" x="40.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="51.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="62.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="73.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="84.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="95.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="106.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="117.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="128.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="139.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="150.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="161.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="172.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="183.0" y="50.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="40.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="51.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="62.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="73.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="84.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="95.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="106.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="117.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="128.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="139.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="150.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="161.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="172.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="183.0" y="61.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="40.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="51.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="62.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="73.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="84.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="95.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="106.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="117.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="128.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="139.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="150.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="161.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="172.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="183.0" y="72.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="40.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="51.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="62.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="73.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="84.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="95.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="106.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="117.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="128.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="139.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="150.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="161.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="172.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="183.0" y="83.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="40.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="51.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="62.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="73.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="84.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="95.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="106.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="117.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="128.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="139.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="150.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="161.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="172.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="183.0" y="94.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="40.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="51.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="62.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="73.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="84.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="95.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="106.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="117.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="128.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="139.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="150.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="161.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="172.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="183.0" y="105.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="40.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="51.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="62.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="73.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="84.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="95.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="106.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="117.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="128.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="139.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="150.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="161.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="172.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="183.0" y="116.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="40.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="51.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="62.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="73.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="84.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="95.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="106.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="117.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="128.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="139.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="150.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="161.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="172.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="183.0" y="127.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="40.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="51.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="62.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="73.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="84.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="95.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="106.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="117.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="128.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="139.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="150.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="161.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="172.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="183.0" y="138.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="40.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="51.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="62.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="73.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="84.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="95.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="106.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="117.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="128.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="139.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="150.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="161.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="172.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="183.0" y="149.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="40.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="51.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="62.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="73.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="84.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="95.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="106.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="117.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="128.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="139.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="150.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="161.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="172.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="183.0" y="160.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="40.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="51.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="62.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="73.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="84.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="95.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="106.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="117.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="128.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="139.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="150.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="161.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="172.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="183.0" y="171.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="40.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="51.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="62.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="73.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="84.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="95.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="106.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="117.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="128.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="139.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="150.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="161.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="172.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="183.0" y="182.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="40.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="51.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="62.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="73.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="84.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="95.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="106.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="117.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="128.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="139.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="150.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="161.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="172.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="183.0" y="193.0" width="10" height="10" rx="1.5" style="fill-opacity:0.65"/><path class="edge" d="M40.0,202.0 L40.0,210.0 L194.0,210.0 L194.0,202.0"/><text class="t-tick" x="117.0" y="228.0" text-anchor="middle">224 px = 14 patches × 16 px</text><text class="t-tick" x="30.0" y="131.0" text-anchor="end">14</text><line class="edge" x1="215.0" y1="125.0" x2="300.0" y2="125.0" marker-end="url(#ah)"/><text class="t-note" x="257.0" y="112.0" text-anchor="middle">reshape</text><text class="t-tick" x="230.0" y="160.0" text-anchor="start">N = HW / P²</text><text class="t-tick" x="230.0" y="176.0" text-anchor="start">= 224 · 224 / 16²</text><text class="t-tick" x="230.0" y="192.0" text-anchor="start">= 50,176 / 256</text><text class="t-tick" x="230.0" y="208.0" text-anchor="start">= 196 patches</text><rect class="box-ghost" x="326.0" y="46.0" width="239.6" height="154.8" rx="5"/><rect class="s1" x="330.0" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="340.6" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="351.2" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="361.8" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="372.4" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="383.0" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="393.6" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="404.2" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="414.8" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="425.4" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="436.0" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="446.6" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="457.2" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="467.8" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="478.4" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="489.0" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="499.6" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="510.2" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="520.8" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="531.4" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="542.0" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="552.6" y="50.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="330.0" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="340.6" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="351.2" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="361.8" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="372.4" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="383.0" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="393.6" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="404.2" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="414.8" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="425.4" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="436.0" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="446.6" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="457.2" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="467.8" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="478.4" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="489.0" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="499.6" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="510.2" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="520.8" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="531.4" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="542.0" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="552.6" y="60.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="330.0" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="340.6" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="351.2" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="361.8" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="372.4" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="383.0" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="393.6" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="404.2" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="414.8" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="425.4" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="436.0" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="446.6" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="457.2" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="467.8" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="478.4" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="489.0" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="499.6" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="510.2" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="520.8" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="531.4" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="542.0" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="552.6" y="71.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="330.0" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="340.6" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="351.2" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="361.8" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="372.4" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="383.0" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="393.6" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="404.2" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="414.8" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="425.4" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="436.0" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="446.6" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="457.2" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="467.8" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="478.4" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="489.0" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="499.6" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="510.2" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="520.8" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="531.4" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="542.0" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="552.6" y="81.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="330.0" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="340.6" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="351.2" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="361.8" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="372.4" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="383.0" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="393.6" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="404.2" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="414.8" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="425.4" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="436.0" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="446.6" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="457.2" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="467.8" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="478.4" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="489.0" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="499.6" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="510.2" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="520.8" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="531.4" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="542.0" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="552.6" y="92.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="330.0" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="340.6" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="351.2" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="361.8" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="372.4" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="383.0" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="393.6" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="404.2" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="414.8" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="425.4" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="436.0" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="446.6" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="457.2" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="467.8" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="478.4" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="489.0" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="499.6" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="510.2" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="520.8" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="531.4" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="542.0" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="552.6" y="103.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="330.0" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="340.6" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="351.2" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="361.8" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="372.4" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="383.0" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="393.6" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="404.2" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="414.8" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="425.4" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="436.0" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="446.6" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="457.2" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="467.8" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="478.4" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="489.0" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="499.6" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="510.2" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="520.8" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="531.4" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="542.0" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="552.6" y="113.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="330.0" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="340.6" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="351.2" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="361.8" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="372.4" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="383.0" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="393.6" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="404.2" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="414.8" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="425.4" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="436.0" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="446.6" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="457.2" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="467.8" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="478.4" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="489.0" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="499.6" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="510.2" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="520.8" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="531.4" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="542.0" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="552.6" y="124.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="330.0" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="340.6" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="351.2" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="361.8" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="372.4" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="383.0" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="393.6" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="404.2" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="414.8" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="425.4" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="436.0" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="446.6" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="457.2" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="467.8" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="478.4" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="489.0" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="499.6" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="510.2" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="520.8" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="531.4" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="542.0" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="552.6" y="134.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="330.0" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="340.6" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="351.2" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="361.8" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="372.4" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="383.0" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="393.6" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="404.2" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="414.8" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="425.4" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="436.0" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="446.6" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="457.2" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="467.8" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="478.4" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="489.0" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="499.6" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="510.2" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="520.8" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="531.4" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="542.0" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="552.6" y="145.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="330.0" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="340.6" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="351.2" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="361.8" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="372.4" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="383.0" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="393.6" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="404.2" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="414.8" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="425.4" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="436.0" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="446.6" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="457.2" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="467.8" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="478.4" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="489.0" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="499.6" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="510.2" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="520.8" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="531.4" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="542.0" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="552.6" y="156.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="330.0" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="340.6" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="351.2" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="361.8" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="372.4" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="383.0" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="393.6" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="404.2" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="414.8" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="425.4" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="436.0" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="446.6" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="457.2" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="467.8" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="478.4" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="489.0" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="499.6" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="510.2" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="520.8" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="531.4" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="542.0" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="552.6" y="166.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="330.0" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="340.6" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="351.2" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="361.8" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="372.4" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="383.0" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="393.6" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="404.2" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="414.8" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="425.4" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="436.0" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="446.6" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="457.2" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="467.8" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="478.4" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="489.0" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="499.6" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="510.2" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="520.8" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="531.4" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="542.0" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="552.6" y="177.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="330.0" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="340.6" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="351.2" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="361.8" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="372.4" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="383.0" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="393.6" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="404.2" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="414.8" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="425.4" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="436.0" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="446.6" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="457.2" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="467.8" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="478.4" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="489.0" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="499.6" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="510.2" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="520.8" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="531.4" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="542.0" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="552.6" y="187.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><text class="t-math" x="445.8" y="38.0" text-anchor="middle">x_p</text><text class="t-tick" x="445.8" y="216.8" text-anchor="middle">196 rows × 768 columns  (N × P²·C)</text><text class="t-tick" x="575.6" y="60.0" text-anchor="start">row 0 = top-left patch</text><text class="t-tick" x="575.6" y="196.8" text-anchor="start">row 195 = bottom-right</text><text class="t-tick" x="575.6" y="120.0" text-anchor="start">each row: the 768</text><text class="t-tick" x="575.6" y="136.0" text-anchor="start">pixel numbers of</text><text class="t-tick" x="575.6" y="152.0" text-anchor="start">one patch (16·16·3)</text><text class="t-muted" x="20.0" y="258.0" text-anchor="start">The grid is read row by row: patch n sits at row ⌊n/14⌋, column n mod 14.</text><text class="t-muted" x="20.0" y="275.0" text-anchor="start">The order is fixed, so position 1 is always the top-left corner.</text></svg><figcaption>The reshape of Section 3.1 drawn out. Left: the 224×224×3 picture as a 14×14 grid of 16-pixel patches (224/16 = 14 per side). Right: the matrix x_p with 196 rows and 768 columns; row 0 is the top-left patch, row 195 the bottom-right. N = 224·224/16² = 50,176/256 = 196.</figcaption></figure>

Here is the picture we use, resized to 224×224 the way the released preprocessing does it, with the 14×14 patch grid drawn on top:

<figure><img src="/img/papers/vit/p2-sample-grid.png" alt="The sample picture: two cats lying on a red sofa with remote controls, resized to 224 by 224 pixels and shown with a 14 by 14 grid of white lines marking the 16 by 16 pixel patches" loading="lazy" width="448" /><figcaption>Our test picture (from the `huggingface/cats-image` dataset) at 224×224 with the 196 patches marked. Each cell is one token for the model.</figcaption></figure>

Cutting and flattening in PyTorch is three lines ([`vit_part2_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part2_math.py)). `unfold` slides a 16-wide window with step 16 along the height and then the width, which gives one 16×16 square per patch and per channel; a `permute` and a `reshape` put the patches as rows:

```python
patches = x.unfold(1, P, P).unfold(2, P, P)                      # (3, 14, 14, 16, 16)
patches = patches.permute(1, 2, 0, 3, 4).reshape(N, C * P * P)   # (196, 768): row n = patch n
```

```text
== 1. cut x (3, 224, 224) into 196 patches of 16x16x3 and flatten each to 768 numbers ==
  x.unfold -> (3, 14, 14, 16, 16)  (channels, 14 rows of patches, 14 columns, 16, 16)
  x_p = flattened patches: (196, 768)   (N = 196 patches, P^2*C = 768 numbers each)
  patch 0 (top-left corner), first 8 of its 768 numbers: [  0.114,   0.169,   0.184,   0.200,   0.208,   0.239,   0.231,   0.200]
  patch 0, number 0 is red channel, pixel (0,0): x[0,0,0] = 0.114; number 256 is green (0,0): -0.804; number 512 is blue (0,0): -0.545
  check: patch 17 (row 1, col 3) == x[:, 16:32, 48:64] flattened: True
```

The numbers are pixel values after the released preprocessing: each channel is scaled from 0 to 255 into $$[0, 1]$$ and then mapped to $$[-1, 1]$$ by $$(v - 0.5)/0.5$$. The top-left pixel of our picture is reddish (red 0.114, green −0.804, blue −0.545), which is the sofa. The check on patch 17 confirms the order: patch $$n$$ sits at grid row $$\lfloor n/14 \rfloor$$ and column $$n \bmod 14$$, so patch 17 is row 1, column 3, and it is exactly the pixels `x[:, 16:32, 48:64]`.

<figure class="fig"><svg viewBox="0 0 760 250" role="img" aria-label="One 16 by 16 by 3 patch flattened into a strip of 768 numbers: the 256 red values first, then 256 green, then 256 blue, with the real first 40 values of patch 0 drawn as shaded squares."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="28.0" text-anchor="start">One 16 × 16 × 3 patch becomes one row of 768 numbers</text><rect class="s1" x="54" y="84" width="80" height="80" rx="4" style="fill-opacity:0.55"/><rect class="s3" x="42" y="72" width="80" height="80" rx="4" style="fill-opacity:0.55"/><rect class="s2" x="30" y="60" width="80" height="80" rx="4" style="fill-opacity:0.55"/><text class="t-tick" x="80.0" y="170.0" text-anchor="middle">patch 0 (top-left)</text><text class="t-tick" x="80.0" y="186.0" text-anchor="middle">C = 3 planes of 16×16</text><line class="edge" x1="150.0" y1="100.0" x2="190.0" y2="100.0" marker-end="url(#ah)"/><rect class="box-ghost" x="197.0" y="91.0" width="524.0" height="17.0" rx="4"/><g class="mark"><title>x_p[0][0] = 0.114</title><rect class="s2" x="200.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><title>x_p[0][1] = 0.169</title><rect class="s2" x="213.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><title>x_p[0][2] = 0.184</title><rect class="s2" x="226.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><title>x_p[0][3] = 0.200</title><rect class="s2" x="239.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><title>x_p[0][4] = 0.208</title><rect class="s2" x="252.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><title>x_p[0][5] = 0.239</title><rect class="s2" x="265.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><title>x_p[0][6] = 0.231</title><rect class="s2" x="278.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><title>x_p[0][7] = 0.200</title><rect class="s2" x="291.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><title>x_p[0][8] = 0.216</title><rect class="s2" x="304.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.67"/></g><g class="mark"><title>x_p[0][9] = 0.239</title><rect class="s2" x="317.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><title>x_p[0][10] = 0.255</title><rect class="s2" x="330.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><title>x_p[0][11] = 0.286</title><rect class="s2" x="343.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.83"/></g><g class="mark"><title>x_p[0][12] = 0.294</title><rect class="s2" x="356.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><title>x_p[0][13] = 0.302</title><rect class="s2" x="369.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><title>x_p[0][14] = 0.302</title><rect class="s2" x="382.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.86"/></g><g class="mark"><title>x_p[0][15] = 0.294</title><rect class="s2" x="395.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><title>x_p[0][16] = 0.137</title><rect class="s2" x="408.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.49"/></g><g class="mark"><title>x_p[0][17] = 0.169</title><rect class="s2" x="421.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.56"/></g><g class="mark"><title>x_p[0][18] = 0.184</title><rect class="s2" x="434.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.60"/></g><g class="mark"><title>x_p[0][19] = 0.224</title><rect class="s2" x="447.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><title>x_p[0][20] = 0.208</title><rect class="s2" x="460.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.65"/></g><g class="mark"><title>x_p[0][21] = 0.239</title><rect class="s2" x="473.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.72"/></g><g class="mark"><title>x_p[0][22] = 0.247</title><rect class="s2" x="486.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.74"/></g><g class="mark"><title>x_p[0][23] = 0.224</title><rect class="s2" x="499.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.68"/></g><g class="mark"><title>x_p[0][24] = 0.231</title><rect class="s2" x="512.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.70"/></g><g class="mark"><title>x_p[0][25] = 0.255</title><rect class="s2" x="525.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.76"/></g><g class="mark"><title>x_p[0][26] = 0.271</title><rect class="s2" x="538.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.79"/></g><g class="mark"><title>x_p[0][27] = 0.294</title><rect class="s2" x="551.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><title>x_p[0][28] = 0.294</title><rect class="s2" x="564.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.84"/></g><g class="mark"><title>x_p[0][29] = 0.326</title><rect class="s2" x="577.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><title>x_p[0][30] = 0.341</title><rect class="s2" x="590.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><title>x_p[0][31] = 0.326</title><rect class="s2" x="603.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.91"/></g><g class="mark"><title>x_p[0][32] = 0.114</title><rect class="s2" x="616.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.44"/></g><g class="mark"><title>x_p[0][33] = 0.153</title><rect class="s2" x="629.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.53"/></g><g class="mark"><title>x_p[0][34] = 0.161</title><rect class="s2" x="642.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.54"/></g><g class="mark"><title>x_p[0][35] = 0.176</title><rect class="s2" x="655.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.58"/></g><g class="mark"><title>x_p[0][36] = 0.192</title><rect class="s2" x="668.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.61"/></g><g class="mark"><title>x_p[0][37] = 0.200</title><rect class="s2" x="681.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.63"/></g><g class="mark"><title>x_p[0][38] = 0.216</title><rect class="s2" x="694.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.67"/></g><g class="mark"><title>x_p[0][39] = 0.216</title><rect class="s2" x="707.0" y="94.0" width="11" height="11" rx="2" style="fill-opacity:0.67"/></g><text class="t-tick" x="200.0" y="80.0" text-anchor="start">first 40 of the 768 numbers (red channel: pixel row 0, start of row 1):</text><text class="t-code" x="200.0" y="128.0" text-anchor="start">0.114, 0.169, 0.184, 0.200, 0.208, 0.239, 0.231, 0.200, …</text><rect class="s2" x="200" y="170" width="176" height="18" rx="3" style="fill-opacity:0.5"/><path class="edge" d="M200.0,186.0 L200.0,194.0 L376.0,194.0 L376.0,186.0"/><text class="t-tick" x="288.0" y="212.0" text-anchor="middle">256 red: 16 rows × 16 pixels</text><rect class="s3" x="380" y="170" width="176" height="18" rx="3" style="fill-opacity:0.5"/><path class="edge" d="M380.0,186.0 L380.0,194.0 L556.0,194.0 L556.0,186.0"/><text class="t-tick" x="468.0" y="212.0" text-anchor="middle">256 green</text><rect class="s1" x="560" y="170" width="176" height="18" rx="3" style="fill-opacity:0.5"/><path class="edge" d="M560.0,186.0 L560.0,194.0 L736.0,194.0 L736.0,186.0"/><text class="t-tick" x="648.0" y="212.0" text-anchor="middle">256 blue</text><text class="t-tick" x="200.0" y="162.0" text-anchor="start">all 768 numbers, in the order (channel, row, column):</text><text class="t-muted" x="20.0" y="236.0" text-anchor="start">number 0 = red at pixel (0,0) = 0.114; number 256 = green at (0,0) = -0.804; number 512 = blue at (0,0) = -0.545</text></svg><figcaption>One 16×16×3 patch becomes one row of 768 numbers: the 256 red values first, then 256 green, then 256 blue. The strip shows the real first 40 numbers of patch 0 (the red channel of pixel row 0, then the start of row 1); the darker the square, the larger the value.</figcaption></figure>

**The projection $$\mathbf{E}$$.** Each row of $$\mathbf{x}_p$$ is multiplied by the same matrix $$\mathbf{E} \in \mathbb{R}^{(P^2 \cdot C) \times D}$$. One matrix product does all 196 patches at once:

$$
\underbrace{\mathbf{x}_p}_{196 \times 768} \cdot \underbrace{\mathbf{E}}_{768 \times 768} = \underbrace{\mathbf{x}_p \mathbf{E}}_{196 \times 768}
$$

<figure class="fig"><svg viewBox="0 0 760 308" role="img" aria-label="The patch embedding as a matrix product: x_p with shape 196 by 768 times E with shape 768 by 768 gives the patch embeddings with shape 196 by 768; E has 589,824 weights plus 768 biases."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="28.0" text-anchor="start">The patch embedding is one matrix multiplication</text><rect class="box-ghost" x="36.0" y="66.0" width="154.8" height="154.8" rx="5"/><rect class="s1" x="40.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="50.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="61.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="71.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="82.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="93.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="103.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="114.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="124.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="135.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="146.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="156.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="167.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="177.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="40.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="50.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="61.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="71.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="82.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="93.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="103.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="114.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="124.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="135.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="146.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="156.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="167.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="177.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="40.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="50.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="61.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="71.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="82.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="93.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="103.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="114.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="124.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="135.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="146.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="156.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="167.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="177.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="40.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="50.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="61.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="71.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="82.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="93.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="103.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="114.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="124.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="135.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="146.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="156.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="167.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="177.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="40.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="50.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="61.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="71.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="82.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="93.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="103.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="114.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="124.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="135.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="146.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="156.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="167.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="177.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="40.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="50.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="61.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="71.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="82.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="93.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="103.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="114.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="124.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="135.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="146.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="156.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="167.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="177.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="40.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="50.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="61.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="71.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="82.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="93.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="103.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="114.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="124.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="135.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="146.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="156.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="167.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="177.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="40.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="50.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="61.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="71.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="82.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="93.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="103.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="114.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="124.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="135.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="146.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="156.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="167.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="177.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="40.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="50.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="61.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="71.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="82.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="93.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="103.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="114.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="124.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="135.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="146.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="156.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="167.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="177.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="40.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="50.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="61.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="71.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="82.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="93.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="103.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="114.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="124.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="135.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="146.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="156.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="167.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="177.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="40.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="50.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="61.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="71.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="82.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="93.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="103.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="114.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="124.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="135.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="146.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="156.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="167.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="177.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="40.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="50.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="61.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="71.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="82.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="93.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="103.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="114.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="124.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="135.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="146.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="156.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="167.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="177.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="40.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="50.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="61.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="71.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="82.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="93.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="103.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="114.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="124.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="135.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="146.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="156.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="167.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="177.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="40.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="50.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="61.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="71.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="82.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="93.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="103.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="114.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="124.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="135.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="146.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="156.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="167.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="177.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><text class="t-math" x="113.4" y="58.0" text-anchor="middle">x_p</text><text class="t-tick" x="113.4" y="236.8" text-anchor="middle">196 × 768</text><text class="t-big" x="208.8" y="149.4" text-anchor="middle">·</text><rect class="box-ghost" x="226.8" y="66.0" width="154.8" height="154.8" rx="5"/><rect class="s4" x="230.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="241.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="252.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="262.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="273.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="283.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="294.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="305.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="315.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="326.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="336.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="347.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="358.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="368.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="230.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="241.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="252.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="262.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="273.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="283.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="294.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="305.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="315.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="326.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="336.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="347.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="358.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="368.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="230.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="241.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="252.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="262.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="273.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="283.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="294.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="305.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="315.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="326.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="336.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="347.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="358.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="368.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="230.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="241.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="252.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="262.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="273.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="283.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="294.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="305.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="315.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="326.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="336.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="347.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="358.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="368.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="230.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="241.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="252.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="262.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="273.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="283.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="294.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="305.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="315.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="326.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="336.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="347.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="358.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="368.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="230.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="241.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="252.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="262.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="273.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="283.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="294.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="305.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="315.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="326.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="336.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="347.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="358.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="368.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="230.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="241.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="252.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="262.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="273.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="283.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="294.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="305.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="315.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="326.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="336.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="347.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="358.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="368.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="230.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="241.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="252.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="262.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="273.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="283.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="294.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="305.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="315.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="326.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="336.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="347.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="358.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="368.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="230.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="241.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="252.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="262.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="273.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="283.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="294.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="305.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="315.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="326.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="336.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="347.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="358.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="368.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="230.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="241.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="252.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="262.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="273.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="283.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="294.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="305.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="315.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="326.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="336.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="347.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="358.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="368.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="230.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="241.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="252.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="262.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="273.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="283.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="294.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="305.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="315.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="326.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="336.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="347.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="358.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="368.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="230.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="241.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="252.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="262.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="273.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="283.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="294.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="305.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="315.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="326.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="336.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="347.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="358.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="368.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="230.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="241.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="252.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="262.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="273.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="283.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="294.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="305.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="315.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="326.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="336.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="347.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="358.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="368.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="230.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="241.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="252.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="262.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="273.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="283.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="294.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="305.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="315.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="326.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="336.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="347.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="358.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="368.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><text class="t-math" x="304.2" y="58.0" text-anchor="middle">E</text><text class="t-tick" x="304.2" y="236.8" text-anchor="middle">768 × 768</text><text class="t-big" x="399.6" y="149.4" text-anchor="middle">=</text><rect class="box-ghost" x="417.6" y="66.0" width="154.8" height="154.8" rx="5"/><rect class="s3" x="421.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="432.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="442.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="453.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="464.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="474.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="485.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="495.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="506.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="517.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="527.6" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="538.2" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="548.8" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="559.4" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="432.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="442.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="453.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="464.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="474.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="485.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="495.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="506.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="517.0" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="527.6" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="538.2" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="548.8" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="559.4" y="80.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="432.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="442.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="453.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="464.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="474.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="485.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="495.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="506.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="517.0" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="527.6" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="538.2" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="548.8" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="559.4" y="91.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="421.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="432.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="442.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="453.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="464.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="474.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="485.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="495.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="506.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="517.0" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="527.6" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="538.2" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="548.8" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="559.4" y="101.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="421.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="432.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="442.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="453.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="464.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="474.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="485.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="495.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="506.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="517.0" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="527.6" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="538.2" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="548.8" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="559.4" y="112.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="421.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="432.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="442.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="453.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="464.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="474.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="485.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="495.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="506.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="517.0" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="527.6" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="538.2" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="548.8" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="559.4" y="123.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="432.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="442.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="453.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="464.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="474.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="485.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="495.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="506.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="517.0" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="527.6" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="538.2" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="548.8" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="559.4" y="133.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="432.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="442.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="453.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="464.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="474.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="485.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="495.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="506.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="517.0" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="527.6" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="538.2" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="548.8" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="559.4" y="144.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="421.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="432.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="442.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="453.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="464.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="474.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="485.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="495.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="506.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="517.0" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="527.6" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="538.2" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="548.8" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="559.4" y="154.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="421.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="432.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="442.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="453.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="464.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="474.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="485.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="495.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="506.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="517.0" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="527.6" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="538.2" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="548.8" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="559.4" y="165.4" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="421.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="432.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="442.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="453.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="464.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="474.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="485.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="495.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="506.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="517.0" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="527.6" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="538.2" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="548.8" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="559.4" y="176.0" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="432.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="442.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="453.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="464.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="474.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="485.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="495.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="506.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="517.0" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="527.6" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="538.2" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="548.8" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="559.4" y="186.6" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="432.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="442.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="453.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="464.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="474.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="485.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="495.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="506.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="517.0" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="527.6" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="538.2" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="548.8" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="559.4" y="197.2" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="421.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="432.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="442.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="453.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="464.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="474.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="485.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="495.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="506.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="517.0" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="527.6" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="538.2" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="548.8" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="559.4" y="207.8" width="9" height="9" rx="2" style="fill-opacity:0.42"/><text class="t-math" x="495.0" y="58.0" text-anchor="middle">x_p E</text><text class="t-tick" x="495.0" y="236.8" text-anchor="middle">196 × 768</text><text class="t-tick" x="598.4" y="80.0" text-anchor="start">Each row of x_p (one</text><text class="t-tick" x="598.4" y="96.0" text-anchor="start">patch, 768 numbers)</text><text class="t-tick" x="598.4" y="112.0" text-anchor="start">times E gives one row</text><text class="t-tick" x="598.4" y="128.0" text-anchor="start">of 768 embedding</text><text class="t-tick" x="598.4" y="144.0" text-anchor="start">numbers.</text><text class="t-tick" x="598.4" y="160.0" text-anchor="start"></text><text class="t-tick" x="598.4" y="176.0" text-anchor="start">E: 768 × 768 = 589,824</text><text class="t-tick" x="598.4" y="192.0" text-anchor="start">weights + 768 biases.</text><text class="t-muted" x="20.0" y="262.0" text-anchor="start">The embedding row for patch 0 starts 0.058, -0.024, -0.248, 3.689 …  (768 numbers in all).</text><text class="t-muted" x="20.0" y="279.0" text-anchor="start">D = 768 equals P²·C = 768 only by coincidence of ViT-B/16 at 224: for ViT-B/32 the rows are 32·32·3 = 3,072 long</text><text class="t-muted" x="20.0" y="296.0" text-anchor="start">and E is 3,072 × 768.</text></svg><figcaption>The patch embedding is one matrix multiplication. x_p (196×768) times E (768×768) gives 196 patch embeddings of 768 numbers. E holds 768 × 768 = 589,824 weights plus a bias of 768, and it is the only thing the model learns about pixels before the Transformer.</figcaption></figure>

There is one small gap between the paper and the released code here, and it is worth closing. The paper says "linear projection". The released PyTorch model (and the authors' original code) implement it as a **convolution** with a 16×16 kernel and a stride of 16. These are the same operation: a convolution with kernel size equal to its stride touches each patch exactly once, and what it does to a patch is a dot product with each of its 768 filters, which is a linear layer. I checked by reshaping the convolution's weight into a $$768 \times 768$$ matrix and multiplying the flattened patches with it:

```python
conv = model.vit.embeddings.patch_embeddings.projection    # Conv2d(3, 768, kernel_size=16, stride=16)
E = conv.weight.reshape(768, 768).T                        # (P²·C) × D: column d is filter d, flattened
ours = patches @ E + conv.bias                             # Eq. 1's x_p E, for all 196 patches
theirs = conv(pixel_values).flatten(2).transpose(1, 2)[0]  # what the library computes
print((ours - theirs).abs().max())
```

```text
== 2. the patch embedding E: the library's Conv2d(3, 768, kernel 16, stride 16) equals Linear(768 -> 768) on x_p ==
  conv weight (768, 3, 16, 16) reshaped to E (768, 768) (rows = the 768 pixel numbers of a patch, columns = D = 768)
  x_p E: (196 x 768) . (768 x 768) = (196, 768)
  max |difference|, our x_p E + b vs the library conv: 5.13e-06
  patch 0 embedding, first 6 of 768: [  0.058,  -0.024,  -0.248,   3.689,   0.356,   0.098]
```

A difference of $$5 \times 10^{-6}$$ is rounding noise in 32-bit arithmetic. So "linear projection of flattened patches" and "16×16 convolution with stride 16" are two names for the same 589,824 weights. The convolution is simply the faster way to run it.

<figure class="fig"><svg viewBox="0 0 760 250" role="img" aria-label="Two views of the same patch embedding: on the left a convolution with a 16 by 16 kernel and stride 16 placed on every patch, on the right a linear layer applied to the flattened patch rows; the weights are the same numbers reshaped and the outputs agree to 5e-6."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="6.0" y="10.0" width="366.0" height="230.0" rx="12"/><circle class="node on" cx="24.0" cy="28.0" r="11"/><text class="t-note" x="24.0" y="32.5" text-anchor="middle">1</text><text class="t-title" x="42.0" y="33.0" text-anchor="start">Library: Conv2d, kernel 16, stride 16</text><rect class="s1" x="30.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="48.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="66.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="84.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="102.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="120.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="138.0" y="54.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="30.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="48.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="66.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="84.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="102.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="120.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="138.0" y="72.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="30.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="48.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="66.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="84.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="102.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="120.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="138.0" y="90.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="30.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="48.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="66.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="84.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="102.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="120.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="138.0" y="108.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="30.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="48.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="66.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="84.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="102.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="120.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="138.0" y="126.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="30.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="48.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="66.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="84.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="102.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="120.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="138.0" y="144.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="30.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="48.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="66.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="84.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="120.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="138.0" y="162.0" width="17" height="17" rx="1.5" style="fill-opacity:0.65"/><rect class="box-on" x="66.0" y="90.0" width="17.0" height="17.0" rx="2"/><text class="t-tick" x="160.0" y="66.0" text-anchor="start">a 16×16×3 filter</text><text class="t-tick" x="160.0" y="82.0" text-anchor="start">placed on each patch</text><text class="t-tick" x="160.0" y="98.0" text-anchor="start">(stride 16: no overlap)</text><text class="t-tick" x="160.0" y="124.0" text-anchor="start">768 filters →</text><text class="t-tick" x="160.0" y="140.0" text-anchor="start">768 numbers per patch</text><text class="t-code" x="30.0" y="206.0" text-anchor="start">weight shape (768, 3, 16, 16)</text><text class="t-code" x="30.0" y="226.0" text-anchor="start">output (768, 14, 14) → (196, 768)</text><rect class="box" x="388.0" y="10.0" width="366.0" height="230.0" rx="12"/><circle class="node on" cx="406.0" cy="28.0" r="11"/><text class="t-note" x="406.0" y="32.5" text-anchor="middle">2</text><text class="t-title" x="424.0" y="33.0" text-anchor="start">Eq. 1: a Linear layer on x_p</text><rect class="box-ghost" x="407.0" y="67.0" width="136.0" height="15.0" rx="4"/><g class="mark"><rect class="s2" x="410.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="421.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="432.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="443.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="454.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="465.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="476.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="487.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="498.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="509.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="520.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s2" x="531.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><text class="t-tick" x="410.0" y="98.0" text-anchor="start">x_p row: 768 numbers</text><line class="edge" x1="546.0" y1="74.0" x2="566.0" y2="74.0" marker-end="url(#ah)"/><rect class="box-4" x="568.0" y="56.0" width="60.0" height="36.0" rx="6"/><text class="t-math" x="598.0" y="72.0" text-anchor="middle">E</text><text class="t-cell" x="598.0" y="86.0" text-anchor="middle">768×768</text><line class="edge" x1="630.0" y1="74.0" x2="650.0" y2="74.0" marker-end="url(#ah)"/><rect class="box-ghost" x="649.0" y="67.0" width="81.0" height="15.0" rx="4"/><g class="mark"><rect class="s4" x="652.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="663.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="674.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="685.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="696.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="707.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><g class="mark"><rect class="s4" x="718.0" y="70.0" width="9" height="9" rx="2" style="fill-opacity:0.95"/></g><text class="t-code" x="410.0" y="140.0" text-anchor="start">E = conv.weight.reshape(768, 768).T</text><text class="t-code" x="410.0" y="160.0" text-anchor="start">x_p @ E + bias</text><text class="t-tick" x="410.0" y="190.0" text-anchor="start">Same 589,824 weights, just reshaped.</text><text class="t-note" x="410.0" y="210.0" text-anchor="start">max |difference|: 5.1e-06</text></svg><figcaption>Two views of the same patch embedding. Left: the library's convolution places a 16×16×3 filter on each patch with stride 16, so there is no overlap, and 768 filters give 768 numbers per patch. Right: Eq. 1's linear layer on the flattened rows. The weights are the same numbers reshaped; the outputs agree to 5e-6.</figcaption></figure>

## The [class] token and the head {§3.1}

> [!PAPER] · §3.1
> [![The second paragraph of Section 3.1: similar to BERT's class token, we prepend a learnable embedding to the sequence of embedded patches (z 0 0 equals x class), whose state at the output of the Transformer encoder (z L 0) serves as the image representation y (Eq. 4). Both during pre-training and fine-tuning, a classification head is attached to z L 0. The classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time](https://ishwarj.com/img/papers/vit/p2-vit-class.pngit/p2-vit-class.png)](/img/papers/vit/p2-vit-class.png)
>
> **What it says:** "Similar to BERT's [class] token, we prepend a learnable embedding to the sequence of embedded patches ($$\mathbf{z}_0^0 = \mathbf{x}_{\text{class}}$$)". Its state at the encoder output, $$\mathbf{z}_L^0$$, "serves as the image representation $$\mathbf{y}$$ (Eq. 4)". A classification head sits on $$\mathbf{z}_L^0$$ both in pre-training and fine-tuning; it is "a MLP with one hidden layer at pre-training time and ... a single linear layer at fine-tuning time".
>
> **What it means:**
>
> The [class] token is the one token that is not a patch. Think of a reporter at a meeting who has no news of their own. They listen to everyone, and at the end only their notes are read out.
>
> So the model does not average the 196 patch outputs. It reads one token, which never saw a pixel directly and must gather everything through attention. This is copied from BERT. Appendix D.3 of the paper (Part 5) shows that averaging the patch outputs would also have worked, with a different learning rate.

> [!DEFINITION] [class] token
> An extra token at position 0 that belongs to no patch. Its input vector $$\mathbf{x}_{\text{class}}$$ is a learned list of 768 numbers, the same for every picture. Through 12 layers of attention it collects information from all patches, and its output vector is what the classifier reads. BERT calls the same thing `[CLS]`.

> [!DEFINITION] Learnable embedding
> A vector of numbers that is a parameter of the model: it starts random and is adjusted by training like any weight. The class token and the position embeddings are learnable embeddings. Nothing computes them from the input; they are simply looked up.

> [!DEFINITION] Image representation
> The one vector that stands for the whole picture, here $$\mathbf{y} = \text{LN}(\mathbf{z}_L^0)$$, 768 numbers. Everything the classifier knows about the picture must be in it.

> [!DEFINITION] Classification head
> The last, small piece of the model that turns the image representation into one score per class. For 1,000 ImageNet classes and $$D = 768$$ a linear head is a $$768 \times 1000$$ matrix and 1,000 biases.

> [!DEFINITION] Logits
> The raw scores the head outputs, one per class, before they are turned into probabilities. Softmax (defined below) converts them: the largest logit becomes the largest probability.

Here is the BERT sentence the paper points to:

> [!PAPER] Devlin et al. (2018), BERT · §3, Input/Output Representations
> [![The BERT paper: the first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks](https://ishwarj.com/img/papers/vit/p2-ext-bert-cls.pngp2-ext-bert-cls.png)](/img/papers/vit/p2-ext-bert-cls.png)
>
> **What it says:** "The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks."
>
> **What it means:**
>
> This is the paragraph of the BERT paper that introduces `[CLS]`. ViT took the sentence almost as it is and renamed the token.
>
> "Aggregate sequence representation" becomes "image representation". The idea is the same in both papers: one extra token collects the whole input, and the classifier reads only that token.

The two heads can be seen in the released checkpoints. The fine-tuned model `google/vit-base-patch16-224` ends in exactly one linear layer, and the pre-trained-only model `google/vit-base-patch16-224-in21k` ends in a dense layer with a tanh (a "pooler", the hidden layer of the pre-training MLP head; the 21,843-class output layer on top of it was not released):

```text
  head of the fine-tuned checkpoint: Linear(in_features=768, out_features=1000, bias=True)   (one linear layer, as Section 3.1 says for fine-tuning)
  the pre-trained-only checkpoint google/vit-base-patch16-224-in21k ends in: ViTPooler(
  (dense): Linear(in_features=768, out_features=768, bias=True)
  (activation): Tanh()
)
  fine-tuned checkpoint has a pooler: False;  in21k checkpoint has a pooler: True
  (the paper: an MLP with one hidden layer at pre-training time, a single linear layer at fine-tuning time)
```

So the released files match the sentence: an MLP with one hidden layer for pre-training, one linear layer after fine-tuning. [Part 3](https://ishwarj.com/papers/vit/part-3-fine-tuning-and-setup/) covers how the head is swapped and why it starts from zeros.

## Position embeddings {§3.1}

> [!PAPER] · §3.1
> [![The third paragraph of Section 3.1: position embeddings are added to the patch embeddings to retain positional information. We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.4). The resulting sequence of embedding vectors serves as input to the encoder](https://ishwarj.com/img/papers/vit/p2-vit-pos.png/vit/p2-vit-pos.png)](/img/papers/vit/p2-vit-pos.png)
>
> **What it says:** position embeddings "are added to the patch embeddings to retain positional information". They are "standard learnable 1D position embeddings", because "more advanced 2D-aware position embeddings (Appendix D.4)" did not help significantly. The sum "serves as input to the encoder".
>
> **What it means:**
>
> Think of a pile of stamps cut from a photo, with no note of where each one came from. A position embedding is that note. Each position gets its own learned vector, added to the patch at that place.
>
> "1D" means the model is told "you are token 37", not "you are at row 2, column 9". It is like seat numbers in a cinema with no row letters. The model must work out the grid from data. Appendix D.4, covered in [Part 5](https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vit/part-5-scaling-and-inside-vit.md), shows that it does: the learned position vectors end up arranged in a grid.

> [!DEFINITION] Position embedding
> A learned vector for each position in the sequence, added to the token at that position. Attention on its own does not know the order of its inputs (we show this below), so without these vectors the model could not tell the top-left patch from the bottom-right one. ViT-B/16 at 224 learns 197 of them, one per patch plus one for the class token.

Why are they needed at all? Because self-attention treats its input as a *set*. Shuffle the input rows and the output rows shuffle in exactly the same way, with the same numbers. We check this on the tiny four-token example of the Appendix A section below ([`vit_part2_attn.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part2_attn.py)):

```text
== permutation equivariance: shuffle the input rows, the output rows shuffle the same way ==
  new order of the tokens: ['t2', 't0', 't3', 't1']
  MSA(z shuffled)  shape (4, 6)      (MSA(z) in the original order is printed in the Appendix A section)
       t2  1.784  2.885  0.000 -0.442 -2.007 -1.173
       t0  1.510  0.045  0.000  0.070 -1.023 -0.240
       t3  2.091  1.299  0.000 -0.228 -1.766 -0.560
       t1  1.679  0.231  0.000 -0.155 -1.116 -0.110
  MSA(z shuffled) == MSA(z) with its rows shuffled the same way: True   (max |difference| 4.8e-07)
  so without position embeddings the model cannot tell where a patch came from: only the set of patches matters.
  with a position row added before attention (z + E_pos), the shuffled run no longer matches: max |difference| 3.875
```

Compare with the original-order output in the Appendix A section: the rows for t2 and t0 swap places and nothing else changes. A model built only from attention and per-token MLPs would give the same class to a picture and to the same picture with its patches scrambled. Adding a different vector to each position breaks the tie: the shuffled run now differs by up to 3.875.

<figure class="fig"><svg viewBox="0 0 760 262" role="img" aria-label="Permutation equivariance of multi-head self-attention in the tiny example: the four input tokens in the original order and in a shuffled order each go through MSA, and the output rows are the same numbers in the shuffled order, joined by dashed lines; adding position embeddings breaks this."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Reorder the input rows: the output rows reorder the same way (no positions)</text><text class="t-note" x="170.0" y="52.0" text-anchor="middle">original order</text><rect class="box-1" x="20.0" y="66.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="45.0" y="83.5" text-anchor="middle">t0</text><line class="edge" x1="72.0" y1="79.0" x2="90.0" y2="79.0" marker-end="url(#ah)"/><rect class="box" x="92.0" y="66.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="122.0" y="83.0" text-anchor="middle">MSA</text><line class="edge" x1="154.0" y1="79.0" x2="172.0" y2="79.0" marker-end="url(#ah)"/><text class="t-code" x="176.0" y="83.0" text-anchor="start">[1.51, 0.04, 0.00, …]</text><rect class="box-1" x="20.0" y="100.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="45.0" y="117.5" text-anchor="middle">t1</text><line class="edge" x1="72.0" y1="113.0" x2="90.0" y2="113.0" marker-end="url(#ah)"/><rect class="box" x="92.0" y="100.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="122.0" y="117.0" text-anchor="middle">MSA</text><line class="edge" x1="154.0" y1="113.0" x2="172.0" y2="113.0" marker-end="url(#ah)"/><text class="t-code" x="176.0" y="117.0" text-anchor="start">[1.68, 0.23, 0.00, …]</text><rect class="box-1" x="20.0" y="134.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="45.0" y="151.5" text-anchor="middle">t2</text><line class="edge" x1="72.0" y1="147.0" x2="90.0" y2="147.0" marker-end="url(#ah)"/><rect class="box" x="92.0" y="134.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="122.0" y="151.0" text-anchor="middle">MSA</text><line class="edge" x1="154.0" y1="147.0" x2="172.0" y2="147.0" marker-end="url(#ah)"/><text class="t-code" x="176.0" y="151.0" text-anchor="start">[1.78, 2.88, 0.00, …]</text><rect class="box-1" x="20.0" y="168.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="45.0" y="185.5" text-anchor="middle">t3</text><line class="edge" x1="72.0" y1="181.0" x2="90.0" y2="181.0" marker-end="url(#ah)"/><rect class="box" x="92.0" y="168.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="122.0" y="185.0" text-anchor="middle">MSA</text><line class="edge" x1="154.0" y1="181.0" x2="172.0" y2="181.0" marker-end="url(#ah)"/><text class="t-code" x="176.0" y="185.0" text-anchor="start">[2.09, 1.30, 0.00, …]</text><text class="t-note" x="550.0" y="52.0" text-anchor="middle">shuffled: t2, t0, t3, t1</text><rect class="box-1" x="400.0" y="66.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="425.0" y="83.5" text-anchor="middle">t2</text><line class="edge" x1="452.0" y1="79.0" x2="470.0" y2="79.0" marker-end="url(#ah)"/><rect class="box" x="472.0" y="66.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="502.0" y="83.0" text-anchor="middle">MSA</text><line class="edge" x1="534.0" y1="79.0" x2="552.0" y2="79.0" marker-end="url(#ah)"/><text class="t-code" x="556.0" y="83.0" text-anchor="start">[1.78, 2.88, 0.00, …]</text><rect class="box-1" x="400.0" y="100.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="425.0" y="117.5" text-anchor="middle">t0</text><line class="edge" x1="452.0" y1="113.0" x2="470.0" y2="113.0" marker-end="url(#ah)"/><rect class="box" x="472.0" y="100.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="502.0" y="117.0" text-anchor="middle">MSA</text><line class="edge" x1="534.0" y1="113.0" x2="552.0" y2="113.0" marker-end="url(#ah)"/><text class="t-code" x="556.0" y="117.0" text-anchor="start">[1.51, 0.04, 0.00, …]</text><rect class="box-1" x="400.0" y="134.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="425.0" y="151.5" text-anchor="middle">t3</text><line class="edge" x1="452.0" y1="147.0" x2="470.0" y2="147.0" marker-end="url(#ah)"/><rect class="box" x="472.0" y="134.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="502.0" y="151.0" text-anchor="middle">MSA</text><line class="edge" x1="534.0" y1="147.0" x2="552.0" y2="147.0" marker-end="url(#ah)"/><text class="t-code" x="556.0" y="151.0" text-anchor="start">[2.09, 1.30, 0.00, …]</text><rect class="box-1" x="400.0" y="168.0" width="50.0" height="26.0" rx="6"/><text class="t-cell" x="425.0" y="185.5" text-anchor="middle">t1</text><line class="edge" x1="452.0" y1="181.0" x2="470.0" y2="181.0" marker-end="url(#ah)"/><rect class="box" x="472.0" y="168.0" width="60.0" height="26.0" rx="6"/><text class="t-cell" x="502.0" y="185.0" text-anchor="middle">MSA</text><line class="edge" x1="534.0" y1="181.0" x2="552.0" y2="181.0" marker-end="url(#ah)"/><text class="t-code" x="556.0" y="185.0" text-anchor="start">[1.68, 0.23, 0.00, …]</text><path class="edge-dim" d="M320,147 C360,147 360,79 396,79"/><path class="edge-dim" d="M320,79 C360,79 360,113 396,113"/><path class="edge-dim" d="M320,181 C360,181 360,147 396,147"/><path class="edge-dim" d="M320,113 C360,113 360,181 396,181"/><text class="t-muted" x="20.0" y="216.0" text-anchor="start">Identical numbers, only the rows moved (max |difference| under 1e-6): attention only sees a set of tokens.</text><text class="t-muted" x="20.0" y="233.0" text-anchor="start">Add a position row to each token before attention and the shuffled run differs (max |difference| 3.875):</text><text class="t-muted" x="20.0" y="250.0" text-anchor="start">now the order matters.</text></svg><figcaption>Permutation equivariance in the tiny example. The four tokens go through multi-head self-attention in the original order (left) and in a shuffled order (right); the dashed lines join equal output rows. The numbers are identical, only the order moved. With a position row added to each token first, the two runs differ.</figcaption></figure>

## The encoder: LayerNorm, attention, MLP {§3.1}

> [!PAPER] · §3.1
> [![The fourth paragraph of Section 3.1: the Transformer encoder (Vaswani et al., 2017) consists of alternating layers of multiheaded self-attention (MSA, see Appendix A) and MLP blocks (Eq. 2, 3). Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al., 2019; Baevski and Auli, 2019)](https://ishwarj.com/img/papers/vit/p2-vit-encoder.png/p2-vit-encoder.png)](/img/papers/vit/p2-vit-encoder.png)
> [![The paragraph continues at the top of page 4: the MLP contains two layers with a GELU non-linearity, followed by Equations 1 to 4](https://ishwarj.com/img/papers/vit/p2-eq1-4.pngrs/vit/p2-eq1-4.png)](/img/papers/vit/p2-eq1-4.png)
>
> **What it says:** the encoder "consists of alternating layers of multiheaded self-attention (MSA, see Appendix A) and MLP blocks (Eq. 2, 3)". "Layernorm (LN) is applied before every block, and residual connections after every block", citing Wang et al. (2019) and Baevski and Auli (2019) for this pre-norm arrangement. "The MLP contains two layers with a GELU non-linearity."
>
> **What it means:**
>
> This is the standard Transformer encoder with one modern change. LayerNorm comes *before* each block, not after it. The two cited papers found that this order makes deep Transformers train more stably. BERT and the 2017 Transformer normalise after.
>
> Think of the residual path as a main road and each block as a side road. In pre-norm, each side road cleans its own traffic before it joins, and the main road changes only by addition. Everything else is the 2017 design. The four equations then write the whole model down in symbols.

> [!DEFINITION] Multi-head self-attention (MSA)
> Several self-attention computations ("heads") run side by side on the same input, each with its own small set of weights, and their outputs joined and mixed by one more matrix. ViT-B runs 12 heads of 64 numbers each. Appendix A, at the end of this part, gives the equations.

> [!DEFINITION] LayerNorm (and why pre-norm)
> A step that takes one token's vector, subtracts its mean and divides by its spread so the 768 numbers have mean 0 and spread 1, then multiplies by a learned vector $$\gamma$$ and adds a learned vector $$\beta$$. It keeps the numbers in a stable range. **Pre-norm** puts it at the entrance of each block, so what is added back on the residual path is never normalised and the main path stays a clean running sum; this trains more stably in deep stacks. **Post-norm** (BERT, the original Transformer) normalises after the addition.

> [!DEFINITION] Residual connection
> Adding a block's input to its output: $$\text{out} = \text{in} + \text{block}(\text{in})$$. The block only has to learn a correction, and information can flow straight through the stack. The two plus signs in Figure 1 are residual connections.

> [!DEFINITION] MLP / feed-forward block
> Two linear layers with a non-linearity between them, applied to each token on its own: $$768 \to 3{,}072 \to 768$$ in ViT-B. It is the part of the layer that does not look at other tokens. The Transformer paper calls it the feed-forward network.

> [!DEFINITION] GELU
> The Gaussian Error Linear Unit (Hendrycks and Gimpel, 2016), $$\text{GELU}(x) = x \cdot \Phi(x)$$ where $$\Phi$$ is the standard normal cumulative distribution. It is a smooth cousin of ReLU ($$\max(0, x)$$): large positive numbers pass through, large negative ones become almost 0, and small negative ones come out slightly negative instead of exactly 0.

<figure class="fig"><svg viewBox="0 0 760 266" role="img" aria-label="One ViT encoder layer drawn left to right: the input passes through LayerNorm and multi-head self-attention and is added back to itself (Equation 2), then through LayerNorm and the MLP and is added back again (Equation 3); the two residual paths arc over the blocks."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">One encoder layer: LayerNorm before each block, residual after it (Eq. 2, 3)</text><rect class="box" x="20.0" y="110.0" width="62.0" height="34.0" rx="6"/><text class="t-tick" x="51.0" y="131.5" text-anchor="middle">z_{l-1}</text><rect class="box-3" x="104.0" y="110.0" width="48.0" height="34.0" rx="6"/><text class="t-tick" x="128.0" y="131.5" text-anchor="middle">LN</text><rect class="box-1" x="174.0" y="110.0" width="70.0" height="34.0" rx="6"/><text class="t-tick" x="209.0" y="131.5" text-anchor="middle">MSA</text><circle class="node" cx="280" cy="127" r="13"/><text class="t-note" x="280.0" y="132.0" text-anchor="middle">+</text><rect class="box" x="316.0" y="110.0" width="50.0" height="34.0" rx="6"/><text class="t-tick" x="341.0" y="131.5" text-anchor="middle">z'_l</text><rect class="box-3" x="388.0" y="110.0" width="48.0" height="34.0" rx="6"/><text class="t-tick" x="412.0" y="131.5" text-anchor="middle">LN</text><rect class="box-4" x="458.0" y="110.0" width="70.0" height="34.0" rx="6"/><text class="t-tick" x="493.0" y="131.5" text-anchor="middle">MLP</text><circle class="node" cx="564" cy="127" r="13"/><text class="t-note" x="564.0" y="132.0" text-anchor="middle">+</text><rect class="box" x="600.0" y="110.0" width="50.0" height="34.0" rx="6"/><text class="t-tick" x="625.0" y="131.5" text-anchor="middle">z_l</text><line class="edge" x1="84.0" y1="127.0" x2="102.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="154.0" y1="127.0" x2="172.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="246.0" y1="127.0" x2="264.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="296.0" y1="127.0" x2="314.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="368.0" y1="127.0" x2="386.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="438.0" y1="127.0" x2="456.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="530.0" y1="127.0" x2="548.0" y2="127.0" marker-end="url(#ah)"/><line class="edge" x1="580.0" y1="127.0" x2="598.0" y2="127.0" marker-end="url(#ah)"/><path class="path" d="M51.0,110 C51.0,50 280.0,50 280.0,112" marker-end="url(#ah-on)"/><text class="t-tick" x="165.5" y="62.0" text-anchor="middle">residual: + z_{l-1}</text><path class="path" d="M341.0,110 C341.0,50 564.0,50 564.0,112" marker-end="url(#ah-on)"/><text class="t-tick" x="452.5" y="62.0" text-anchor="middle">residual: + z'_l</text><path class="edge" d="M20.0,152.0 L20.0,160.0 L366.0,160.0 L366.0,152.0"/><text class="t-tick" x="193.0" y="178.0" text-anchor="middle">Eq. 2:  z'_l = MSA(LN(z_{l-1})) + z_{l-1}</text><path class="edge" d="M316.0,186.0 L316.0,194.0 L650.0,194.0 L650.0,186.0"/><text class="t-tick" x="483.0" y="212.0" text-anchor="middle">Eq. 3:  z_l = MLP(LN(z'_l)) + z'_l</text><text class="t-muted" x="20.0" y="236.0" text-anchor="start">Every box keeps the shape 197 × 768. LN and the MLP work on each token on its own;</text><text class="t-muted" x="20.0" y="253.0" text-anchor="start">only MSA mixes information between tokens.</text></svg><figcaption>One encoder layer, left to right. The input from the previous layer goes through LayerNorm and multi-head self-attention and is added back to itself (Eq. 2, the first residual arc), then through LayerNorm and the MLP and is added back again (Eq. 3, the second arc). Every box keeps the shape 197×768. LayerNorm and the MLP work on one token at a time; only MSA mixes tokens.</figcaption></figure>

## Equations 1 to 4, with the real numbers {§3.1}

The four equations are the whole model. We take them in order and, for each, run the real ViT-B/16 on the cat picture by hand.

### Equation 1: the input sequence

The idea in one sentence: put the class token in front of the 196 projected patches, then add a position vector to each of the 197.

$$
\mathbf{z}_0 = [\mathbf{x}_{\text{class}};\ \mathbf{x}_p^1 \mathbf{E};\ \mathbf{x}_p^2 \mathbf{E};\ \cdots;\ \mathbf{x}_p^N \mathbf{E}] + \mathbf{E}_{pos}, \qquad \mathbf{E} \in \mathbb{R}^{(P^2 \cdot C) \times D},\ \mathbf{E}_{pos} \in \mathbb{R}^{(N+1) \times D}
$$

where:

- $$\mathbf{x}_p^n$$ is row $$n$$ of the patch matrix, the 768 pixel numbers of patch $$n$$ ($$1 \times 768$$);
- $$\mathbf{E}$$ is the projection, $$768 \times 768$$ here (the released model also adds a bias of 768, which the equation leaves out);
- $$\mathbf{x}_p^n \mathbf{E}$$ is the patch embedding of patch $$n$$, $$1 \times 768$$;
- $$\mathbf{x}_{\text{class}}$$ is the learned class token, $$1 \times 768$$;
- the square brackets stack the 197 rows into a $$197 \times 768$$ matrix;
- $$\mathbf{E}_{pos}$$ is the table of 197 learned position vectors, $$197 \times 768$$, added row by row: row 0 to the class token, row $$n$$ to patch $$n$$;
- $$\mathbf{z}_0$$ is the result, $$197 \times 768$$, the input of layer 1.

In code, with the library's own tables and our own patch embeddings from above:

```python
emb = model.vit.embeddings
x_class = emb.cls_token.detach()[0]                 # (1, 768)
E_pos = emb.position_embeddings.detach()[0]         # (197, 768)
z0 = torch.cat([x_class, our_patch_emb], 0) + E_pos # (197, 768)   Eq. 1
print((z0 - emb(pixel_values)[0]).abs().max())      # against the library's embedding layer
```

```text
== 3. Eq. 1: z_0 = [x_class; x_p^1 E; ...; x_p^N E] + E_pos ==
  x_class (1, 768)   [x_class; patches] (197, 768)   E_pos (197, 768)   z_0 (197, 768)
  x_class, first 6:        [  0.010,   0.015,  -0.267,  -0.001,   0.405,   0.054]
  E_pos[0] (for [class]):  [  0.010,   0.015,  -0.267,  -0.000,   0.406,   0.054]
  z_0[0] = x_class + E_pos[0]: [  0.020,   0.030,  -0.534,  -0.001,   0.811,   0.108]
  E_pos[1] (for patch 1):  [  0.156,  -0.124,   0.447,   0.009,   0.428,  -0.462]
  z_0[1] = x_p^1 E + E_pos[1]: [  0.214,  -0.148,   0.198,   3.698,   0.783,  -0.365]
  max |difference|, our z_0 vs the library embedding output: 5.13e-06
```

Check the second row by hand: patch 1's embedding started $$0.058, -0.024, -0.248, 3.689, \dots$$ (step 2 above); add $$\mathbf{E}_{pos}[1] = 0.156, -0.124, 0.447, 0.009, \dots$$ and you get $$0.214, -0.148, 0.198, 3.698$$, which is what the script prints. The class token and its position vector are a curiosity: in this checkpoint they are almost identical lists of numbers ($$0.010, 0.015, -0.267, \dots$$ in both), so $$\mathbf{z}_0[0]$$ is close to $$2\,\mathbf{x}_{\text{class}}$$. Nothing in the model requires that; the two vectors are only ever used as a sum, so training was free to split the sum between them any way it liked.

<figure class="fig"><svg viewBox="0 0 760 348" role="img" aria-label="Equation 1 drawn: the learned class embedding is prepended to the 196 patch embeddings, the 197 learned position rows are added, and the result is z_0 with shape 197 by 768; the real first six numbers of the class position are listed."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Eq. 1 with the real shapes and numbers</text><rect class="box-2" x="110.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="140.0" y="62.5" text-anchor="middle">x_class</text><rect class="box-1" x="176.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="206.0" y="62.5" text-anchor="middle">x_p¹E</text><rect class="box-1" x="242.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="272.0" y="62.5" text-anchor="middle">x_p²E</text><rect class="box-1" x="308.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="338.0" y="62.5" text-anchor="middle">x_p³E</text><rect class="box-1" x="374.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="404.0" y="62.5" text-anchor="middle">x_p⁴E</text><rect class="box-1" x="440.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="470.0" y="62.5" text-anchor="middle">…</text><rect class="box-1" x="506.0" y="44.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="536.0" y="62.5" text-anchor="middle">x_p¹⁹⁶E</text><text class="t-tick" x="100.0" y="62.0" text-anchor="end">[ … ]</text><text class="t-tick" x="576.0" y="62.0" text-anchor="start">197 × 768</text><rect class="box-3" x="110.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="140.0" y="110.5" text-anchor="middle">E_pos[0]</text><rect class="box-3" x="176.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="206.0" y="110.5" text-anchor="middle">E_pos[1]</text><rect class="box-3" x="242.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="272.0" y="110.5" text-anchor="middle">E_pos[2]</text><rect class="box-3" x="308.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="338.0" y="110.5" text-anchor="middle">E_pos[3]</text><rect class="box-3" x="374.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="404.0" y="110.5" text-anchor="middle">E_pos[4]</text><rect class="box-3" x="440.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="470.0" y="110.5" text-anchor="middle">…</text><rect class="box-3" x="506.0" y="92.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="536.0" y="110.5" text-anchor="middle">E_pos[196]</text><text class="t-big" x="100.0" y="110.0" text-anchor="end">+</text><text class="t-tick" x="576.0" y="110.0" text-anchor="start">197 × 768</text><rect class="box" x="110.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="140.0" y="158.5" text-anchor="middle">z₀[0]</text><rect class="box" x="176.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="206.0" y="158.5" text-anchor="middle">z₀[1]</text><rect class="box" x="242.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="272.0" y="158.5" text-anchor="middle">z₀[2]</text><rect class="box" x="308.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="338.0" y="158.5" text-anchor="middle">z₀[3]</text><rect class="box" x="374.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="404.0" y="158.5" text-anchor="middle">z₀[4]</text><rect class="box" x="440.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="470.0" y="158.5" text-anchor="middle">…</text><rect class="box" x="506.0" y="140.0" width="60.0" height="28.0" rx="6"/><text class="t-cell" x="536.0" y="158.5" text-anchor="middle">z₀[196]</text><text class="t-big" x="100.0" y="158.0" text-anchor="end">=</text><text class="t-tick" x="576.0" y="158.0" text-anchor="start">197 × 768</text><line class="edge" x1="140.0" y1="172.0" x2="140.0" y2="188.0" marker-end="url(#ah)"/><text class="t-tick" x="20.0" y="206.0" text-anchor="start">the [class] position, first 6 of 768 numbers:</text><text class="t-code" x="20.0" y="226.0" text-anchor="start">x_class    [ 0.010,  0.015, -0.267, -0.001,  0.405,  0.054, …]</text><text class="t-code" x="20.0" y="244.0" text-anchor="start">E_pos[0]   [ 0.010,  0.015, -0.267, -0.000,  0.406,  0.054, …]</text><text class="t-code" x="20.0" y="262.0" text-anchor="start">z_0[0]     [ 0.019,  0.030, -0.534, -0.001,  0.811,  0.108, …]</text><text class="t-muted" x="20.0" y="288.0" text-anchor="start">the first patch: x_p¹E [ 0.058, -0.024, -0.248,  3.689,  0.356,  0.098, …]</text><text class="t-muted" x="20.0" y="304.0" text-anchor="start">+ E_pos[1] [ 0.156, -0.124,  0.447,  0.009,  0.428, -0.462, …]</text><text class="t-muted" x="20.0" y="320.0" text-anchor="start">= z_0[1] [ 0.214, -0.148,  0.198,  3.698,  0.783, -0.364, …]</text><text class="t-muted" x="20.0" y="336.0" text-anchor="start">Our z_0 matches the library embedding output to 5.1e-06.</text></svg><figcaption>Equation 1 drawn with the real shapes. Top row: the learned class embedding and the 196 patch embeddings, 197 rows of 768. Middle: the 197 learned position rows. Bottom: their sum z_0. Below, the real first six numbers of the class position; our z_0 matches the library's embedding output to 5e-6.</figcaption></figure>

### Equation 2: LayerNorm, attention, residual

The idea in one sentence: normalise every token, let the tokens look at each other, and add the result back to what came in.

$$
\mathbf{z}'_\ell = \text{MSA}(\text{LN}(\mathbf{z}_{\ell-1})) + \mathbf{z}_{\ell-1}, \qquad \ell = 1 \dots L
$$

where:

- $$\mathbf{z}_{\ell-1}$$ is the output of the previous layer ($$\mathbf{z}_0$$ for the first), $$197 \times 768$$;
- $$\text{LN}$$ normalises each of the 197 rows on its own (formula below), shape unchanged;
- $$\text{MSA}$$ is multi-head self-attention, Appendix A, $$197 \times 768$$ in and out;
- $$+\ \mathbf{z}_{\ell-1}$$ is the residual connection;
- $$\mathbf{z}'_\ell$$ is the intermediate result, $$197 \times 768$$, which Equation 3 continues; $$L = 12$$ for ViT-B.

**LayerNorm by hand.** For one token's vector $$x$$ of $$D$$ numbers:

$$
\mu = \frac{1}{D}\sum_{i=1}^{D} x_i, \qquad \sigma = \sqrt{\frac{1}{D}\sum_{i=1}^{D}(x_i - \mu)^2 + \epsilon}, \qquad \text{LN}(x)_i = \gamma_i\,\frac{x_i - \mu}{\sigma} + \beta_i
$$

where $$\mu$$ is the mean of the 768 numbers, $$\sigma$$ their standard deviation (with a tiny $$\epsilon = 10^{-12}$$ added so that it is never zero), and $$\gamma, \beta$$ are two learned vectors of 768 numbers, the same for every token. Here it is on the class token entering layer 1:

```python
v = z0[0]                                                  # the [class] token, 768 numbers
ln = model.vit.layers[0].layernorm_before
mu, var = v.mean(), ((v - v.mean()) ** 2).mean()
ours = ln.weight * (v - mu) / torch.sqrt(var + 1e-12) + ln.bias
```

```text
  LayerNorm of the [class] token (768 numbers), eps = 1e-12:
    input, first 6:            [  0.020,   0.030,  -0.534,  -0.001,   0.811,   0.108]
    mean mu = 0.0034   variance = 0.0349   std = sqrt(var + eps) = 0.1869
    (x - mu)/std, first 6:     [  0.086,   0.141,  -2.875,  -0.023,   4.320,   0.561]
    gamma, first 6:            [  0.156,   0.191,   0.033,   0.119,   0.065,   0.126]
    beta, first 6:             [  0.008,   0.016,   0.091,   0.067,   0.007,  -0.034]
    gamma*(..)+beta, first 6:  [  0.022,   0.043,  -0.005,   0.064,   0.287,   0.036]
    library LayerNorm:         [  0.022,   0.043,  -0.005,   0.064,   0.287,   0.036]   max |difference| 6.0e-08
```

Check the fifth number: $$(0.811 - 0.0034)/0.1869 = 4.321$$, then $$0.065 \times 4.321 + 0.007 = 0.288$$, against the printed 0.287 (the inputs shown are rounded). Notice how small $$\gamma$$ is here (0.03 to 0.19): the trained model scales the normalised numbers down a lot before attention sees them.

<figure class="fig"><svg viewBox="0 0 760 276" role="img" aria-label="LayerNorm step by step on the first six numbers of the class token: the input, minus the mean, divided by the standard deviation, then multiplied by the learned gamma and shifted by the learned beta, with the real values from ViT-B/16."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">LayerNorm on the [class] token entering layer 1, first 6 of its 768 numbers</text><text class="t-tick" x="165.0" y="48.0" text-anchor="middle">[0]</text><text class="t-tick" x="235.0" y="48.0" text-anchor="middle">[1]</text><text class="t-tick" x="305.0" y="48.0" text-anchor="middle">[2]</text><text class="t-tick" x="375.0" y="48.0" text-anchor="middle">[3]</text><text class="t-tick" x="445.0" y="48.0" text-anchor="middle">[4]</text><text class="t-tick" x="515.0" y="48.0" text-anchor="middle">[5]</text><text class="t-tick" x="122.0" y="71.0" text-anchor="end">x (input)</text><rect class="cell" x="131.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="165.0" y="71.0" text-anchor="middle">0.019</text><rect class="cell" x="201.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="235.0" y="71.0" text-anchor="middle">0.030</text><rect class="s2" x="271.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.15"/><text class="t-cell" x="305.0" y="71.0" text-anchor="middle">-0.534</text><rect class="s2" x="341.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="375.0" y="71.0" text-anchor="middle">-0.001</text><rect class="cell" x="411.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.19"/><text class="t-cell" x="445.0" y="71.0" text-anchor="middle">0.811</text><rect class="cell" x="481.0" y="55.0" width="68" height="24" rx="3" style="fill-opacity:0.10"/><text class="t-cell" x="515.0" y="71.0" text-anchor="middle">0.108</text><text class="t-tick" x="122.0" y="97.0" text-anchor="end">x − μ</text><rect class="cell" x="131.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="165.0" y="97.0" text-anchor="middle">0.016</text><rect class="cell" x="201.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="235.0" y="97.0" text-anchor="middle">0.026</text><rect class="s2" x="271.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.15"/><text class="t-cell" x="305.0" y="97.0" text-anchor="middle">-0.537</text><rect class="s2" x="341.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="375.0" y="97.0" text-anchor="middle">-0.004</text><rect class="cell" x="411.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.19"/><text class="t-cell" x="445.0" y="97.0" text-anchor="middle">0.808</text><rect class="cell" x="481.0" y="81.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="515.0" y="97.0" text-anchor="middle">0.105</text><text class="t-tick" x="122.0" y="123.0" text-anchor="end">(x − μ) / σ</text><rect class="cell" x="131.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="165.0" y="123.0" text-anchor="middle">0.086</text><rect class="cell" x="201.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.10"/><text class="t-cell" x="235.0" y="123.0" text-anchor="middle">0.141</text><rect class="s2" x="271.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.48"/><text class="t-cell" x="305.0" y="123.0" text-anchor="middle">-2.875</text><rect class="s2" x="341.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="375.0" y="123.0" text-anchor="middle">-0.023</text><rect class="cell" x="411.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.68"/><text class="t-cell" x="445.0" y="123.0" text-anchor="middle">4.320</text><rect class="cell" x="481.0" y="107.0" width="68" height="24" rx="3" style="fill-opacity:0.16"/><text class="t-cell" x="515.0" y="123.0" text-anchor="middle">0.561</text><text class="t-tick" x="122.0" y="149.0" text-anchor="end">γ</text><rect class="cell" x="131.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.10"/><text class="t-cell" x="165.0" y="149.0" text-anchor="middle">0.156</text><rect class="cell" x="201.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.11"/><text class="t-cell" x="235.0" y="149.0" text-anchor="middle">0.191</text><rect class="cell" x="271.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="305.0" y="149.0" text-anchor="middle">0.033</text><rect class="cell" x="341.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.10"/><text class="t-cell" x="375.0" y="149.0" text-anchor="middle">0.119</text><rect class="cell" x="411.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="445.0" y="149.0" text-anchor="middle">0.065</text><rect class="cell" x="481.0" y="133.0" width="68" height="24" rx="3" style="fill-opacity:0.10"/><text class="t-cell" x="515.0" y="149.0" text-anchor="middle">0.126</text><text class="t-tick" x="122.0" y="175.0" text-anchor="end">β</text><rect class="cell" x="131.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="165.0" y="175.0" text-anchor="middle">0.008</text><rect class="cell" x="201.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="235.0" y="175.0" text-anchor="middle">0.016</text><rect class="cell" x="271.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="305.0" y="175.0" text-anchor="middle">0.091</text><rect class="cell" x="341.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="375.0" y="175.0" text-anchor="middle">0.067</text><rect class="cell" x="411.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="445.0" y="175.0" text-anchor="middle">0.007</text><rect class="s2" x="481.0" y="159.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="515.0" y="175.0" text-anchor="middle">-0.034</text><text class="t-tick" x="122.0" y="201.0" text-anchor="end">γ·(…) + β</text><rect class="cell" x="131.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="165.0" y="201.0" text-anchor="middle">0.022</text><rect class="cell" x="201.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="235.0" y="201.0" text-anchor="middle">0.043</text><rect class="s2" x="271.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="305.0" y="201.0" text-anchor="middle">-0.005</text><rect class="cell" x="341.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="375.0" y="201.0" text-anchor="middle">0.064</text><rect class="cell" x="411.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.12"/><text class="t-cell" x="445.0" y="201.0" text-anchor="middle">0.287</text><rect class="cell" x="481.0" y="185.0" width="68" height="24" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="515.0" y="201.0" text-anchor="middle">0.036</text><text class="t-tick" x="570.0" y="70.0" text-anchor="start">μ = mean of all 768 = 0.0034</text><text class="t-tick" x="570.0" y="86.0" text-anchor="start">σ = √(variance + ε) = 0.1869</text><text class="t-tick" x="570.0" y="102.0" text-anchor="start">ε = 1e-12</text><text class="t-tick" x="570.0" y="118.0" text-anchor="start"></text><text class="t-tick" x="570.0" y="134.0" text-anchor="start">γ and β are learned,</text><text class="t-tick" x="570.0" y="150.0" text-anchor="start">one value per position,</text><text class="t-tick" x="570.0" y="166.0" text-anchor="start">the same for every token.</text><text class="t-tick" x="570.0" y="182.0" text-anchor="start"></text><text class="t-tick" x="570.0" y="198.0" text-anchor="start">input length 5.18,</text><text class="t-tick" x="570.0" y="214.0" text-anchor="start">output length 1.50.</text><text class="t-muted" x="20.0" y="246.0" text-anchor="start">Our hand computation matches the library LayerNorm to 6e-08.</text><text class="t-muted" x="20.0" y="263.0" text-anchor="start">Rows 2 and 3 use the exact μ and σ, so they differ slightly from what the rounded row 1 would give.</text></svg><figcaption>LayerNorm step by step on the first six of the 768 numbers of the class token: the input, minus the mean (0.0034), divided by the standard deviation (0.1869), times the learned gamma, plus the learned beta. Orange cells are negative. Our hand computation matches the library to 6e-8.</figcaption></figure>

**Attention, head by head.** The normalised $$197 \times 768$$ matrix is multiplied by three matrices to give queries, keys and values, each $$197 \times 768$$, which are split into 12 heads of 64 columns. For each head, every token's query is compared with every token's key, the 197 scores are divided by $$\sqrt{64} = 8$$ and turned into weights by softmax, and the token's output is the weighted sum of the 197 value rows. The 12 outputs ($$197 \times 64$$ each) are placed side by side into $$197 \times 768$$ and multiplied by one more $$768 \times 768$$ matrix. Appendix A, at the end of this part, is the formal version; here is the real one, for the class token's query in head 1 of layer 1:

```python
lay = model.vit.layers[0]; at = lay.attention
q = ln1 @ at.q_proj.weight.T + at.q_proj.bias          # (197, 768); the same for k and v
qh, kh, vh = (t.view(197, 12, 64).transpose(0, 1) for t in (q, k, v))   # (12, 197, 64)
scores = qh @ kh.transpose(1, 2) / 8.0                 # (12, 197, 197)
A = torch.softmax(scores, -1)                          # every row sums to 1
SA = A @ vh                                            # (12, 197, 64)
concat = SA.transpose(0, 1).reshape(197, 768)          # the 12 heads side by side
msa = concat @ at.o_proj.weight.T + at.o_proj.bias     # U_msa
z_prime = msa + z0                                     # Eq. 2
```

```text
  q = LN(z_0) W_q + b_q: (197, 768)  (the same for k and v); split into 12 heads of 64: (12, 197, 64)
  head 1: q_class . k_j / sqrt(64) for all 197 tokens j: (197,); first 6 scores: [  5.125,  -0.698,  -0.880,  -0.946,  -1.040,  -1.146]
  softmax -> weights, first 6: [  0.761,   0.002,   0.002,   0.002,   0.002,   0.001]   sum of all 197 = 1.0000   max 0.7613   min 0.0005
  the 6 largest weights of the [class] query in head 1 of layer 1:
    token   0  weight 0.7613   [class] itself
    token  14  weight 0.0026   patch  13 = grid row  0, col 13
    token   1  weight 0.0023   patch   0 = grid row  0, col  0
    token 109  weight 0.0022   patch 108 = grid row  7, col 10
    token  13  weight 0.0021   patch  12 = grid row  0, col 12
    token 153  weight 0.0020   patch 152 = grid row 10, col 12
  weight on [class] itself: 0.7613;  on all 196 patches together: 0.2387
  weight of the [class] query on itself in each of the 12 heads of layer 1: 0.76 0.90 0.78 0.11 0.90 0.65 0.79 0.02 0.82 1.00 0.37 0.39
  head 8 spreads the [class] query most (self weight 0.022); its top-5 patches:
    token   0  weight 0.0219   [class] itself
    token   8  weight 0.0146   patch   7 = grid row  0, col  7
    token   3  weight 0.0122   patch   2 = grid row  0, col  2
    token   7  weight 0.0122   patch   6 = grid row  0, col  6
    token   4  weight 0.0120   patch   3 = grid row  0, col  3
```

> [!DEFINITION] Softmax
> A function that turns a list of scores into a list of positive weights that add up to 1: each score $$s_i$$ becomes $$e^{s_i} / \sum_j e^{s_j}$$. Larger scores get larger weights, and the gap grows quickly: a score 5 above the others takes almost all of the weight.

Two things are worth noticing. First, the score of the class token against itself is 5.125 while the others are around −1, and after softmax that one score takes 0.761 of the weight. Head 1 of layer 1 mostly leaves the class token alone. The same is true of most heads in this layer (0.65 to 1.00 on itself); only heads 4, 8, 11 and 12 look outward. Second, when a head does look outward, as head 8 does, no patch gets much: the largest weight is 0.0146, and the five largest all sit in the top row of the picture (the red sofa behind the cats). This is layer 1; there is no "cat detector" yet. [Part 5](https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vit/part-5-scaling-and-inside-vit.md) measures how far the heads look in every layer and shows that later layers attend to the object.

<figure class="fig"><svg viewBox="0 0 760 392" role="img" aria-label="Two 14 by 14 heatmaps of the class token attention weights in layer 1 of the real ViT-B/16 on the cat picture: head 1 puts 0.76 on the class token itself and spreads tiny weights over the patches; head 8 spreads its weight, mostly along the top row of patches."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">The [class] query in layer 1: its 196 patch weights on the 14 × 14 grid</text><text class="t-tick" x="20.0" y="50.0" text-anchor="start">head 1: 0.76 on [class] itself, largest patch weight 0.0026</text><text class="t-tick" x="46.0" y="76.5" text-anchor="end">0</text><g class="mark"><title>0 → 0: 0.88</title><rect class="cell" x="55.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.875"/></g><g class="mark"><title>0 → 1: 0.73</title><rect class="cell" x="72.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.732"/></g><g class="mark"><title>0 → 2: 0.68</title><rect class="cell" x="89.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.685"/></g><g class="mark"><title>0 → 3: 0.62</title><rect class="cell" x="106.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.623"/></g><g class="mark"><title>0 → 4: 0.56</title><rect class="cell" x="123.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.560"/></g><g class="mark"><title>0 → 5: 0.61</title><rect class="cell" x="140.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.607"/></g><g class="mark"><title>0 → 6: 0.61</title><rect class="cell" x="157.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.615"/></g><g class="mark"><title>0 → 7: 0.71</title><rect class="cell" x="174.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.712"/></g><g class="mark"><title>0 → 8: 0.64</title><rect class="cell" x="191.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.638"/></g><g class="mark"><title>0 → 9: 0.57</title><rect class="cell" x="208.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.572"/></g><g class="mark"><title>0 → 10: 0.58</title><rect class="cell" x="225.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.580"/></g><g class="mark"><title>0 → 11: 0.72</title><rect class="cell" x="242.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.724"/></g><g class="mark"><title>0 → 12: 0.81</title><rect class="cell" x="259.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.805"/></g><g class="mark"><title>0 → 13: 1.00</title><rect class="cell" x="276.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:1.000"/></g><text class="t-tick" x="46.0" y="93.5" text-anchor="end">1</text><g class="mark"><title>1 → 0: 0.77</title><rect class="cell" x="55.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.770"/></g><g class="mark"><title>1 → 1: 0.61</title><rect class="cell" x="72.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.607"/></g><g class="mark"><title>1 → 2: 0.54</title><rect class="cell" x="89.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.541"/></g><g class="mark"><title>1 → 3: 0.47</title><rect class="cell" x="106.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.471"/></g><g class="mark"><title>1 → 4: 0.47</title><rect class="cell" x="123.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.475"/></g><g class="mark"><title>1 → 5: 0.47</title><rect class="cell" x="140.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.475"/></g><g class="mark"><title>1 → 6: 0.56</title><rect class="cell" x="157.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.556"/></g><g class="mark"><title>1 → 7: 0.53</title><rect class="cell" x="174.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.533"/></g><g class="mark"><title>1 → 8: 0.48</title><rect class="cell" x="191.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.479"/></g><g class="mark"><title>1 → 9: 0.47</title><rect class="cell" x="208.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.467"/></g><g class="mark"><title>1 → 10: 0.32</title><rect class="cell" x="225.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.323"/></g><g class="mark"><title>1 → 11: 0.35</title><rect class="cell" x="242.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.346"/></g><g class="mark"><title>1 → 12: 0.68</title><rect class="cell" x="259.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.685"/></g><g class="mark"><title>1 → 13: 0.69</title><rect class="cell" x="276.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.693"/></g><text class="t-tick" x="46.0" y="110.5" text-anchor="end">2</text><g class="mark"><title>2 → 0: 0.60</title><rect class="cell" x="55.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.603"/></g><g class="mark"><title>2 → 1: 0.33</title><rect class="cell" x="72.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.327"/></g><g class="mark"><title>2 → 2: 0.30</title><rect class="cell" x="89.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.300"/></g><g class="mark"><title>2 → 3: 0.33</title><rect class="cell" x="106.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.335"/></g><g class="mark"><title>2 → 4: 0.25</title><rect class="cell" x="123.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.253"/></g><g class="mark"><title>2 → 5: 0.42</title><rect class="cell" x="140.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.420"/></g><g class="mark"><title>2 → 6: 0.38</title><rect class="cell" x="157.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><g class="mark"><title>2 → 7: 0.44</title><rect class="cell" x="174.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.444"/></g><g class="mark"><title>2 → 8: 0.28</title><rect class="cell" x="191.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.276"/></g><g class="mark"><title>2 → 9: 0.54</title><rect class="cell" x="208.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.537"/></g><g class="mark"><title>2 → 10: 0.38</title><rect class="cell" x="225.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><g class="mark"><title>2 → 11: 0.25</title><rect class="cell" x="242.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.249"/></g><g class="mark"><title>2 → 12: 0.33</title><rect class="cell" x="259.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.331"/></g><g class="mark"><title>2 → 13: 0.38</title><rect class="cell" x="276.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><text class="t-tick" x="46.0" y="127.5" text-anchor="end">3</text><g class="mark"><title>3 → 0: 0.48</title><rect class="cell" x="55.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.482"/></g><g class="mark"><title>3 → 1: 0.68</title><rect class="cell" x="72.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.677"/></g><g class="mark"><title>3 → 2: 0.54</title><rect class="cell" x="89.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.541"/></g><g class="mark"><title>3 → 3: 0.38</title><rect class="cell" x="106.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><g class="mark"><title>3 → 4: 0.43</title><rect class="cell" x="123.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.428"/></g><g class="mark"><title>3 → 5: 0.47</title><rect class="cell" x="140.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.467"/></g><g class="mark"><title>3 → 6: 0.41</title><rect class="cell" x="157.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.412"/></g><g class="mark"><title>3 → 7: 0.33</title><rect class="cell" x="174.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.327"/></g><g class="mark"><title>3 → 8: 0.58</title><rect class="cell" x="191.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.584"/></g><g class="mark"><title>3 → 9: 0.25</title><rect class="cell" x="208.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.245"/></g><g class="mark"><title>3 → 10: 0.72</title><rect class="cell" x="225.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.720"/></g><g class="mark"><title>3 → 11: 0.40</title><rect class="cell" x="242.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.397"/></g><g class="mark"><title>3 → 12: 0.71</title><rect class="cell" x="259.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.708"/></g><g class="mark"><title>3 → 13: 0.31</title><rect class="cell" x="276.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.311"/></g><text class="t-tick" x="46.0" y="144.5" text-anchor="end">4</text><g class="mark"><title>4 → 0: 0.52</title><rect class="cell" x="55.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.518"/></g><g class="mark"><title>4 → 1: 0.54</title><rect class="cell" x="72.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><g class="mark"><title>4 → 2: 0.28</title><rect class="cell" x="89.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.276"/></g><g class="mark"><title>4 → 3: 0.55</title><rect class="cell" x="106.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.549"/></g><g class="mark"><title>4 → 4: 0.24</title><rect class="cell" x="123.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.241"/></g><g class="mark"><title>4 → 5: 0.28</title><rect class="cell" x="140.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.276"/></g><g class="mark"><title>4 → 6: 0.49</title><rect class="cell" x="157.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.490"/></g><g class="mark"><title>4 → 7: 0.32</title><rect class="cell" x="174.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.323"/></g><g class="mark"><title>4 → 8: 0.47</title><rect class="cell" x="191.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.467"/></g><g class="mark"><title>4 → 9: 0.19</title><rect class="cell" x="208.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.187"/></g><g class="mark"><title>4 → 10: 0.31</title><rect class="cell" x="225.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.311"/></g><g class="mark"><title>4 → 11: 0.46</title><rect class="cell" x="242.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.455"/></g><g class="mark"><title>4 → 12: 0.49</title><rect class="cell" x="259.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.486"/></g><g class="mark"><title>4 → 13: 0.51</title><rect class="cell" x="276.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.510"/></g><text class="t-tick" x="46.0" y="161.5" text-anchor="end">5</text><g class="mark"><title>5 → 0: 0.53</title><rect class="cell" x="55.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.529"/></g><g class="mark"><title>5 → 1: 0.42</title><rect class="cell" x="72.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.420"/></g><g class="mark"><title>5 → 2: 0.72</title><rect class="cell" x="89.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.724"/></g><g class="mark"><title>5 → 3: 0.52</title><rect class="cell" x="106.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.521"/></g><g class="mark"><title>5 → 4: 0.50</title><rect class="cell" x="123.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.498"/></g><g class="mark"><title>5 → 5: 0.52</title><rect class="cell" x="140.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.518"/></g><g class="mark"><title>5 → 6: 0.69</title><rect class="cell" x="157.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.693"/></g><g class="mark"><title>5 → 7: 0.58</title><rect class="cell" x="174.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.576"/></g><g class="mark"><title>5 → 8: 0.43</title><rect class="cell" x="191.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.428"/></g><g class="mark"><title>5 → 9: 0.28</title><rect class="cell" x="208.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.284"/></g><g class="mark"><title>5 → 10: 0.53</title><rect class="cell" x="225.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.529"/></g><g class="mark"><title>5 → 11: 0.67</title><rect class="cell" x="242.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.665"/></g><g class="mark"><title>5 → 12: 0.51</title><rect class="cell" x="259.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.514"/></g><g class="mark"><title>5 → 13: 0.56</title><rect class="cell" x="276.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.560"/></g><text class="t-tick" x="46.0" y="178.5" text-anchor="end">6</text><g class="mark"><title>6 → 0: 0.60</title><rect class="cell" x="55.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.599"/></g><g class="mark"><title>6 → 1: 0.27</title><rect class="cell" x="72.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.268"/></g><g class="mark"><title>6 → 2: 0.43</title><rect class="cell" x="89.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.428"/></g><g class="mark"><title>6 → 3: 0.42</title><rect class="cell" x="106.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.424"/></g><g class="mark"><title>6 → 4: 0.35</title><rect class="cell" x="123.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.350"/></g><g class="mark"><title>6 → 5: 0.36</title><rect class="cell" x="140.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.358"/></g><g class="mark"><title>6 → 6: 0.41</title><rect class="cell" x="157.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.412"/></g><g class="mark"><title>6 → 7: 0.32</title><rect class="cell" x="174.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.319"/></g><g class="mark"><title>6 → 8: 0.51</title><rect class="cell" x="191.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.510"/></g><g class="mark"><title>6 → 9: 0.32</title><rect class="cell" x="208.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.323"/></g><g class="mark"><title>6 → 10: 0.55</title><rect class="cell" x="225.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.549"/></g><g class="mark"><title>6 → 11: 0.46</title><rect class="cell" x="242.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.463"/></g><g class="mark"><title>6 → 12: 0.47</title><rect class="cell" x="259.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.475"/></g><g class="mark"><title>6 → 13: 0.56</title><rect class="cell" x="276.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.560"/></g><text class="t-tick" x="46.0" y="195.5" text-anchor="end">7</text><g class="mark"><title>7 → 0: 0.50</title><rect class="cell" x="55.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.502"/></g><g class="mark"><title>7 → 1: 0.43</title><rect class="cell" x="72.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.428"/></g><g class="mark"><title>7 → 2: 0.49</title><rect class="cell" x="89.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.486"/></g><g class="mark"><title>7 → 3: 0.47</title><rect class="cell" x="106.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.471"/></g><g class="mark"><title>7 → 4: 0.38</title><rect class="cell" x="123.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.377"/></g><g class="mark"><title>7 → 5: 0.37</title><rect class="cell" x="140.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.370"/></g><g class="mark"><title>7 → 6: 0.38</title><rect class="cell" x="157.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.377"/></g><g class="mark"><title>7 → 7: 0.30</title><rect class="cell" x="174.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.296"/></g><g class="mark"><title>7 → 8: 0.74</title><rect class="cell" x="191.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.739"/></g><g class="mark"><title>7 → 9: 0.20</title><rect class="cell" x="208.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.202"/></g><g class="mark"><title>7 → 10: 0.85</title><rect class="cell" x="225.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.848"/></g><g class="mark"><title>7 → 11: 0.27</title><rect class="cell" x="242.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.268"/></g><g class="mark"><title>7 → 12: 0.46</title><rect class="cell" x="259.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.459"/></g><g class="mark"><title>7 → 13: 0.53</title><rect class="cell" x="276.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.533"/></g><text class="t-tick" x="46.0" y="212.5" text-anchor="end">8</text><g class="mark"><title>8 → 0: 0.32</title><rect class="cell" x="55.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.315"/></g><g class="mark"><title>8 → 1: 0.48</title><rect class="cell" x="72.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.482"/></g><g class="mark"><title>8 → 2: 0.40</title><rect class="cell" x="89.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.405"/></g><g class="mark"><title>8 → 3: 0.54</title><rect class="cell" x="106.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><g class="mark"><title>8 → 4: 0.42</title><rect class="cell" x="123.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.416"/></g><g class="mark"><title>8 → 5: 0.39</title><rect class="cell" x="140.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.385"/></g><g class="mark"><title>8 → 6: 0.49</title><rect class="cell" x="157.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.494"/></g><g class="mark"><title>8 → 7: 0.48</title><rect class="cell" x="174.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.479"/></g><g class="mark"><title>8 → 8: 0.40</title><rect class="cell" x="191.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.401"/></g><g class="mark"><title>8 → 9: 0.30</title><rect class="cell" x="208.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.304"/></g><g class="mark"><title>8 → 10: 0.28</title><rect class="cell" x="225.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.280"/></g><g class="mark"><title>8 → 11: 0.37</title><rect class="cell" x="242.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.366"/></g><g class="mark"><title>8 → 12: 0.45</title><rect class="cell" x="259.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.451"/></g><g class="mark"><title>8 → 13: 0.54</title><rect class="cell" x="276.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><text class="t-tick" x="46.0" y="229.5" text-anchor="end">9</text><g class="mark"><title>9 → 0: 0.46</title><rect class="cell" x="55.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.455"/></g><g class="mark"><title>9 → 1: 0.54</title><rect class="cell" x="72.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><g class="mark"><title>9 → 2: 0.44</title><rect class="cell" x="89.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.444"/></g><g class="mark"><title>9 → 3: 0.48</title><rect class="cell" x="106.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.482"/></g><g class="mark"><title>9 → 4: 0.42</title><rect class="cell" x="123.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.416"/></g><g class="mark"><title>9 → 5: 0.33</title><rect class="cell" x="140.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.335"/></g><g class="mark"><title>9 → 6: 0.34</title><rect class="cell" x="157.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.339"/></g><g class="mark"><title>9 → 7: 0.38</title><rect class="cell" x="174.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><g class="mark"><title>9 → 8: 0.41</title><rect class="cell" x="191.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.409"/></g><g class="mark"><title>9 → 9: 0.48</title><rect class="cell" x="208.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.482"/></g><g class="mark"><title>9 → 10: 0.35</title><rect class="cell" x="225.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.346"/></g><g class="mark"><title>9 → 11: 0.32</title><rect class="cell" x="242.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.319"/></g><g class="mark"><title>9 → 12: 0.51</title><rect class="cell" x="259.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.506"/></g><g class="mark"><title>9 → 13: 0.58</title><rect class="cell" x="276.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.584"/></g><text class="t-tick" x="46.0" y="246.5" text-anchor="end">10</text><g class="mark"><title>10 → 0: 0.57</title><rect class="cell" x="55.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.572"/></g><g class="mark"><title>10 → 1: 0.28</title><rect class="cell" x="72.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.280"/></g><g class="mark"><title>10 → 2: 0.44</title><rect class="cell" x="89.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.436"/></g><g class="mark"><title>10 → 3: 0.58</title><rect class="cell" x="106.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.584"/></g><g class="mark"><title>10 → 4: 0.33</title><rect class="cell" x="123.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.331"/></g><g class="mark"><title>10 → 5: 0.36</title><rect class="cell" x="140.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.362"/></g><g class="mark"><title>10 → 6: 0.41</title><rect class="cell" x="157.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.409"/></g><g class="mark"><title>10 → 7: 0.39</title><rect class="cell" x="174.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.389"/></g><g class="mark"><title>10 → 8: 0.42</title><rect class="cell" x="191.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.424"/></g><g class="mark"><title>10 → 9: 0.47</title><rect class="cell" x="208.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.475"/></g><g class="mark"><title>10 → 10: 0.38</title><rect class="cell" x="225.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.381"/></g><g class="mark"><title>10 → 11: 0.46</title><rect class="cell" x="242.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.459"/></g><g class="mark"><title>10 → 12: 0.79</title><rect class="cell" x="259.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.790"/></g><g class="mark"><title>10 → 13: 0.52</title><rect class="cell" x="276.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.521"/></g><text class="t-tick" x="46.0" y="263.5" text-anchor="end">11</text><g class="mark"><title>11 → 0: 0.58</title><rect class="cell" x="55.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.584"/></g><g class="mark"><title>11 → 1: 0.61</title><rect class="cell" x="72.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.615"/></g><g class="mark"><title>11 → 2: 0.40</title><rect class="cell" x="89.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.397"/></g><g class="mark"><title>11 → 3: 0.36</title><rect class="cell" x="106.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.362"/></g><g class="mark"><title>11 → 4: 0.36</title><rect class="cell" x="123.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.358"/></g><g class="mark"><title>11 → 5: 0.37</title><rect class="cell" x="140.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.366"/></g><g class="mark"><title>11 → 6: 0.35</title><rect class="cell" x="157.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.350"/></g><g class="mark"><title>11 → 7: 0.42</title><rect class="cell" x="174.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.424"/></g><g class="mark"><title>11 → 8: 0.39</title><rect class="cell" x="191.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.385"/></g><g class="mark"><title>11 → 9: 0.37</title><rect class="cell" x="208.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.374"/></g><g class="mark"><title>11 → 10: 0.39</title><rect class="cell" x="225.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.385"/></g><g class="mark"><title>11 → 11: 0.44</title><rect class="cell" x="242.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.440"/></g><g class="mark"><title>11 → 12: 0.47</title><rect class="cell" x="259.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.467"/></g><g class="mark"><title>11 → 13: 0.52</title><rect class="cell" x="276.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.521"/></g><text class="t-tick" x="46.0" y="280.5" text-anchor="end">12</text><g class="mark"><title>12 → 0: 0.54</title><rect class="cell" x="55.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><g class="mark"><title>12 → 1: 0.61</title><rect class="cell" x="72.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.611"/></g><g class="mark"><title>12 → 2: 0.46</title><rect class="cell" x="89.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.459"/></g><g class="mark"><title>12 → 3: 0.42</title><rect class="cell" x="106.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.420"/></g><g class="mark"><title>12 → 4: 0.39</title><rect class="cell" x="123.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.389"/></g><g class="mark"><title>12 → 5: 0.36</title><rect class="cell" x="140.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.362"/></g><g class="mark"><title>12 → 6: 0.48</title><rect class="cell" x="157.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.479"/></g><g class="mark"><title>12 → 7: 0.43</title><rect class="cell" x="174.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.432"/></g><g class="mark"><title>12 → 8: 0.41</title><rect class="cell" x="191.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.412"/></g><g class="mark"><title>12 → 9: 0.38</title><rect class="cell" x="208.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.377"/></g><g class="mark"><title>12 → 10: 0.44</title><rect class="cell" x="225.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.436"/></g><g class="mark"><title>12 → 11: 0.39</title><rect class="cell" x="242.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.393"/></g><g class="mark"><title>12 → 12: 0.46</title><rect class="cell" x="259.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.455"/></g><g class="mark"><title>12 → 13: 0.52</title><rect class="cell" x="276.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.521"/></g><text class="t-tick" x="46.0" y="297.5" text-anchor="end">13</text><g class="mark"><title>13 → 0: 0.74</title><rect class="cell" x="55.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.739"/></g><g class="mark"><title>13 → 1: 0.53</title><rect class="cell" x="72.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.529"/></g><g class="mark"><title>13 → 2: 0.52</title><rect class="cell" x="89.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.518"/></g><g class="mark"><title>13 → 3: 0.49</title><rect class="cell" x="106.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.490"/></g><g class="mark"><title>13 → 4: 0.46</title><rect class="cell" x="123.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.459"/></g><g class="mark"><title>13 → 5: 0.50</title><rect class="cell" x="140.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.498"/></g><g class="mark"><title>13 → 6: 0.55</title><rect class="cell" x="157.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.549"/></g><g class="mark"><title>13 → 7: 0.54</title><rect class="cell" x="174.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.545"/></g><g class="mark"><title>13 → 8: 0.52</title><rect class="cell" x="191.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.521"/></g><g class="mark"><title>13 → 9: 0.50</title><rect class="cell" x="208.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.502"/></g><g class="mark"><title>13 → 10: 0.51</title><rect class="cell" x="225.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.506"/></g><g class="mark"><title>13 → 11: 0.60</title><rect class="cell" x="242.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.595"/></g><g class="mark"><title>13 → 12: 0.60</title><rect class="cell" x="259.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.595"/></g><g class="mark"><title>13 → 13: 0.63</title><rect class="cell" x="276.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.626"/></g><text class="t-tick" x="62.5" y="312.0" text-anchor="end" transform="rotate(-50 62.5 312.0)">0</text><text class="t-tick" x="79.5" y="312.0" text-anchor="end" transform="rotate(-50 79.5 312.0)">1</text><text class="t-tick" x="96.5" y="312.0" text-anchor="end" transform="rotate(-50 96.5 312.0)">2</text><text class="t-tick" x="113.5" y="312.0" text-anchor="end" transform="rotate(-50 113.5 312.0)">3</text><text class="t-tick" x="130.5" y="312.0" text-anchor="end" transform="rotate(-50 130.5 312.0)">4</text><text class="t-tick" x="147.5" y="312.0" text-anchor="end" transform="rotate(-50 147.5 312.0)">5</text><text class="t-tick" x="164.5" y="312.0" text-anchor="end" transform="rotate(-50 164.5 312.0)">6</text><text class="t-tick" x="181.5" y="312.0" text-anchor="end" transform="rotate(-50 181.5 312.0)">7</text><text class="t-tick" x="198.5" y="312.0" text-anchor="end" transform="rotate(-50 198.5 312.0)">8</text><text class="t-tick" x="215.5" y="312.0" text-anchor="end" transform="rotate(-50 215.5 312.0)">9</text><text class="t-tick" x="232.5" y="312.0" text-anchor="end" transform="rotate(-50 232.5 312.0)">10</text><text class="t-tick" x="249.5" y="312.0" text-anchor="end" transform="rotate(-50 249.5 312.0)">11</text><text class="t-tick" x="266.5" y="312.0" text-anchor="end" transform="rotate(-50 266.5 312.0)">12</text><text class="t-tick" x="283.5" y="312.0" text-anchor="end" transform="rotate(-50 283.5 312.0)">13</text><text class="t-muted" x="173.0" y="338.0" text-anchor="middle">patch column 0 to 13</text><text class="t-tick" x="400.0" y="50.0" text-anchor="start">head 8: 0.022 on itself, largest patch weight 0.0146</text><text class="t-tick" x="426.0" y="76.5" text-anchor="end">0</text><g class="mark"><title>0 → 0: 0.75</title><rect class="cell" x="435.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.748"/></g><g class="mark"><title>0 → 1: 0.79</title><rect class="cell" x="452.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.788"/></g><g class="mark"><title>0 → 2: 0.83</title><rect class="cell" x="469.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.834"/></g><g class="mark"><title>0 → 3: 0.82</title><rect class="cell" x="486.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.817"/></g><g class="mark"><title>0 → 4: 0.78</title><rect class="cell" x="503.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.778"/></g><g class="mark"><title>0 → 5: 0.79</title><rect class="cell" x="520.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.787"/></g><g class="mark"><title>0 → 6: 0.83</title><rect class="cell" x="537.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.834"/></g><g class="mark"><title>0 → 7: 1.00</title><rect class="cell" x="554.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:1.000"/></g><g class="mark"><title>0 → 8: 0.73</title><rect class="cell" x="571.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.734"/></g><g class="mark"><title>0 → 9: 0.54</title><rect class="cell" x="588.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.541"/></g><g class="mark"><title>0 → 10: 0.27</title><rect class="cell" x="605.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.266"/></g><g class="mark"><title>0 → 11: 0.41</title><rect class="cell" x="622.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.411"/></g><g class="mark"><title>0 → 12: 0.68</title><rect class="cell" x="639.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.678"/></g><g class="mark"><title>0 → 13: 0.73</title><rect class="cell" x="656.0" y="65.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.732"/></g><text class="t-tick" x="426.0" y="93.5" text-anchor="end">1</text><g class="mark"><title>1 → 0: 0.16</title><rect class="cell" x="435.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.161"/></g><g class="mark"><title>1 → 1: 0.46</title><rect class="cell" x="452.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.461"/></g><g class="mark"><title>1 → 2: 0.68</title><rect class="cell" x="469.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.678"/></g><g class="mark"><title>1 → 3: 0.52</title><rect class="cell" x="486.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.525"/></g><g class="mark"><title>1 → 4: 0.44</title><rect class="cell" x="503.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.436"/></g><g class="mark"><title>1 → 5: 0.27</title><rect class="cell" x="520.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.267"/></g><g class="mark"><title>1 → 6: 0.29</title><rect class="cell" x="537.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.289"/></g><g class="mark"><title>1 → 7: 0.53</title><rect class="cell" x="554.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.526"/></g><g class="mark"><title>1 → 8: 0.32</title><rect class="cell" x="571.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.324"/></g><g class="mark"><title>1 → 9: 0.11</title><rect class="cell" x="588.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.108"/></g><g class="mark"><title>1 → 10: 0.10</title><rect class="cell" x="605.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.100"/></g><g class="mark"><title>1 → 11: 0.11</title><rect class="cell" x="622.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.105"/></g><g class="mark"><title>1 → 12: 0.30</title><rect class="cell" x="639.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.295"/></g><g class="mark"><title>1 → 13: 0.51</title><rect class="cell" x="656.0" y="82.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.513"/></g><text class="t-tick" x="426.0" y="110.5" text-anchor="end">2</text><g class="mark"><title>2 → 0: 0.24</title><rect class="cell" x="435.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.244"/></g><g class="mark"><title>2 → 1: 0.14</title><rect class="cell" x="452.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.135"/></g><g class="mark"><title>2 → 2: 0.10</title><rect class="cell" x="469.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.096"/></g><g class="mark"><title>2 → 3: 0.15</title><rect class="cell" x="486.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.150"/></g><g class="mark"><title>2 → 4: 0.10</title><rect class="cell" x="503.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.096"/></g><g class="mark"><title>2 → 5: 0.13</title><rect class="cell" x="520.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.131"/></g><g class="mark"><title>2 → 6: 0.11</title><rect class="cell" x="537.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.110"/></g><g class="mark"><title>2 → 7: 0.08</title><rect class="cell" x="554.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.076"/></g><g class="mark"><title>2 → 8: 0.07</title><rect class="cell" x="571.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.070"/></g><g class="mark"><title>2 → 9: 0.23</title><rect class="cell" x="588.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.228"/></g><g class="mark"><title>2 → 10: 0.06</title><rect class="cell" x="605.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.057"/></g><g class="mark"><title>2 → 11: 0.07</title><rect class="cell" x="622.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.066"/></g><g class="mark"><title>2 → 12: 0.13</title><rect class="cell" x="639.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.130"/></g><g class="mark"><title>2 → 13: 0.10</title><rect class="cell" x="656.0" y="99.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.098"/></g><text class="t-tick" x="426.0" y="127.5" text-anchor="end">3</text><g class="mark"><title>3 → 0: 0.33</title><rect class="cell" x="435.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.335"/></g><g class="mark"><title>3 → 1: 0.17</title><rect class="cell" x="452.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.166"/></g><g class="mark"><title>3 → 2: 0.21</title><rect class="cell" x="469.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.215"/></g><g class="mark"><title>3 → 3: 0.17</title><rect class="cell" x="486.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.174"/></g><g class="mark"><title>3 → 4: 0.08</title><rect class="cell" x="503.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.078"/></g><g class="mark"><title>3 → 5: 0.13</title><rect class="cell" x="520.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.133"/></g><g class="mark"><title>3 → 6: 0.11</title><rect class="cell" x="537.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.111"/></g><g class="mark"><title>3 → 7: 0.12</title><rect class="cell" x="554.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.120"/></g><g class="mark"><title>3 → 8: 0.05</title><rect class="cell" x="571.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.054"/></g><g class="mark"><title>3 → 9: 0.08</title><rect class="cell" x="588.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.084"/></g><g class="mark"><title>3 → 10: 0.18</title><rect class="cell" x="605.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.181"/></g><g class="mark"><title>3 → 11: 0.08</title><rect class="cell" x="622.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.078"/></g><g class="mark"><title>3 → 12: 0.15</title><rect class="cell" x="639.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.150"/></g><g class="mark"><title>3 → 13: 0.11</title><rect class="cell" x="656.0" y="116.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.115"/></g><text class="t-tick" x="426.0" y="144.5" text-anchor="end">4</text><g class="mark"><title>4 → 0: 0.54</title><rect class="cell" x="435.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.541"/></g><g class="mark"><title>4 → 1: 0.58</title><rect class="cell" x="452.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.582"/></g><g class="mark"><title>4 → 2: 0.07</title><rect class="cell" x="469.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.070"/></g><g class="mark"><title>4 → 3: 0.15</title><rect class="cell" x="486.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.154"/></g><g class="mark"><title>4 → 4: 0.09</title><rect class="cell" x="503.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.086"/></g><g class="mark"><title>4 → 5: 0.11</title><rect class="cell" x="520.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.106"/></g><g class="mark"><title>4 → 6: 0.39</title><rect class="cell" x="537.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.386"/></g><g class="mark"><title>4 → 7: 0.26</title><rect class="cell" x="554.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.263"/></g><g class="mark"><title>4 → 8: 0.11</title><rect class="cell" x="571.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.109"/></g><g class="mark"><title>4 → 9: 0.06</title><rect class="cell" x="588.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.058"/></g><g class="mark"><title>4 → 10: 0.09</title><rect class="cell" x="605.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.092"/></g><g class="mark"><title>4 → 11: 0.15</title><rect class="cell" x="622.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.151"/></g><g class="mark"><title>4 → 12: 0.46</title><rect class="cell" x="639.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.461"/></g><g class="mark"><title>4 → 13: 0.37</title><rect class="cell" x="656.0" y="133.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.368"/></g><text class="t-tick" x="426.0" y="161.5" text-anchor="end">5</text><g class="mark"><title>5 → 0: 0.62</title><rect class="cell" x="435.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.620"/></g><g class="mark"><title>5 → 1: 0.38</title><rect class="cell" x="452.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.379"/></g><g class="mark"><title>5 → 2: 0.07</title><rect class="cell" x="469.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.067"/></g><g class="mark"><title>5 → 3: 0.11</title><rect class="cell" x="486.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.111"/></g><g class="mark"><title>5 → 4: 0.08</title><rect class="cell" x="503.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.081"/></g><g class="mark"><title>5 → 5: 0.07</title><rect class="cell" x="520.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.074"/></g><g class="mark"><title>5 → 6: 0.17</title><rect class="cell" x="537.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.172"/></g><g class="mark"><title>5 → 7: 0.29</title><rect class="cell" x="554.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.289"/></g><g class="mark"><title>5 → 8: 0.06</title><rect class="cell" x="571.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.062"/></g><g class="mark"><title>5 → 9: 0.12</title><rect class="cell" x="588.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.116"/></g><g class="mark"><title>5 → 10: 0.08</title><rect class="cell" x="605.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.081"/></g><g class="mark"><title>5 → 11: 0.27</title><rect class="cell" x="622.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.273"/></g><g class="mark"><title>5 → 12: 0.44</title><rect class="cell" x="639.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.444"/></g><g class="mark"><title>5 → 13: 0.62</title><rect class="cell" x="656.0" y="150.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.620"/></g><text class="t-tick" x="426.0" y="178.5" text-anchor="end">6</text><g class="mark"><title>6 → 0: 0.60</title><rect class="cell" x="435.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.596"/></g><g class="mark"><title>6 → 1: 0.14</title><rect class="cell" x="452.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.141"/></g><g class="mark"><title>6 → 2: 0.13</title><rect class="cell" x="469.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.135"/></g><g class="mark"><title>6 → 3: 0.06</title><rect class="cell" x="486.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.062"/></g><g class="mark"><title>6 → 4: 0.23</title><rect class="cell" x="503.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.234"/></g><g class="mark"><title>6 → 5: 0.40</title><rect class="cell" x="520.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.396"/></g><g class="mark"><title>6 → 6: 0.08</title><rect class="cell" x="537.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.077"/></g><g class="mark"><title>6 → 7: 0.40</title><rect class="cell" x="554.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.401"/></g><g class="mark"><title>6 → 8: 0.12</title><rect class="cell" x="571.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.118"/></g><g class="mark"><title>6 → 9: 0.10</title><rect class="cell" x="588.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.100"/></g><g class="mark"><title>6 → 10: 0.08</title><rect class="cell" x="605.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.083"/></g><g class="mark"><title>6 → 11: 0.40</title><rect class="cell" x="622.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.399"/></g><g class="mark"><title>6 → 12: 0.39</title><rect class="cell" x="639.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.393"/></g><g class="mark"><title>6 → 13: 0.63</title><rect class="cell" x="656.0" y="167.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.631"/></g><text class="t-tick" x="426.0" y="195.5" text-anchor="end">7</text><g class="mark"><title>7 → 0: 0.42</title><rect class="cell" x="435.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.422"/></g><g class="mark"><title>7 → 1: 0.17</title><rect class="cell" x="452.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.172"/></g><g class="mark"><title>7 → 2: 0.13</title><rect class="cell" x="469.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.129"/></g><g class="mark"><title>7 → 3: 0.16</title><rect class="cell" x="486.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.163"/></g><g class="mark"><title>7 → 4: 0.46</title><rect class="cell" x="503.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.461"/></g><g class="mark"><title>7 → 5: 0.47</title><rect class="cell" x="520.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.468"/></g><g class="mark"><title>7 → 6: 0.53</title><rect class="cell" x="537.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.529"/></g><g class="mark"><title>7 → 7: 0.28</title><rect class="cell" x="554.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.284"/></g><g class="mark"><title>7 → 8: 0.07</title><rect class="cell" x="571.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.069"/></g><g class="mark"><title>7 → 9: 0.05</title><rect class="cell" x="588.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.053"/></g><g class="mark"><title>7 → 10: 0.12</title><rect class="cell" x="605.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.118"/></g><g class="mark"><title>7 → 11: 0.20</title><rect class="cell" x="622.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.204"/></g><g class="mark"><title>7 → 12: 0.47</title><rect class="cell" x="639.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.468"/></g><g class="mark"><title>7 → 13: 0.65</title><rect class="cell" x="656.0" y="184.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.651"/></g><text class="t-tick" x="426.0" y="212.5" text-anchor="end">8</text><g class="mark"><title>8 → 0: 0.28</title><rect class="cell" x="435.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.278"/></g><g class="mark"><title>8 → 1: 0.11</title><rect class="cell" x="452.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.111"/></g><g class="mark"><title>8 → 2: 0.09</title><rect class="cell" x="469.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.093"/></g><g class="mark"><title>8 → 3: 0.27</title><rect class="cell" x="486.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.265"/></g><g class="mark"><title>8 → 4: 0.40</title><rect class="cell" x="503.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.403"/></g><g class="mark"><title>8 → 5: 0.50</title><rect class="cell" x="520.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.499"/></g><g class="mark"><title>8 → 6: 0.60</title><rect class="cell" x="537.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.599"/></g><g class="mark"><title>8 → 7: 0.42</title><rect class="cell" x="554.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.418"/></g><g class="mark"><title>8 → 8: 0.12</title><rect class="cell" x="571.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.123"/></g><g class="mark"><title>8 → 9: 0.18</title><rect class="cell" x="588.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.176"/></g><g class="mark"><title>8 → 10: 0.10</title><rect class="cell" x="605.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.100"/></g><g class="mark"><title>8 → 11: 0.08</title><rect class="cell" x="622.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.082"/></g><g class="mark"><title>8 → 12: 0.14</title><rect class="cell" x="639.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.135"/></g><g class="mark"><title>8 → 13: 0.60</title><rect class="cell" x="656.0" y="201.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.600"/></g><text class="t-tick" x="426.0" y="229.5" text-anchor="end">9</text><g class="mark"><title>9 → 0: 0.27</title><rect class="cell" x="435.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.268"/></g><g class="mark"><title>9 → 1: 0.18</title><rect class="cell" x="452.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.181"/></g><g class="mark"><title>9 → 2: 0.07</title><rect class="cell" x="469.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.071"/></g><g class="mark"><title>9 → 3: 0.13</title><rect class="cell" x="486.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.134"/></g><g class="mark"><title>9 → 4: 0.09</title><rect class="cell" x="503.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.088"/></g><g class="mark"><title>9 → 5: 0.40</title><rect class="cell" x="520.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.395"/></g><g class="mark"><title>9 → 6: 0.46</title><rect class="cell" x="537.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.458"/></g><g class="mark"><title>9 → 7: 0.53</title><rect class="cell" x="554.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.526"/></g><g class="mark"><title>9 → 8: 0.15</title><rect class="cell" x="571.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.148"/></g><g class="mark"><title>9 → 9: 0.10</title><rect class="cell" x="588.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.103"/></g><g class="mark"><title>9 → 10: 0.22</title><rect class="cell" x="605.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.219"/></g><g class="mark"><title>9 → 11: 0.09</title><rect class="cell" x="622.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.089"/></g><g class="mark"><title>9 → 12: 0.30</title><rect class="cell" x="639.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.305"/></g><g class="mark"><title>9 → 13: 0.72</title><rect class="cell" x="656.0" y="218.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.718"/></g><text class="t-tick" x="426.0" y="246.5" text-anchor="end">10</text><g class="mark"><title>10 → 0: 0.33</title><rect class="cell" x="435.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.334"/></g><g class="mark"><title>10 → 1: 0.09</title><rect class="cell" x="452.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.087"/></g><g class="mark"><title>10 → 2: 0.06</title><rect class="cell" x="469.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.059"/></g><g class="mark"><title>10 → 3: 0.14</title><rect class="cell" x="486.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.144"/></g><g class="mark"><title>10 → 4: 0.39</title><rect class="cell" x="503.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.386"/></g><g class="mark"><title>10 → 5: 0.45</title><rect class="cell" x="520.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.455"/></g><g class="mark"><title>10 → 6: 0.46</title><rect class="cell" x="537.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.461"/></g><g class="mark"><title>10 → 7: 0.52</title><rect class="cell" x="554.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.519"/></g><g class="mark"><title>10 → 8: 0.18</title><rect class="cell" x="571.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.185"/></g><g class="mark"><title>10 → 9: 0.13</title><rect class="cell" x="588.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.126"/></g><g class="mark"><title>10 → 10: 0.43</title><rect class="cell" x="605.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.431"/></g><g class="mark"><title>10 → 11: 0.35</title><rect class="cell" x="622.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.349"/></g><g class="mark"><title>10 → 12: 0.14</title><rect class="cell" x="639.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.142"/></g><g class="mark"><title>10 → 13: 0.64</title><rect class="cell" x="656.0" y="235.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.640"/></g><text class="t-tick" x="426.0" y="263.5" text-anchor="end">11</text><g class="mark"><title>11 → 0: 0.47</title><rect class="cell" x="435.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.467"/></g><g class="mark"><title>11 → 1: 0.14</title><rect class="cell" x="452.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.141"/></g><g class="mark"><title>11 → 2: 0.43</title><rect class="cell" x="469.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.430"/></g><g class="mark"><title>11 → 3: 0.45</title><rect class="cell" x="486.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.446"/></g><g class="mark"><title>11 → 4: 0.45</title><rect class="cell" x="503.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.446"/></g><g class="mark"><title>11 → 5: 0.41</title><rect class="cell" x="520.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.411"/></g><g class="mark"><title>11 → 6: 0.45</title><rect class="cell" x="537.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.448"/></g><g class="mark"><title>11 → 7: 0.49</title><rect class="cell" x="554.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.494"/></g><g class="mark"><title>11 → 8: 0.51</title><rect class="cell" x="571.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.508"/></g><g class="mark"><title>11 → 9: 0.52</title><rect class="cell" x="588.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.515"/></g><g class="mark"><title>11 → 10: 0.47</title><rect class="cell" x="605.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.465"/></g><g class="mark"><title>11 → 11: 0.59</title><rect class="cell" x="622.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.586"/></g><g class="mark"><title>11 → 12: 0.51</title><rect class="cell" x="639.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.514"/></g><g class="mark"><title>11 → 13: 0.71</title><rect class="cell" x="656.0" y="252.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.708"/></g><text class="t-tick" x="426.0" y="280.5" text-anchor="end">12</text><g class="mark"><title>12 → 0: 0.54</title><rect class="cell" x="435.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.539"/></g><g class="mark"><title>12 → 1: 0.16</title><rect class="cell" x="452.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.165"/></g><g class="mark"><title>12 → 2: 0.48</title><rect class="cell" x="469.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.477"/></g><g class="mark"><title>12 → 3: 0.49</title><rect class="cell" x="486.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.493"/></g><g class="mark"><title>12 → 4: 0.45</title><rect class="cell" x="503.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.455"/></g><g class="mark"><title>12 → 5: 0.44</title><rect class="cell" x="520.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.439"/></g><g class="mark"><title>12 → 6: 0.50</title><rect class="cell" x="537.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.502"/></g><g class="mark"><title>12 → 7: 0.50</title><rect class="cell" x="554.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.502"/></g><g class="mark"><title>12 → 8: 0.53</title><rect class="cell" x="571.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.528"/></g><g class="mark"><title>12 → 9: 0.50</title><rect class="cell" x="588.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.500"/></g><g class="mark"><title>12 → 10: 0.56</title><rect class="cell" x="605.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.563"/></g><g class="mark"><title>12 → 11: 0.40</title><rect class="cell" x="622.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.400"/></g><g class="mark"><title>12 → 12: 0.64</title><rect class="cell" x="639.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.643"/></g><g class="mark"><title>12 → 13: 0.55</title><rect class="cell" x="656.0" y="269.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.553"/></g><text class="t-tick" x="426.0" y="297.5" text-anchor="end">13</text><g class="mark"><title>13 → 0: 0.72</title><rect class="cell" x="435.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.725"/></g><g class="mark"><title>13 → 1: 0.13</title><rect class="cell" x="452.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.127"/></g><g class="mark"><title>13 → 2: 0.61</title><rect class="cell" x="469.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.606"/></g><g class="mark"><title>13 → 3: 0.62</title><rect class="cell" x="486.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.618"/></g><g class="mark"><title>13 → 4: 0.58</title><rect class="cell" x="503.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.580"/></g><g class="mark"><title>13 → 5: 0.60</title><rect class="cell" x="520.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.597"/></g><g class="mark"><title>13 → 6: 0.69</title><rect class="cell" x="537.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.694"/></g><g class="mark"><title>13 → 7: 0.59</title><rect class="cell" x="554.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.586"/></g><g class="mark"><title>13 → 8: 0.64</title><rect class="cell" x="571.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.644"/></g><g class="mark"><title>13 → 9: 0.57</title><rect class="cell" x="588.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.571"/></g><g class="mark"><title>13 → 10: 0.62</title><rect class="cell" x="605.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.623"/></g><g class="mark"><title>13 → 11: 0.75</title><rect class="cell" x="622.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.753"/></g><g class="mark"><title>13 → 12: 0.80</title><rect class="cell" x="639.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.802"/></g><g class="mark"><title>13 → 13: 0.49</title><rect class="cell" x="656.0" y="286.0" width="15.0" height="15.0" rx="2" style="fill-opacity:0.494"/></g><text class="t-tick" x="442.5" y="312.0" text-anchor="end" transform="rotate(-50 442.5 312.0)">0</text><text class="t-tick" x="459.5" y="312.0" text-anchor="end" transform="rotate(-50 459.5 312.0)">1</text><text class="t-tick" x="476.5" y="312.0" text-anchor="end" transform="rotate(-50 476.5 312.0)">2</text><text class="t-tick" x="493.5" y="312.0" text-anchor="end" transform="rotate(-50 493.5 312.0)">3</text><text class="t-tick" x="510.5" y="312.0" text-anchor="end" transform="rotate(-50 510.5 312.0)">4</text><text class="t-tick" x="527.5" y="312.0" text-anchor="end" transform="rotate(-50 527.5 312.0)">5</text><text class="t-tick" x="544.5" y="312.0" text-anchor="end" transform="rotate(-50 544.5 312.0)">6</text><text class="t-tick" x="561.5" y="312.0" text-anchor="end" transform="rotate(-50 561.5 312.0)">7</text><text class="t-tick" x="578.5" y="312.0" text-anchor="end" transform="rotate(-50 578.5 312.0)">8</text><text class="t-tick" x="595.5" y="312.0" text-anchor="end" transform="rotate(-50 595.5 312.0)">9</text><text class="t-tick" x="612.5" y="312.0" text-anchor="end" transform="rotate(-50 612.5 312.0)">10</text><text class="t-tick" x="629.5" y="312.0" text-anchor="end" transform="rotate(-50 629.5 312.0)">11</text><text class="t-tick" x="646.5" y="312.0" text-anchor="end" transform="rotate(-50 646.5 312.0)">12</text><text class="t-tick" x="663.5" y="312.0" text-anchor="end" transform="rotate(-50 663.5 312.0)">13</text><text class="t-muted" x="553.0" y="338.0" text-anchor="middle">patch column 0 to 13</text><text class="t-muted" x="20.0" y="364.0" text-anchor="start">Rows are patch rows 0 to 13. Shade = weight divided by the largest weight in that head, so each head</text><text class="t-muted" x="20.0" y="381.0" text-anchor="start">is on its own scale. Hover a cell for the value.</text></svg><figcaption>The class token's 196 patch weights in layer 1, drawn on the 14×14 grid, for two heads of the real model. Head 1 (left) puts 0.76 on the class token itself and at most 0.0026 on any patch. Head 8 (right) puts only 0.022 on itself and spreads its weight, mostly along the top row of the picture. Each heatmap is scaled to its own largest weight.</figcaption></figure>

<figure class="fig"><svg viewBox="0 0 760 300" role="img" aria-label="The 14 by 14 patch grid of the cat picture shaded by the class token attention weights of head 8 in layer 1, with the five strongest patches outlined and numbered; all five sit in the top row of the picture and the largest weight is 0.015."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">Head 8 of layer 1: where the [class] token looks, on the patch grid of the picture</text><rect class="s1" x="40.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.77"/><rect class="s1" x="56.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.81"/><rect class="s1" x="72.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.85"/><rect class="s1" x="88.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.84"/><rect class="s1" x="104.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.80"/><rect class="s1" x="120.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.81"/><rect class="s1" x="136.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.85"/><rect class="s1" x="152.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:1.00"/><rect class="s1" x="168.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.76"/><rect class="s1" x="184.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.59"/><rect class="s1" x="200.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.34"/><rect class="s1" x="216.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.47"/><rect class="s1" x="232.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.71"/><rect class="s1" x="248.0" y="44.0" width="15" height="15" rx="1.5" style="fill-opacity:0.76"/><rect class="s1" x="40.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.24"/><rect class="s1" x="56.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="72.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.71"/><rect class="s1" x="88.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.57"/><rect class="s1" x="104.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.49"/><rect class="s1" x="120.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.34"/><rect class="s1" x="136.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.36"/><rect class="s1" x="152.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.57"/><rect class="s1" x="168.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.39"/><rect class="s1" x="184.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="200.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="216.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="232.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.37"/><rect class="s1" x="248.0" y="60.0" width="15" height="15" rx="1.5" style="fill-opacity:0.56"/><rect class="s1" x="40.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.32"/><rect class="s1" x="56.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="72.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="88.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.24"/><rect class="s1" x="104.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="120.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="136.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="152.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="168.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="184.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.31"/><rect class="s1" x="200.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.15"/><rect class="s1" x="216.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="232.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="248.0" y="76.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="40.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="56.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.25"/><rect class="s1" x="72.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.29"/><rect class="s1" x="88.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="104.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="120.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="136.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="152.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="168.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.15"/><rect class="s1" x="184.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="200.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="216.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="232.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.24"/><rect class="s1" x="248.0" y="92.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="40.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.59"/><rect class="s1" x="56.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.62"/><rect class="s1" x="72.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="88.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.24"/><rect class="s1" x="104.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="120.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="136.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="152.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.34"/><rect class="s1" x="168.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="184.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.15"/><rect class="s1" x="200.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="216.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.24"/><rect class="s1" x="232.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="248.0" y="108.0" width="15" height="15" rx="1.5" style="fill-opacity:0.43"/><rect class="s1" x="40.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.66"/><rect class="s1" x="56.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.44"/><rect class="s1" x="72.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="88.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="104.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="120.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="136.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="152.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.36"/><rect class="s1" x="168.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="184.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="200.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="216.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="232.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="248.0" y="124.0" width="15" height="15" rx="1.5" style="fill-opacity:0.66"/><rect class="s1" x="40.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.64"/><rect class="s1" x="56.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.23"/><rect class="s1" x="72.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="88.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="104.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.31"/><rect class="s1" x="120.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="136.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="152.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="168.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="184.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="200.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="216.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="232.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="248.0" y="140.0" width="15" height="15" rx="1.5" style="fill-opacity:0.67"/><rect class="s1" x="40.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.48"/><rect class="s1" x="56.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="72.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="88.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.25"/><rect class="s1" x="104.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="120.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="136.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.58"/><rect class="s1" x="152.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.36"/><rect class="s1" x="168.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="184.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.15"/><rect class="s1" x="200.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="216.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.28"/><rect class="s1" x="232.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="248.0" y="156.0" width="15" height="15" rx="1.5" style="fill-opacity:0.69"/><rect class="s1" x="40.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.35"/><rect class="s1" x="56.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.20"/><rect class="s1" x="72.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="88.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.34"/><rect class="s1" x="104.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="120.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="136.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.64"/><rect class="s1" x="152.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.48"/><rect class="s1" x="168.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="184.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="200.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="216.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.17"/><rect class="s1" x="232.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="248.0" y="172.0" width="15" height="15" rx="1.5" style="fill-opacity:0.64"/><rect class="s1" x="40.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.34"/><rect class="s1" x="56.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.26"/><rect class="s1" x="72.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.16"/><rect class="s1" x="88.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.22"/><rect class="s1" x="104.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="120.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="136.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.51"/><rect class="s1" x="152.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.57"/><rect class="s1" x="168.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.23"/><rect class="s1" x="184.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.19"/><rect class="s1" x="200.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.30"/><rect class="s1" x="216.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="232.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.37"/><rect class="s1" x="248.0" y="188.0" width="15" height="15" rx="1.5" style="fill-opacity:0.75"/><rect class="s1" x="40.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.40"/><rect class="s1" x="56.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.18"/><rect class="s1" x="72.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.15"/><rect class="s1" x="88.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.23"/><rect class="s1" x="104.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.45"/><rect class="s1" x="120.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.51"/><rect class="s1" x="136.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="152.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.57"/><rect class="s1" x="168.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.27"/><rect class="s1" x="184.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="200.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.49"/><rect class="s1" x="216.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.41"/><rect class="s1" x="232.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.23"/><rect class="s1" x="248.0" y="204.0" width="15" height="15" rx="1.5" style="fill-opacity:0.68"/><rect class="s1" x="40.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="56.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.23"/><rect class="s1" x="72.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.49"/><rect class="s1" x="88.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="104.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="120.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.47"/><rect class="s1" x="136.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.50"/><rect class="s1" x="152.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.54"/><rect class="s1" x="168.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.56"/><rect class="s1" x="184.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.56"/><rect class="s1" x="200.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.52"/><rect class="s1" x="216.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.63"/><rect class="s1" x="232.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.56"/><rect class="s1" x="248.0" y="220.0" width="15" height="15" rx="1.5" style="fill-opacity:0.74"/><rect class="s1" x="40.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.59"/><rect class="s1" x="56.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.25"/><rect class="s1" x="72.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.53"/><rect class="s1" x="88.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.54"/><rect class="s1" x="104.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.51"/><rect class="s1" x="120.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.49"/><rect class="s1" x="136.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="152.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="168.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.57"/><rect class="s1" x="184.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="200.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.61"/><rect class="s1" x="216.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.46"/><rect class="s1" x="232.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.68"/><rect class="s1" x="248.0" y="236.0" width="15" height="15" rx="1.5" style="fill-opacity:0.60"/><rect class="s1" x="40.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.75"/><rect class="s1" x="56.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.21"/><rect class="s1" x="72.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.65"/><rect class="s1" x="88.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.66"/><rect class="s1" x="104.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.62"/><rect class="s1" x="120.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.64"/><rect class="s1" x="136.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.72"/><rect class="s1" x="152.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.63"/><rect class="s1" x="168.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.68"/><rect class="s1" x="184.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.61"/><rect class="s1" x="200.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.66"/><rect class="s1" x="216.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.78"/><rect class="s1" x="232.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.82"/><rect class="s1" x="248.0" y="252.0" width="15" height="15" rx="1.5" style="fill-opacity:0.54"/><rect class="box-on" x="151.0" y="43.0" width="16.0" height="16.0" rx="2"/><text class="t-cell on" x="160.0" y="56.0" text-anchor="middle">1</text><rect class="box-on" x="71.0" y="43.0" width="16.0" height="16.0" rx="2"/><text class="t-cell on" x="80.0" y="56.0" text-anchor="middle">2</text><rect class="box-on" x="135.0" y="43.0" width="16.0" height="16.0" rx="2"/><text class="t-cell on" x="144.0" y="56.0" text-anchor="middle">3</text><rect class="box-on" x="87.0" y="43.0" width="16.0" height="16.0" rx="2"/><text class="t-cell on" x="96.0" y="56.0" text-anchor="middle">4</text><path class="edge" d="M40.0,266.0 L40.0,274.0 L264.0,274.0 L264.0,266.0"/><text class="t-tick" x="152.0" y="292.0" text-anchor="middle">the picture, 14 × 14 patches</text><text class="t-code" x="320.0" y="56.0" text-anchor="start">rank   weight   patch (row, col)</text><text class="t-code" x="320.0" y="76.0" text-anchor="start">  -    0.0219   [class] itself</text><text class="t-code" x="320.0" y="94.0" text-anchor="start">  1    0.0146   patch   7  (row  0, col  7)</text><text class="t-code" x="320.0" y="112.0" text-anchor="start">  2    0.0122   patch   2  (row  0, col  2)</text><text class="t-code" x="320.0" y="130.0" text-anchor="start">  3    0.0122   patch   6  (row  0, col  6)</text><text class="t-code" x="320.0" y="148.0" text-anchor="start">  4    0.0120   patch   3  (row  0, col  3)</text><text class="t-tick" x="320.0" y="200.0" text-anchor="start">All 197 weights of this row add up to 1, so no single patch</text><text class="t-tick" x="320.0" y="216.0" text-anchor="start">can be large: the top patch has 0.015, and the top row of the</text><text class="t-tick" x="320.0" y="232.0" text-anchor="start">picture (the red sofa behind the cats) takes most of the weight.</text><text class="t-tick" x="320.0" y="248.0" text-anchor="start">In head 1 the same row puts 0.76 on [class] itself and at most</text><text class="t-tick" x="320.0" y="264.0" text-anchor="start">0.0026 on any patch. Part 5 looks at what later layers attend to.</text></svg><figcaption>Head 8 of layer 1 on top of the patch grid of the picture, with the five strongest patches outlined and numbered. All five are in the top row (patches 7, 2, 6 and 3 of row 0), and the largest weight is 0.0146: with 197 weights that add up to 1, no single patch can be large.</figcaption></figure>

The weighted sum, the join of the heads, the output matrix and the residual, still for the class token:

```text
  SA_1 for [class] = sum_j A_0j v_j: (64,), first 6: [ -0.019,   0.029,  -0.053,   0.131,  -0.007,   0.001]
  concat of 12 heads: (197, 768);  times U_msa (768, 768) + bias: (197, 768)
  Eq. 2: z'_1 = MSA + z_0, [class] first 6: [ -0.178,  -0.015,  -0.848,  -0.393,   0.661,   0.162]
```

<figure class="fig"><svg viewBox="0 0 760 276" role="img" aria-label="The shapes inside multi-head self-attention for ViT-B/16: the normalised input of 197 by 768 goes through 12 heads, each computing q, k, v of 197 by 64, an attention matrix of 197 by 197 and an output of 197 by 64; the heads are concatenated to 197 by 768 and multiplied by U_msa of 768 by 768."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Inside MSA for ViT-B/16: 197 tokens, 12 heads of 64</text><rect class="box-ghost" x="26.0" y="66.0" width="90.6" height="124.2" rx="5"/><rect class="s1" x="30.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="38.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="46.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="55.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="63.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="72.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="80.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="88.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="97.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="105.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="30.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="38.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="46.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="55.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="63.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="72.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="80.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="88.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="97.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="105.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="30.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="38.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="46.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="55.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="63.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="72.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="80.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="88.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="97.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="105.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="30.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="38.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="46.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="55.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="63.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="72.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="80.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="88.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="97.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="105.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="30.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="38.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="46.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="55.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="63.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="72.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="80.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="88.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="97.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="105.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="30.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="38.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="46.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="55.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="63.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="72.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="80.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="88.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="97.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="105.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="30.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="38.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="46.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="55.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="63.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="72.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="80.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="88.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="97.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="105.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="30.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="38.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="46.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="55.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="63.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="72.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="80.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="88.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="97.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="105.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="30.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="38.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="46.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="55.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="63.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="72.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="80.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="88.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="97.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="105.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="30.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="38.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="46.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="55.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="63.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="72.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="80.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="88.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="97.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="105.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="30.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="38.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="46.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="55.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="63.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="72.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="80.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="88.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="97.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="105.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="30.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="38.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="46.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="55.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="63.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="72.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="80.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="88.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="97.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="105.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="30.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="38.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="46.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="55.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="63.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="72.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="80.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="88.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="97.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="105.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="30.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="38.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="46.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="55.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="63.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s1" x="72.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s1" x="80.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s1" x="88.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s1" x="97.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s1" x="105.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><text class="t-math" x="71.3" y="58.0" text-anchor="middle">LN(z)</text><text class="t-tick" x="71.3" y="206.2" text-anchor="middle">197 × 768</text><line class="edge" x1="116.6" y1="128.1" x2="134.6" y2="128.1" marker-end="url(#ah)"/><rect class="box" x="152.6" y="70.0" width="196.0" height="118.0" rx="8"/><rect class="box" x="145.6" y="63.0" width="196.0" height="118.0" rx="8"/><rect class="box" x="138.6" y="56.0" width="196.0" height="118.0" rx="8"/><text class="t-note" x="236.6" y="76.0" text-anchor="middle">head h (12 of these)</text><text class="t-tick" x="148.6" y="96.0" text-anchor="start">q, k, v = LN(z) U_qkv</text><text class="t-tick" x="148.6" y="112.0" text-anchor="start">each 197 × 64</text><text class="t-tick" x="148.6" y="132.0" text-anchor="start">A = softmax(q kᵀ/√64)</text><text class="t-tick" x="148.6" y="148.0" text-anchor="start">197 × 197, rows sum to 1</text><text class="t-tick" x="148.6" y="166.0" text-anchor="start">SA_h = A v: 197 × 64</text><line class="edge" x1="350.6" y1="120.0" x2="366.6" y2="120.0" marker-end="url(#ah)"/><rect class="box-ghost" x="366.6" y="66.0" width="90.6" height="124.2" rx="5"/><rect class="s3" x="370.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="379.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="387.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="395.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="404.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="412.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="421.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="429.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="437.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="446.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="370.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="379.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="387.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="395.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="404.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="412.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="421.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="429.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="437.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="446.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="370.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="379.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="387.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="395.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="404.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="412.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="429.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="437.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="446.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="370.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="379.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="387.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="395.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="404.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="412.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="429.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="437.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="446.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="370.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="379.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="387.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="395.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="404.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="412.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="421.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="429.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="437.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="446.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="370.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="379.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="387.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="395.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="404.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="412.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="421.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="429.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="437.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="446.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="370.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="379.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="387.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="395.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="404.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="412.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="421.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="429.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="437.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="446.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="370.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="379.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="387.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="395.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="404.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="412.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="429.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="437.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="446.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="370.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="379.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="387.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="395.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="404.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="412.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="429.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="437.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="446.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="370.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="379.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="387.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="395.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="404.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="412.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="421.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="429.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="437.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="446.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="370.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="379.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="387.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="395.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="404.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="412.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="421.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="429.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="437.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="446.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="370.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="379.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="387.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="395.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="404.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="412.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="421.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="429.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="437.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="446.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="370.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="379.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="387.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="395.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="404.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="412.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="421.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="429.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="437.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="446.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="370.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="379.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="387.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="395.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="404.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s3" x="412.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s3" x="421.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s3" x="429.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s3" x="437.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s3" x="446.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><text class="t-math" x="411.9" y="58.0" text-anchor="middle">concat</text><text class="t-tick" x="411.9" y="206.2" text-anchor="middle">197 × 768</text><line class="edge-dim" x1="377.5" y1="68.0" x2="377.5" y2="188.2"/><line class="edge-dim" x1="384.4" y1="68.0" x2="384.4" y2="188.2"/><line class="edge-dim" x1="391.2" y1="68.0" x2="391.2" y2="188.2"/><line class="edge-dim" x1="398.1" y1="68.0" x2="398.1" y2="188.2"/><line class="edge-dim" x1="405.0" y1="68.0" x2="405.0" y2="188.2"/><line class="edge-dim" x1="411.9" y1="68.0" x2="411.9" y2="188.2"/><line class="edge-dim" x1="418.8" y1="68.0" x2="418.8" y2="188.2"/><line class="edge-dim" x1="425.7" y1="68.0" x2="425.7" y2="188.2"/><line class="edge-dim" x1="432.6" y1="68.0" x2="432.6" y2="188.2"/><line class="edge-dim" x1="439.4" y1="68.0" x2="439.4" y2="188.2"/><line class="edge-dim" x1="446.3" y1="68.0" x2="446.3" y2="188.2"/><line class="edge" x1="467.2" y1="120.0" x2="483.2" y2="120.0" marker-end="url(#ah)"/><rect class="box-ghost" x="483.2" y="66.0" width="90.6" height="90.6" rx="5"/><rect class="s4" x="487.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="495.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="504.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="512.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="520.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="529.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="537.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="546.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="554.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="562.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="487.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="495.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="504.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="512.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="520.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="529.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="537.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="546.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="554.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="562.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="487.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="495.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="504.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="512.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="520.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="529.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="537.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="546.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="554.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="562.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="487.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="495.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="504.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="512.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="520.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="529.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="537.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="546.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="554.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="562.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="487.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="495.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="504.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="512.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="520.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="529.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="537.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="546.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="554.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="562.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="487.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="495.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="504.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="512.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="520.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="529.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="537.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="546.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="554.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="562.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="487.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="495.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="504.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="512.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="520.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="529.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="537.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="546.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="554.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="562.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="487.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="495.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="504.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="512.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="520.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="529.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="537.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="546.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="554.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="562.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="487.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="495.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="504.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="512.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="520.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="529.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="537.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="546.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="554.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="562.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="487.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="495.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="504.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="512.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="520.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s4" x="529.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s4" x="537.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s4" x="546.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s4" x="554.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s4" x="562.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><text class="t-math" x="528.5" y="58.0" text-anchor="middle">U_msa</text><text class="t-tick" x="528.5" y="172.6" text-anchor="middle">768 × 768</text><line class="edge" x1="583.8" y1="120.0" x2="599.8" y2="120.0" marker-end="url(#ah)"/><rect class="box-ghost" x="599.8" y="66.0" width="90.6" height="124.2" rx="5"/><rect class="s2" x="603.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="612.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="620.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="629.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="637.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="645.8" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="654.2" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="662.6" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="671.0" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="679.4" y="70.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="603.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="612.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="620.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="629.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="637.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="645.8" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="654.2" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="662.6" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="671.0" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="679.4" y="78.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="603.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="612.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="620.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="629.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="637.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="645.8" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="654.2" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="662.6" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="671.0" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="679.4" y="86.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="603.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="612.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="620.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="629.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="637.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="645.8" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="654.2" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="662.6" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="671.0" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="679.4" y="95.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="603.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="612.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="620.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="629.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="637.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="645.8" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="654.2" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="662.6" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="671.0" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="679.4" y="103.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="603.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="612.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="620.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="629.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="637.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="645.8" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="654.2" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="662.6" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="671.0" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="679.4" y="112.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="603.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="612.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="620.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="629.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="637.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="645.8" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="654.2" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="662.6" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="671.0" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="679.4" y="120.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="603.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="612.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="620.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="629.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="637.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="645.8" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="654.2" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="662.6" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="671.0" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="679.4" y="128.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="603.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="612.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="620.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="629.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="637.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="645.8" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="654.2" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="662.6" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="671.0" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="679.4" y="137.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="603.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="612.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="620.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="629.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="637.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="645.8" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="654.2" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="662.6" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="671.0" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="679.4" y="145.6" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="603.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="612.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="620.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="629.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="637.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="645.8" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="654.2" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="662.6" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="671.0" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="679.4" y="154.0" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="603.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="612.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="620.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="629.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="637.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="645.8" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="654.2" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="662.6" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="671.0" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="679.4" y="162.4" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="603.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="612.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="620.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="629.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="637.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="645.8" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="654.2" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="662.6" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="671.0" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="679.4" y="170.8" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="603.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="612.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="620.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="629.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="637.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><rect class="s2" x="645.8" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.49"/><rect class="s2" x="654.2" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.69"/><rect class="s2" x="662.6" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.55"/><rect class="s2" x="671.0" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.42"/><rect class="s2" x="679.4" y="179.2" width="7" height="7" rx="2" style="fill-opacity:0.62"/><text class="t-math" x="645.1" y="58.0" text-anchor="middle">MSA(z)</text><text class="t-tick" x="645.1" y="206.2" text-anchor="middle">197 × 768</text><text class="t-muted" x="20.0" y="228.0" text-anchor="start">12 heads × 64 numbers = 768 = D, so the concatenation has the same width as the input.</text><text class="t-muted" x="20.0" y="245.0" text-anchor="start">Setting D_h = D/k keeps the parameter count the same whatever the number of heads k is (Appendix A).</text><text class="t-muted" x="20.0" y="262.0" text-anchor="start">Attention parameters per layer: four 768 × 768 matrices (q, k, v, U_msa) and four biases = 2,362,368.</text></svg><figcaption>The shapes inside multi-head self-attention for ViT-B/16. The normalised input (197×768) goes through 12 heads; in each, q, k and v are 197×64, the attention matrix is 197×197 with rows that sum to 1, and the output is 197×64. The 12 outputs are concatenated to 197×768 and multiplied by U_msa (768×768). Four 768×768 matrices and four biases make 2,362,368 attention parameters per layer.</figcaption></figure>

### Equation 3: LayerNorm, MLP, residual

The idea in one sentence: normalise again, push every token through the same small two-layer network, and add the result back.

$$
\mathbf{z}_\ell = \text{MLP}(\text{LN}(\mathbf{z}'_\ell)) + \mathbf{z}'_\ell, \qquad \ell = 1 \dots L
$$

where:

- $$\mathbf{z}'_\ell$$ is the output of Equation 2, $$197 \times 768$$;
- $$\text{LN}$$ is a second LayerNorm with its own $$\gamma, \beta$$;
- $$\text{MLP}(x) = \text{GELU}(x W_1 + b_1)\, W_2 + b_2$$ with $$W_1$$ of $$768 \times 3{,}072$$ and $$W_2$$ of $$3{,}072 \times 768$$, applied to each of the 197 rows separately;
- $$\mathbf{z}_\ell$$ is the output of layer $$\ell$$, $$197 \times 768$$, the input of the next layer.

And GELU itself, number by number:

$$
\text{GELU}(x) = x \cdot \Phi(x), \qquad \Phi(x) = \tfrac{1}{2}\left(1 + \operatorname{erf}\!\left(x/\sqrt{2}\right)\right)
$$

where $$\Phi(x)$$ is the probability that a standard normal random number is below $$x$$: close to 0 for very negative $$x$$, 0.5 at 0, close to 1 for large $$x$$. So GELU keeps a number roughly in proportion to how "positive" it is. The script computes it from `torch.erf` and compares with the library's `F.gelu`:

```text
  GELU(x) = x * Phi(x) at a few values, against torch.nn.functional.gelu:
    x:                 -3.00   -2.00   -1.00   -0.50    0.00    0.50    1.00    2.00    3.00
    Phi(x):           0.0013  0.0228  0.1587  0.3085  0.5000  0.6915  0.8413  0.9772  0.9987
    x*Phi(x):        -0.0040 -0.0455 -0.1587 -0.1543  0.0000  0.3457  0.8413  1.9545  2.9960
    F.gelu(x):       -0.0040 -0.0455 -0.1587 -0.1543  0.0000  0.3457  0.8413  1.9545  2.9960
    relu(x):          0.0000  0.0000  0.0000  0.0000  0.0000  0.5000  1.0000  2.0000  3.0000
```

$$\text{GELU}(-1) = -1 \times 0.1587 = -0.159$$, not 0 as ReLU would give; $$\text{GELU}(2) = 2 \times 0.9772 = 1.955$$, almost the 2 of ReLU. The two agree exactly with the library's function. Now the MLP on the class token in layer 1:

```python
ln2 = lay.layernorm_after(z_prime)                               # second LayerNorm, own gamma and beta
h = ln2 @ lay.mlp.fc1.weight.T + lay.mlp.fc1.bias                # (197, 3072)
mlp = gelu_exact(h) @ lay.mlp.fc2.weight.T + lay.mlp.fc2.bias    # (197, 768)
z1 = mlp + z_prime                                               # Eq. 3
```

```text
  LN(z'_1) [class] first 6:   [ -0.055,  -0.043,  -0.247,   0.003,   0.249,   0.020]
  MLP: LN(z') W_1 + b_1 -> (197, 3072); GELU; W_2 + b_2 -> (197, 768)
  [class] before GELU, first 6: [ -2.016,  -2.504,   0.053,  -1.484,  -0.649,   0.060]
  [class] after GELU, first 6:  [ -0.044,  -0.015,   0.028,  -0.102,  -0.168,   0.031]
  share of the 3072 numbers that are positive before GELU ([class]): 0.071
  Eq. 3: z_1 = MLP + z'_1, [class] first 6: [ -0.212,  -0.046,  -0.840,  -0.748,   0.774,   0.083]
  max |difference|, our z_1 vs the library hidden_states[1]: 6.68e-06
  max |difference|, our head-1 weights vs the library attentions[0] (all 12 heads): 2.68e-06
```

Only 7.1% of the 3,072 hidden numbers of the class token are positive before GELU in this layer; most of the wide hidden layer is "off" for this token, and the off numbers come out small but not zero ($$-2.016 \to -0.044$$). Our $$\mathbf{z}_1$$ matches the library's output of layer 1 to $$6.7 \times 10^{-6}$$, and our attention weights match its weights to $$2.7 \times 10^{-6}$$. So Equations 2 and 3, as written above, are the complete layer.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="Left, the MLP block of one ViT layer: 768 numbers widened to 3072 by W1, passed through GELU and narrowed back to 768 by W2. Right, the GELU curve x times Phi of x computed in code, with ReLU dashed for comparison; GELU of minus 1 is minus 0.159 and GELU of 2 is 1.955."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">The MLP block: 768 → 3072 → 768 with GELU in between</text><rect class="box" x="60.0" y="252.0" width="110.0" height="30.0" rx="6"/><text class="t-tick" x="115.0" y="271.5" text-anchor="middle">LN(z′): 768</text><line class="edge" x1="115.0" y1="250.0" x2="115.0" y2="212.0" marker-end="url(#ah)"/><text class="t-tick" x="125.0" y="236.0" text-anchor="start">W₁ (768 × 3072) + b₁</text><rect class="box-4" x="20.0" y="180.0" width="190.0" height="30.0" rx="6"/><text class="t-tick" x="115.0" y="199.5" text-anchor="middle">h: 3072 numbers</text><line class="edge" x1="115.0" y1="178.0" x2="115.0" y2="140.0" marker-end="url(#ah)"/><text class="t-tick" x="125.0" y="164.0" text-anchor="start">GELU, number by number</text><rect class="box-4" x="20.0" y="108.0" width="190.0" height="30.0" rx="6"/><text class="t-tick" x="115.0" y="127.5" text-anchor="middle">GELU(h): 3072</text><line class="edge" x1="115.0" y1="106.0" x2="115.0" y2="68.0" marker-end="url(#ah)"/><text class="t-tick" x="125.0" y="92.0" text-anchor="start">W₂ (3072 × 768) + b₂</text><rect class="box" x="60.0" y="36.0" width="110.0" height="30.0" rx="6"/><text class="t-tick" x="115.0" y="55.5" text-anchor="middle">MLP(…): 768</text><line class="axis" x1="440" y1="226.6" x2="730" y2="226.6"/><line class="axis" x1="585.0" y1="44" x2="585.0" y2="254"/><text class="t-tick" x="440.0" y="242.6" text-anchor="middle">-4</text><text class="t-tick" x="512.5" y="242.6" text-anchor="middle">-2</text><text class="t-tick" x="657.5" y="242.6" text-anchor="middle">2</text><text class="t-tick" x="730.0" y="242.6" text-anchor="middle">4</text><line class="grid" x1="440" y1="181.0" x2="730" y2="181.0"/><text class="t-tick" x="579.0" y="185.0" text-anchor="end">1</text><line class="grid" x1="440" y1="135.3" x2="730" y2="135.3"/><text class="t-tick" x="579.0" y="139.3" text-anchor="end">2</text><line class="grid" x1="440" y1="89.7" x2="730" y2="89.7"/><text class="t-tick" x="579.0" y="93.7" text-anchor="end">3</text><line class="grid" x1="440" y1="44.0" x2="730" y2="44.0"/><text class="t-tick" x="579.0" y="48.0" text-anchor="end">4</text><polyline class="edge-dim" points="440.0,226.6 443.6,226.6 447.3,226.6 450.9,226.6 454.5,226.6 458.1,226.6 461.7,226.6 465.4,226.6 469.0,226.6 472.6,226.6 476.2,226.6 479.9,226.6 483.5,226.6 487.1,226.6 490.8,226.6 494.4,226.6 498.0,226.6 501.6,226.6 505.2,226.6 508.9,226.6 512.5,226.6 516.1,226.6 519.8,226.6 523.4,226.6 527.0,226.6 530.6,226.6 534.3,226.6 537.9,226.6 541.5,226.6 545.1,226.6 548.8,226.6 552.4,226.6 556.0,226.6 559.6,226.6 563.3,226.6 566.9,226.6 570.5,226.6 574.1,226.6 577.8,226.6 581.4,226.6 585.0,226.6 588.6,222.0 592.2,217.5 595.9,212.9 599.5,208.3 603.1,203.8 606.7,199.2 610.4,194.7 614.0,190.1 617.6,185.5 621.2,181.0 624.9,176.4 628.5,171.8 632.1,167.3 635.7,162.7 639.4,158.1 643.0,153.6 646.6,149.0 650.2,144.4 653.9,139.9 657.5,135.3 661.1,130.7 664.8,126.2 668.4,121.6 672.0,117.0 675.6,112.5 679.2,107.9 682.9,103.3 686.5,98.8 690.1,94.2 693.8,89.7 697.4,85.1 701.0,80.5 704.6,76.0 708.3,71.4 711.9,66.8 715.5,62.3 719.1,57.7 722.7,53.1 726.4,48.6 730.0,44.0"/><polyline class="l4" points="440.0,226.6 443.6,226.6 447.3,226.6 450.9,226.6 454.5,226.6 458.1,226.6 461.7,226.7 465.4,226.7 469.0,226.7 472.6,226.7 476.2,226.8 479.9,226.9 483.5,226.9 487.1,227.0 490.8,227.2 494.4,227.3 498.0,227.5 501.6,227.7 505.2,228.0 508.9,228.3 512.5,228.7 516.1,229.1 519.8,229.6 523.4,230.1 527.0,230.6 530.6,231.2 534.3,231.8 537.9,232.4 541.5,232.9 545.1,233.4 548.8,233.9 552.4,234.2 556.0,234.3 559.6,234.3 563.3,234.1 566.9,233.7 570.5,232.9 574.1,231.8 577.8,230.5 581.4,228.7 585.0,226.6 588.6,224.1 592.2,221.3 595.9,218.1 599.5,214.6 603.1,210.8 606.7,206.7 610.4,202.4 614.0,197.8 617.6,193.1 621.2,188.2 624.9,183.2 628.5,178.1 632.1,173.0 635.7,167.9 639.4,162.7 643.0,157.6 646.6,152.5 650.2,147.4 653.9,142.4 657.5,137.4 661.1,132.5 664.8,127.6 668.4,122.7 672.0,117.9 675.6,113.2 679.2,108.5 682.9,103.8 686.5,99.1 690.1,94.5 693.8,89.8 697.4,85.2 701.0,80.6 704.6,76.0 708.3,71.4 711.9,66.9 715.5,62.3 719.1,57.7 722.7,53.1 726.4,48.6 730.0,44.0"/><g class="mark"><title>GELU(-3.0) = -0.0040</title><circle class="s4 ring" cx="476.2" cy="226.8" r="4"/></g><g class="mark"><title>GELU(-2.0) = -0.0455</title><circle class="s4 ring" cx="512.5" cy="228.7" r="4"/></g><g class="mark"><title>GELU(-1.0) = -0.1587</title><circle class="s4 ring" cx="548.8" cy="233.9" r="4"/></g><g class="mark"><title>GELU(-0.5) = -0.1543</title><circle class="s4 ring" cx="566.9" cy="233.7" r="4"/></g><g class="mark"><title>GELU(0.0) = 0.0000</title><circle class="s4 ring" cx="585.0" cy="226.6" r="4"/></g><g class="mark"><title>GELU(0.5) = 0.3457</title><circle class="s4 ring" cx="603.1" cy="210.8" r="4"/></g><g class="mark"><title>GELU(1.0) = 0.8413</title><circle class="s4 ring" cx="621.2" cy="188.2" r="4"/></g><g class="mark"><title>GELU(2.0) = 1.9545</title><circle class="s4 ring" cx="657.5" cy="137.4" r="4"/></g><g class="mark"><title>GELU(3.0) = 2.9960</title><circle class="s4 ring" cx="693.8" cy="89.8" r="4"/></g><text class="t-note" x="444.0" y="56.0" text-anchor="start">GELU(x) = x · Φ(x)</text><text class="t-muted" x="444.0" y="72.0" text-anchor="start">dashed: ReLU = max(0, x)</text><text class="t-tick" x="443.6" y="192.4" text-anchor="start">GELU(−1) = −0.159</text><text class="t-tick" x="649.5" y="141.4" text-anchor="end">GELU(2) = 1.955</text><text class="t-muted" x="20.0" y="296.0" text-anchor="start">In layer 1 only 7.1% of the 3072 numbers of the [class] token are positive before GELU;</text><text class="t-muted" x="20.0" y="313.0" text-anchor="start">the negative ones come out small but not zero.</text><text class="t-muted" x="20.0" y="330.0" text-anchor="start">MLP parameters per layer: 768·3072 + 3072 + 3072·768 + 768 = 4,722,432 (8D² + 5D).</text></svg><figcaption>Left: the MLP block of one layer, 768 numbers widened to 3,072 by W₁, passed through GELU and narrowed back to 768 by W₂; 4,722,432 parameters per layer, two thirds of the layer. Right: the GELU curve x·Φ(x) computed in code, with ReLU dashed; GELU(−1) = −0.159 and GELU(2) = 1.955.</figcaption></figure>

**All twelve layers.** The same two equations run 12 times, each layer with its own weights. Our loop against the library, layer by layer:

```text
== 5. all 12 layers, our loop vs the library ==
  layer  1: z_1 (197, 768)   max |difference| vs hidden_states[1]: 6.68e-06   (largest value in z_1:    21.2, so relative 3.1e-07)
  layer  2: z_2 (197, 768)   max |difference| vs hidden_states[2]: 8.34e-06   (largest value in z_2:    23.7, so relative 3.5e-07)
  layer  3: z_3 (197, 768)   max |difference| vs hidden_states[3]: 9.54e-06   (largest value in z_3:    24.1, so relative 4.0e-07)
  layer  4: z_4 (197, 768)   max |difference| vs hidden_states[4]: 8.39e-05   (largest value in z_4:    56.0, so relative 1.5e-06)
  layer  5: z_5 (197, 768)   max |difference| vs hidden_states[5]: 9.16e-04   (largest value in z_5:   336.9, so relative 2.7e-06)
  layer  6: z_6 (197, 768)   max |difference| vs hidden_states[6]: 1.50e-03   (largest value in z_6:   989.2, so relative 1.5e-06)
  ... (layers 7 to 11: 1.65e-03 to 1.77e-03, relative 1.0e-06 to 1.1e-06)
  layer 12: z_12 (197, 768)   max |difference| vs hidden_states[12]: 1.83e-03   (largest value in z_12:  1699.6, so relative 1.1e-06)
  the final logits from our z_12 vs the library differ by at most 2.86e-06 (see step 6)
```

An honesty note on the numbers in the middle column: the absolute difference grows from $$7 \times 10^{-6}$$ in layer 1 to $$1.8 \times 10^{-3}$$ in layer 12. That is not a bug in the equations. The residual stream of a trained ViT contains a few enormous values (the largest number in $$\mathbf{z}_{12}$$ is about 1,700, against 21 in $$\mathbf{z}_1$$), and 32-bit floating point keeps about seven significant digits, so the rounding noise grows with the values. Relative to the largest value the difference stays at about $$10^{-6}$$ in every layer, and the final class scores agree to $$3 \times 10^{-6}$$.

### Equation 4: the image representation

The idea in one sentence: after the last layer, normalise the class token one more time; that vector is the picture.

$$
\mathbf{y} = \text{LN}(\mathbf{z}_L^0)
$$

where $$\mathbf{z}_L^0$$ is row 0 (the class token) of the last layer's output, 768 numbers, and $$\text{LN}$$ is a final LayerNorm with its own $$\gamma, \beta$$. $$\mathbf{y}$$, also 768 numbers, is the image representation. The 196 patch rows of $$\mathbf{z}_L$$ are computed and then ignored. The head is not in the equation; in the fine-tuned model it is $$\text{logits} = \mathbf{y}\,W_{\text{head}} + b_{\text{head}}$$ with $$W_{\text{head}}$$ of $$768 \times 1000$$:

```python
y = model.vit.layernorm(zL[0])                                 # Eq. 4, 768 numbers
logits = y @ model.classifier.weight.T + model.classifier.bias # (1000,)
probs = torch.softmax(logits, -1)
```

```text
== 6. Eq. 4: y = LN(z_L^0), then the classification head ==
  z_L^0 (the [class] token after 12 layers), first 6: [  2.313,   5.512,  11.788,   0.577,   6.548,  -2.913]   mean 0.073 var 34.490
  y = LN(z_L^0), first 6: [  0.294,   0.835,   1.904,   0.081,   1.039,  -0.518]   max |difference| vs the library: 3.81e-06
  head of the fine-tuned checkpoint: Linear(in_features=768, out_features=1000, bias=True)   (one linear layer, as Section 3.1 says for fine-tuning)
  logits = y W_head + b: (1000,)   max |difference| vs model.logits: 2.86e-06
  top-5 ImageNet classes:
     0.937  class 285  Egyptian cat
     0.038  class 281  tabby, tabby cat
     0.014  class 282  tiger cat
     0.003  class 287  lynx, catamount
     0.001  class 284  Siamese cat, Siamese
```

The model says "Egyptian cat" with probability 0.937, and the next four guesses are all cats. Everything from the 50,176 pixels to this answer was computed above with nothing but matrix multiplications, LayerNorm, softmax and GELU, and matched the library at every step.

## Inductive bias {§3.1}

> [!PAPER] · §3.1, Inductive bias
> [![The Inductive bias paragraph: we note that Vision Transformer has much less image-specific inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model. In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global. The two-dimensional neighborhood structure is used very sparingly: in the beginning of the model by cutting the image into patches and at fine-tuning time for adjusting the position embeddings for images of different resolution. Other than that, the position embeddings at initialization time carry no information about the 2D positions of the patches and all spatial relations between the patches have to be learned from scratch](https://ishwarj.com/img/papers/vit/p2-inductive.pngit/p2-inductive.png)](/img/papers/vit/p2-inductive.png)
>
> **What it says:** ViT "has much less image-specific inductive bias than CNNs". In a CNN, "locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer". In ViT "only MLP layers are local and translationally equivariant, while the self-attention layers are global". The 2D grid is used "very sparingly": when "cutting the image into patches" and at fine-tuning time when "adjusting the position embeddings for images of different resolution". The position embeddings "at initialization time carry no information about the 2D positions of the patches", so "all spatial relations between the patches have to be learned from scratch".
>
> **What it means:**
>
> This paragraph restates the thesis of the introduction in the words of the model. A CNN has its assumptions about pictures built into every layer. ViT has them in only two places: the cut into patches, and the resizing of position embeddings at fine-tuning time.
>
> That is the mechanism behind the paper's main finding. Less built-in knowledge means more must be learned from data. So ViT loses on small datasets and wins on huge ones (Part 4).
>
> It also tells you where to look for the learned knowledge: in the position embeddings and the attention patterns. Part 5 looks there.

> [!DEFINITION] Inductive bias
> The assumptions a model has before it sees any data. A CNN assumes that nearby pixels matter most and that a cat is a cat wherever it appears. ViT assumes almost nothing about pictures. Part 1 has the long version.

> [!DEFINITION] Translation equivariance
> If the input shifts, the output shifts the same way and is otherwise unchanged. A convolution has it because the same filter is used at every position. In ViT the MLP has it in a weak sense (the same weights act on every token), but shifting a picture by less than a patch changes every patch's content, so the model as a whole does not.

The paragraph's two claims about ViT are easy to confirm from what we computed. The MLP is "local and translationally equivariant" because it acts on one token at a time with the same $$W_1, W_2$$ for every token; the attention is "global" because every row of the $$197 \times 197$$ matrix had a non-zero weight for every column (the smallest weight in the class token's row was 0.0005, not 0). And the position embeddings carry no 2D information at the start because they are 197 independent learnable rows; nothing in Equation 1 says that row 15 is below row 1.

<figure class="fig"><svg viewBox="0 0 760 262" role="img" aria-label="The two places where two-dimensional structure enters ViT, highlighted: cutting the picture into patches at the start, and the two-dimensional interpolation of position embeddings when fine-tuning at a new resolution; the embedding and the encoder in between carry no built-in 2D knowledge."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Where the 2D structure of the picture enters ViT: only at two points</text><rect class="box-on" x="20.0" y="50.0" width="140.0" height="44.0" rx="10"/><text class="t-tick" x="90.0" y="77.0" text-anchor="middle">cut into 16 × 16 patches</text><text class="t-tick" x="26.0" y="120.0" text-anchor="start">2D used here:</text><text class="t-tick" x="26.0" y="135.0" text-anchor="start">neighbouring pixels</text><text class="t-tick" x="26.0" y="150.0" text-anchor="start">stay in one patch</text><line class="edge" x1="162.0" y1="72.0" x2="176.0" y2="72.0" marker-end="url(#ah)"/><rect class="box" x="176.0" y="50.0" width="140.0" height="44.0" rx="10"/><text class="t-tick" x="246.0" y="77.0" text-anchor="middle">E + learned positions</text><text class="t-tick" x="182.0" y="120.0" text-anchor="start">no 2D at the start:</text><text class="t-tick" x="182.0" y="135.0" text-anchor="start">E_pos rows are random</text><text class="t-tick" x="182.0" y="150.0" text-anchor="start">until trained</text><line class="edge" x1="318.0" y1="72.0" x2="332.0" y2="72.0" marker-end="url(#ah)"/><rect class="box" x="332.0" y="50.0" width="200.0" height="44.0" rx="10"/><text class="t-tick" x="432.0" y="68.0" text-anchor="middle">encoder:</text><text class="t-tick" x="432.0" y="84.0" text-anchor="middle">global MSA, per-token MLP</text><text class="t-tick" x="338.0" y="120.0" text-anchor="start">no 2D: every patch sees</text><text class="t-tick" x="338.0" y="135.0" text-anchor="start">every patch; the MLP</text><text class="t-tick" x="338.0" y="150.0" text-anchor="start">treats each token alike</text><line class="edge" x1="534.0" y1="72.0" x2="548.0" y2="72.0" marker-end="url(#ah)"/><rect class="box-on" x="548.0" y="50.0" width="196.0" height="44.0" rx="10"/><text class="t-tick" x="646.0" y="68.0" text-anchor="middle">fine-tune at a new size:</text><text class="t-tick" x="646.0" y="84.0" text-anchor="middle">2D interpolation of E_pos</text><text class="t-tick" x="554.0" y="120.0" text-anchor="start">2D used here:</text><text class="t-tick" x="554.0" y="135.0" text-anchor="start">position rows laid on</text><text class="t-tick" x="554.0" y="150.0" text-anchor="start">the grid and resized</text><text class="t-muted" x="20.0" y="190.0" text-anchor="start">CNN: locality, 2D neighbourhoods and translation equivariance are built into every layer.</text><text class="t-muted" x="20.0" y="207.0" text-anchor="start">ViT: the MLP is local (one token at a time) and translation equivariant (the same weights for every token);</text><text class="t-muted" x="20.0" y="224.0" text-anchor="start">self-attention is global. Which patch is next to which has to be learned from data, which is why ViT</text><text class="t-muted" x="20.0" y="241.0" text-anchor="start">needs so much of it (Part 4).</text></svg><figcaption>The two places where the 2D structure of the picture enters ViT, highlighted: cutting into 16×16 patches at the start, and the 2D interpolation of position embeddings when fine-tuning at a new resolution (Part 3). The embedding and the encoder in between carry no built-in knowledge of the grid; the position rows start random and the global attention treats all patches alike.</figcaption></figure>

## Hybrid architecture {§3.1}

> [!PAPER] · §3.1, Hybrid Architecture
> [![The Hybrid Architecture paragraph: as an alternative to raw image patches, the input sequence can be formed from feature maps of a CNN (LeCun et al., 1989). In this hybrid model, the patch embedding projection E (Eq. 1) is applied to patches extracted from a CNN feature map. As a special case, the patches can have spatial size 1x1, which means that the input sequence is obtained by simply flattening the spatial dimensions of the feature map and projecting to the Transformer dimension. The classification input embedding and position embeddings are added as described above](https://ishwarj.com/img/papers/vit/p2-hybrid.pngs/vit/p2-hybrid.png)](/img/papers/vit/p2-hybrid.png)
>
> **What it says:** "the input sequence can be formed from feature maps of a CNN". The projection $$\mathbf{E}$$ "is applied to patches extracted from a CNN feature map". In the special case of "spatial size 1x1" patches, the sequence is "obtained by simply flattening the spatial dimensions of the feature map and projecting to the Transformer dimension". The class embedding and position embeddings "are added as described above".
>
> **What it means:**
>
> This paragraph defines the third model family of the paper, next to the ResNet and the pure ViT. In the hybrid, a CNN does the first steps of seeing, and its output grid becomes the tokens.
>
> It is the "CNN + attention" family of Section 2, built inside ViT's own design so that the three families can be compared fairly. Part 4 shows the hybrid winning at small compute budgets. With more compute, the pure ViT catches up and passes it.

> [!DEFINITION] CNN stage
> A group of convolutional layers that work at one resolution; between stages the grid is halved. A ResNet-50 has four stages after its first layer, with feature maps of 56×56, 28×28, 14×14 and 7×7 for a 224×224 input. The hybrids in the paper take either the 7×7 output of stage 4 or the 14×14 output of a stage 3 that is extended to replace stage 4 (Section 4.1, Part 3).

> [!DEFINITION] Hybrid
> A ViT whose tokens are positions of a CNN feature map instead of raw pixel patches. The CNN does some of the seeing; the Transformer does the rest. The paper writes it "R50+ViT-B/16".

In shapes: a ResNet turns the $$224 \times 224 \times 3$$ picture into a $$14 \times 14 \times 1{,}024$$ feature map (with the extended-stage-3 option the paper pairs with ViT-B/16). Reading each position as a $$1 \times 1$$ "patch" gives $$N = 196$$ tokens of $$1{,}024$$ numbers, so $$\mathbf{E}$$ becomes $$1{,}024 \times 768$$ and everything from Equation 1 on is unchanged. The sequence length is the same 196 as ViT-B/16, which is why the paper pairs them.

<figure class="fig"><svg viewBox="0 0 760 258" role="img" aria-label="The hybrid architecture: the picture goes through ResNet stages, the resulting 14 by 14 by 1024 feature map is read as 196 one by one patches of 1024 numbers, projected to 768 by E, and then handled exactly like the plain ViT sequence with a class token and position embeddings."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Hybrid: the patches come from a CNN feature map instead of raw pixels</text><rect class="s2" x="24.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.35"/><rect class="s2" x="36.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.50"/><rect class="s2" x="48.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="60.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.45"/><rect class="s2" x="72.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.60"/><rect class="s2" x="84.0" y="60.0" width="11" height="11" rx="1.5" style="fill-opacity:0.40"/><rect class="s2" x="24.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.60"/><rect class="s2" x="36.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.40"/><rect class="s2" x="48.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.55"/><rect class="s2" x="60.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.35"/><rect class="s2" x="72.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.50"/><rect class="s2" x="84.0" y="72.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="24.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.50"/><rect class="s2" x="36.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="48.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.45"/><rect class="s2" x="60.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.60"/><rect class="s2" x="72.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.40"/><rect class="s2" x="84.0" y="84.0" width="11" height="11" rx="1.5" style="fill-opacity:0.55"/><rect class="s2" x="24.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.40"/><rect class="s2" x="36.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.55"/><rect class="s2" x="48.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.35"/><rect class="s2" x="60.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.50"/><rect class="s2" x="72.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="84.0" y="96.0" width="11" height="11" rx="1.5" style="fill-opacity:0.45"/><rect class="s2" x="24.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="36.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.45"/><rect class="s2" x="48.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.60"/><rect class="s2" x="60.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.40"/><rect class="s2" x="72.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.55"/><rect class="s2" x="84.0" y="108.0" width="11" height="11" rx="1.5" style="fill-opacity:0.35"/><rect class="s2" x="24.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.55"/><rect class="s2" x="36.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.35"/><rect class="s2" x="48.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.50"/><rect class="s2" x="60.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.65"/><rect class="s2" x="72.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.45"/><rect class="s2" x="84.0" y="120.0" width="11" height="11" rx="1.5" style="fill-opacity:0.60"/><text class="t-tick" x="60.0" y="148.0" text-anchor="middle">picture</text><text class="t-tick" x="60.0" y="164.0" text-anchor="middle">224 × 224 × 3</text><line class="edge" x1="100.0" y1="96.0" x2="124.0" y2="96.0" marker-end="url(#ah)"/><rect class="box-2" x="126.0" y="72.0" width="110.0" height="48.0" rx="8"/><text class="t-tick" x="181.0" y="92.0" text-anchor="middle">ResNet stages</text><text class="t-tick" x="181.0" y="108.0" text-anchor="middle">(convolutions)</text><line class="edge" x1="238.0" y1="96.0" x2="262.0" y2="96.0" marker-end="url(#ah)"/><rect class="box-ghost" x="276.0" y="62.0" width="92.0" height="92.0" rx="3"/><rect class="box-ghost" x="270.0" y="56.0" width="92.0" height="92.0" rx="3"/><rect class="box-ghost" x="264.0" y="50.0" width="92.0" height="92.0" rx="3"/><rect class="s3" x="264.0" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="270.6" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="277.2" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="283.8" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="290.4" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="297.0" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="303.6" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="310.2" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="316.8" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="323.4" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="330.0" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="336.6" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="343.2" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="349.8" y="50.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="264.0" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="270.6" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="277.2" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="283.8" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="290.4" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="297.0" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="303.6" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="310.2" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="316.8" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="323.4" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="330.0" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="336.6" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="343.2" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="349.8" y="56.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="264.0" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="270.6" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="277.2" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="283.8" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="290.4" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="297.0" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="303.6" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="310.2" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="316.8" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="323.4" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="330.0" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="336.6" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="343.2" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="349.8" y="63.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="264.0" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="270.6" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="277.2" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="283.8" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="290.4" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="297.0" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="303.6" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="310.2" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="316.8" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="323.4" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="330.0" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="336.6" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="343.2" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="349.8" y="69.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="264.0" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="270.6" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="277.2" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="283.8" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="290.4" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="297.0" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="303.6" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="310.2" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="316.8" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="323.4" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="330.0" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="336.6" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="343.2" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="349.8" y="76.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="264.0" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="270.6" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="277.2" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="283.8" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="290.4" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="297.0" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="303.6" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="310.2" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="316.8" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="323.4" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="330.0" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="336.6" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="343.2" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="349.8" y="83.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="264.0" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="270.6" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="277.2" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="283.8" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="290.4" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="297.0" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="303.6" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="310.2" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="316.8" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="323.4" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="330.0" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="336.6" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="343.2" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="349.8" y="89.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="264.0" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="270.6" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="277.2" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="283.8" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="290.4" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="297.0" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="303.6" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="310.2" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="316.8" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="323.4" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="330.0" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="336.6" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="343.2" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="349.8" y="96.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="264.0" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="270.6" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="277.2" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="283.8" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="290.4" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="297.0" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="303.6" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="310.2" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="316.8" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="323.4" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="330.0" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="336.6" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="343.2" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="349.8" y="102.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="264.0" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="270.6" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="277.2" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="283.8" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="290.4" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="297.0" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="303.6" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="310.2" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="316.8" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="323.4" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="330.0" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="336.6" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="343.2" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="349.8" y="109.4" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="264.0" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="270.6" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="277.2" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="283.8" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="290.4" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="297.0" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="303.6" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="310.2" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="316.8" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="323.4" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="330.0" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="336.6" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="343.2" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="349.8" y="116.0" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="264.0" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="270.6" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="277.2" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="283.8" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="290.4" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="297.0" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="303.6" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="310.2" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="316.8" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="323.4" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="330.0" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="336.6" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="343.2" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="349.8" y="122.6" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="264.0" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="270.6" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="277.2" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="283.8" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="290.4" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="297.0" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="303.6" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="310.2" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="316.8" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="323.4" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="330.0" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="336.6" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="343.2" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="349.8" y="129.2" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="264.0" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="270.6" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="277.2" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="283.8" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="290.4" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="297.0" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="303.6" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><rect class="s3" x="310.2" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.45"/><rect class="s3" x="316.8" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.60"/><rect class="s3" x="323.4" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.40"/><rect class="s3" x="330.0" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.55"/><rect class="s3" x="336.6" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.35"/><rect class="s3" x="343.2" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.50"/><rect class="s3" x="349.8" y="135.8" width="5.6" height="5.6" rx="1.5" style="fill-opacity:0.65"/><text class="t-tick" x="310.0" y="162.0" text-anchor="middle">feature map</text><text class="t-tick" x="310.0" y="178.0" text-anchor="middle">14 × 14 × 1024</text><line class="edge" x1="360.0" y1="96.0" x2="384.0" y2="96.0" marker-end="url(#ah)"/><rect class="box" x="386.0" y="60.0" width="120.0" height="72.0" rx="8"/><text class="t-tick" x="446.0" y="82.0" text-anchor="middle">1 × 1 "patches":</text><text class="t-tick" x="446.0" y="98.0" text-anchor="middle">196 vectors</text><text class="t-tick" x="446.0" y="114.0" text-anchor="middle">of 1024 numbers</text><line class="edge" x1="508.0" y1="96.0" x2="532.0" y2="96.0" marker-end="url(#ah)"/><rect class="box-4" x="534.0" y="74.0" width="56.0" height="44.0" rx="6"/><text class="t-math" x="562.0" y="92.0" text-anchor="middle">E</text><text class="t-cell" x="562.0" y="110.0" text-anchor="middle">1024 → 768</text><line class="edge" x1="592.0" y1="96.0" x2="616.0" y2="96.0" marker-end="url(#ah)"/><rect class="box-1" x="618.0" y="60.0" width="130.0" height="72.0" rx="8"/><text class="t-tick" x="683.0" y="82.0" text-anchor="middle">[class] + 196 tokens</text><text class="t-tick" x="683.0" y="98.0" text-anchor="middle">+ positions, then</text><text class="t-tick" x="683.0" y="114.0" text-anchor="middle">the same encoder</text><text class="t-muted" x="20.0" y="206.0" text-anchor="start">The paper (Section 4.1) takes the 7 × 7 output of stage 4 of a ResNet50, or the 14 × 14 output of a</text><text class="t-muted" x="20.0" y="223.0" text-anchor="start">stage 3 extended to replace stage 4, and feeds every position of the map as one token.</text><text class="t-muted" x="20.0" y="240.0" text-anchor="start">Everything after E is unchanged. The 14 × 14 map gives 196 tokens, as in ViT-B/16.</text></svg><figcaption>The hybrid architecture. The picture goes through ResNet stages; the 14×14×1024 feature map is read as 196 one-by-one patches of 1,024 numbers; E (now 1,024 × 768) projects them; the class token and position embeddings are added; and the same encoder follows. Everything after E is identical to the plain ViT.</figcaption></figure>

## Table 1, as far as this part needs it {§4.1}

The next part reads the experimental setup in full. We need one table from it now, because every number above (768, 12, 3,072) came from it:

> [!PAPER] · Table 1
> [![Table 1, details of Vision Transformer model variants: ViT-Base has 12 layers, hidden size D 768, MLP size 3072, 12 heads, 86M parameters; ViT-Large has 24 layers, 1024, 4096, 16 heads, 307M; ViT-Huge has 32 layers, 1280, 5120, 16 heads, 632M](https://ishwarj.com/img/papers/vit/p2-table1.pngs/vit/p2-table1.png)](/img/papers/vit/p2-table1.png)
>
> **What it says:** **ViT-Base**: 12 layers, $$D = 768$$, MLP 3,072, 12 heads, 86M parameters. **ViT-Large**: 24 layers, $$D = 1{,}024$$, MLP 4,096, 16 heads, 307M. **ViT-Huge**: 32 layers, $$D = 1{,}280$$, MLP 5,120, 16 heads, 632M. The MLP size is always $$4D$$ and the head size always $$D/\text{heads} = 64$$ for Base and Large, 80 for Huge.
>
> **What it means:**
>
> Table 1 belongs to Section 4.1, "Model Variants". The paper says the Base and Large sizes "are directly adopted from BERT", so ViT-B has the same shape as BERT-base.
>
> Each row is a recipe: how many layers, how wide, how many heads. With these numbers and the equations above, we can count every parameter of every model, which we do next. [Part 3](https://ishwarj.com/papers/vit/part-3-fine-tuning-and-setup/) reads the table together with the patch-size notation and the baselines.

<figure class="fig"><svg viewBox="0 0 760 408" role="img" aria-label="Table 1 of the paper as three stacks of blocks: ViT-Base with 12 layers of width 768, ViT-Large with 24 layers of width 1024 and ViT-Huge with 32 layers of width 1280, with the paper parameter counts of 86M, 307M and 632M and our counts without a head."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Table 1 drawn: one block per layer, block width ∝ hidden size D</text><rect class="s1" x="102.0" y="216.8" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="225.4" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="234.0" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="242.6" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="251.2" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="259.8" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="268.4" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="277.0" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="285.6" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="294.2" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="302.8" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="102.0" y="311.4" width="96.0" height="7" rx="1.5" style="fill-opacity:0.55"/><text class="t-note" x="150.0" y="206.8" text-anchor="middle">ViT-Base</text><text class="t-tick" x="150.0" y="338.0" text-anchor="middle">L = 12 layers, D = 768</text><text class="t-tick" x="150.0" y="354.0" text-anchor="middle">MLP 3072, 12 heads</text><text class="t-tick" x="150.0" y="370.0" text-anchor="middle">paper: 86M; ours: 85.8M</text><rect class="s1" x="316.0" y="113.6" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="122.2" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="130.8" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="139.4" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="148.0" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="156.6" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="165.2" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="173.8" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="182.4" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="191.0" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="199.6" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="208.2" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="216.8" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="225.4" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="234.0" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="242.6" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="251.2" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="259.8" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="268.4" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="277.0" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="285.6" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="294.2" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="302.8" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="316.0" y="311.4" width="128.0" height="7" rx="1.5" style="fill-opacity:0.55"/><text class="t-note" x="380.0" y="103.6" text-anchor="middle">ViT-Large</text><text class="t-tick" x="380.0" y="338.0" text-anchor="middle">L = 24 layers, D = 1024</text><text class="t-tick" x="380.0" y="354.0" text-anchor="middle">MLP 4096, 16 heads</text><text class="t-tick" x="380.0" y="370.0" text-anchor="middle">paper: 307M; ours: 303.3M</text><rect class="s1" x="530.0" y="44.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="53.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="62.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="70.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="79.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="87.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="96.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="105.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="113.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="122.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="130.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="139.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="148.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="156.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="165.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="173.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="182.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="191.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="199.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="208.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="216.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="225.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="234.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="242.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="251.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="259.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="268.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="277.0" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="285.6" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="294.2" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="302.8" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><rect class="s1" x="530.0" y="311.4" width="160.0" height="7" rx="1.5" style="fill-opacity:0.55"/><text class="t-note" x="610.0" y="34.8" text-anchor="middle">ViT-Huge</text><text class="t-tick" x="610.0" y="338.0" text-anchor="middle">L = 32 layers, D = 1280</text><text class="t-tick" x="610.0" y="354.0" text-anchor="middle">MLP 5120, 16 heads</text><text class="t-tick" x="610.0" y="370.0" text-anchor="middle">paper: 632M; ours: 630.8M</text><text class="t-muted" x="20.0" y="396.0" text-anchor="start">"ours" counts the model without any classification head (the formula of this part); Part 3 reads Table 1 properly.</text></svg><figcaption>Table 1 as three stacks of blocks, one block per layer, block width proportional to D. ViT-Base: 12 layers of 768. ViT-Large: 24 of 1,024. ViT-Huge: 32 of 1,280. Under each, the paper's parameter count and ours without a classification head.</figcaption></figure>

### Where the 86 million parameters come from

> [!DEFINITION] Parameter count
> The number of learned values in a model: every entry of every weight matrix, bias vector, embedding table and LayerNorm scale. "86M" means about 86 million such numbers.

Every piece of the model above has a known shape, so the count is a formula. For hidden size $$D$$, $$L$$ layers, MLP size $$4D$$, patch size $$P$$, $$C$$ channels, $$N$$ patches and $$K$$ classes:

$$
\underbrace{P^2 C \cdot D + D}_{\text{patch embedding } \mathbf{E} \text{ + bias}} + \underbrace{D}_{\mathbf{x}_{\text{class}}} + \underbrace{(N+1)\,D}_{\mathbf{E}_{pos}} + L\,\big(\underbrace{4(D^2 + D)}_{\text{q, k, v, } U_{msa}} + \underbrace{8D^2 + 5D}_{\text{MLP}} + \underbrace{4D}_{\text{2 LayerNorms}}\big) + \underbrace{2D}_{\text{final LN}} + \underbrace{D K + K}_{\text{head}}
$$

The per-layer part simplifies to $$12D^2 + 13D$$. Each term, for ViT-B/16 at 224 with $$K = 1{,}000$$:

- patch embedding: $$16 \cdot 16 \cdot 3 \cdot 768 + 768 = 590{,}592$$;
- class token: $$768$$; positions: $$197 \times 768 = 151{,}296$$;
- attention per layer: $$4 \times (768^2 + 768) = 2{,}362{,}368$$ (four $$768 \times 768$$ matrices with biases);
- MLP per layer: $$768 \cdot 3{,}072 + 3{,}072 + 3{,}072 \cdot 768 + 768 = 4{,}722{,}432$$;
- two LayerNorms per layer: $$4 \times 768 = 3{,}072$$; so one layer is $$7{,}087{,}872$$ and twelve are $$85{,}054{,}464$$;
- final LayerNorm: $$1{,}536$$; head: $$768 \times 1{,}000 + 1{,}000 = 769{,}000$$.

The script evaluates the formula and counts the real model's parameters. For ViT-L/16 and ViT-H/14 it builds the model from its configuration on PyTorch's "meta" device, which creates every tensor's shape without allocating any memory, so no weights are needed:

```python
def formula(D, L, P, C, N, K, mlp=None):
    mlp = mlp or 4 * D
    return (P * P * C * D + D) + D + (N + 1) * D + L * (4 * (D * D + D) + D * mlp + mlp + mlp * D + D + 4 * D) + 2 * D + (D * K + K)

c = ViTConfig(hidden_size=1280, num_hidden_layers=32, num_attention_heads=16, intermediate_size=5120, patch_size=14, num_labels=1000)
with torch.device('meta'):                       # shapes only, no memory for weights
    huge = ViTForImageClassification(c)
print(sum(p.numel() for p in huge.parameters()))
```

```text
== 7. counting the parameters ==
  formula: (P^2 C D + D) + D + (N+1) D + L (12 D^2 + 13 D) + 2 D + (D K + K)
           patch emb       cls  position  L layers            LN   head
  ViT-B/16 at 224, K = 1000:
    patch embedding      590,592   [class]    768   positions   151,296
    one layer: attention 4(D^2 + D) = 2,362,368   MLP 8D^2 + 5D = 4,722,432   2 LayerNorms 4D = 3,072   -> 7,087,872   x 12 = 85,054,464
    final LayerNorm 1,536   head D K + K = 769,000
    formula total 86,567,656   counted in model.parameters() 86,567,656   same: True
    without the 1000-class head: 85,798,656   (paper, Table 1: 86M)
    where they sit: attention 28.35M (32.7%)   mlp 56.67M (65.5%)   embeddings 0.74M (0.9%)   layernorms 0.04M (0.0%)   head 0.77M (0.9%)
  ViT-L/16 at 224 (meta device, no weights): D=1024 L=24 heads=16 MLP=4096 N=196
    formula 304,326,632   counted 304,326,632   same: True
    with the 1000-class head 304.3M, without it 303.3M   (paper: 307M)
  ViT-H/14 at 224 (meta device, no weights): D=1280 L=32 heads=16 MLP=5120 N=256
    formula 632,045,800   counted 632,045,800   same: True
    with the 1000-class head 632.0M, without it 630.8M   (paper: 632M)
```

The formula matches the counted parameters exactly for all three sizes: **86,567,656** for ViT-B/16 with its 1,000-class head, **304,326,632** for ViT-L/16 and **632,045,800** for ViT-H/14. Compared with Table 1: ViT-Base rounds to 86M with or without the head (85.8M or 86.6M). ViT-Huge rounds to 632M *with* a 1,000-class head (632.0M; without it, 630.8M), so Table 1's figure for Huge appears to include a head of that size. ViT-Large is the odd one: we get 303.3M or 304.3M, and the paper says 307M. An honesty note: the 3M gap is real and we cannot explain it from the model's configuration (ViT-L/16 at 384 pixels has 577 position rows instead of 197, which adds only 0.4M; the 21,843-class ImageNet-21k head would add 22M, far too much). The released ViT-L/16 checkpoints have 304M parameters, so we trust the count and flag the table.

<figure class="fig"><svg viewBox="0 0 760 240" role="img" aria-label="Horizontal bars of the parameter count of ViT-B/16: the MLP blocks hold 56.67 million (65.5 percent), attention 28.35 million (32.7 percent), the embeddings 0.74 million, the head 0.77 million and the LayerNorms 0.04 million."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="40.0" text-anchor="start">Where the 86,567,656 parameters of ViT-B/16 (1000-class head) sit</text><text class="t-tick" x="232.0" y="71.0" text-anchor="end">attention (12 layers)</text><g class="mark"><title>attention (12 layers): 28.35 M  (32.7%)</title><rect class="s1" x="240.0" y="53.0" width="179.5" height="26.0" rx="3"/></g><text class="t-val" x="425.5" y="71.0" text-anchor="start">28.35 M  (32.7%)</text><text class="t-tick" x="232.0" y="105.0" text-anchor="end">MLP (12 layers)</text><g class="mark"><title>MLP (12 layers): 56.67 M  (65.5%)</title><rect class="s1" x="240.0" y="87.0" width="358.9" height="26.0" rx="3"/></g><text class="t-val" x="604.9" y="105.0" text-anchor="start">56.67 M  (65.5%)</text><text class="t-tick" x="232.0" y="139.0" text-anchor="end">embeddings (E, [class], E_pos)</text><g class="mark"><title>embeddings (E, [class], E_pos): 0.74 M  (0.9%)</title><rect class="s1" x="240.0" y="121.0" width="4.7" height="26.0" rx="3"/></g><text class="t-val" x="250.7" y="139.0" text-anchor="start">0.74 M  (0.9%)</text><text class="t-tick" x="232.0" y="173.0" text-anchor="end">head (768 × 1000 + 1000)</text><g class="mark"><title>head (768 × 1000 + 1000): 0.77 M  (0.9%)</title><rect class="s1" x="240.0" y="155.0" width="4.9" height="26.0" rx="3"/></g><text class="t-val" x="250.9" y="173.0" text-anchor="start">0.77 M  (0.9%)</text><text class="t-tick" x="232.0" y="207.0" text-anchor="end">LayerNorms (25 of them)</text><g class="mark"><title>LayerNorms (25 of them): 0.04 M  (0.0%)</title><rect class="s1" x="240.0" y="189.0" width="2.0" height="26.0" rx="3"/></g><text class="t-val" x="248.0" y="207.0" text-anchor="start">0.04 M  (0.0%)</text></svg><figcaption>Where the 86,567,656 parameters of ViT-B/16 sit. The MLP blocks hold 56.67 million (65.5%), attention 28.35 million (32.7%), the embeddings (E, the class token and the 197 position rows) only 0.74 million, the head 0.77 million and all 25 LayerNorms 0.04 million.</figcaption></figure>

Two things stand out. The embeddings are tiny: 0.9% of the model, against about a fifth for BERT-base, because BERT carries a 30,522-row vocabulary table and ViT carries one $$768 \times 768$$ matrix. And two thirds of ViT is MLP, one third attention; the ratio $$8D^2 : 4D^2$$ is fixed by the $$4D$$ MLP width and is the same for every Transformer with that width.

## Appendix A: the attention equations {§A}

The paper keeps the attention formulas out of the main text and puts them in Appendix A. They are the inside of the "MSA" box above.

> [!PAPER] · Appendix A
> [![Appendix A, Multihead Self-Attention: standard qkv self-attention is a popular building block. For each element in an input sequence z in R N by D, we compute a weighted sum over all values v in the sequence. The attention weights A i j are based on the pairwise similarity between two elements of the sequence and their respective query q i and key k j representations. Equation 5: q, k, v equals z U qkv, with U qkv in R D by 3 D h. Equation 6: A equals softmax of q k transpose over square root of D h, with A in R N by N. Equation 7: SA of z equals A v](https://ishwarj.com/img/papers/vit/p2-appendix-a.pngt/p2-appendix-a.png)](/img/papers/vit/p2-appendix-a.png)
>
> **What it says:** for each element of the input $$\mathbf{z} \in \mathbb{R}^{N \times D}$$, "we compute a weighted sum over all values $$\mathbf{v}$$ in the sequence", with weights $$A_{ij}$$ "based on the pairwise similarity between two elements of the sequence and their respective query $$\mathbf{q}^i$$ and key $$\mathbf{k}^j$$ representations". Then Equations 5, 6 and 7.
>
> **What it means:**
>
> Think of a library. Each token writes a question (its query), puts a label on its own shelf (its key), and keeps a book there (its value). A token compares its question with every label, and takes most from the books whose labels match best.
>
> Equations 5 to 7 are those steps, for one head. They define the one operation in the whole model that lets tokens interact. Everything else in ViT works on one token at a time.

> [!DEFINITION] Query, key and value
> Three vectors computed from each token by three linear projections. A token's **query** is what it is looking for; a token's **key** is what it offers to be matched against; its **value** is what it hands over when it is attended to. The weight from token $$i$$ to token $$j$$ is the similarity (dot product) of $$i$$'s query with $$j$$'s key.

> [!DEFINITION] Head
> One complete self-attention computation with its own $$U_{qkv}$$, working on a slice of $$D_h$$ numbers (64 in ViT-B). A layer runs $$k$$ heads in parallel so that different heads can learn different ways of relating tokens.

### Equations 5 to 7: one head

The idea: from the same input compute queries, keys and values; score every pair; turn scores into weights; average the values with those weights.

$$
[\mathbf{q}, \mathbf{k}, \mathbf{v}] = \mathbf{z}\,\mathbf{U}_{qkv}, \qquad \mathbf{U}_{qkv} \in \mathbb{R}^{D \times 3D_h}
$$

$$
A = \text{softmax}\!\left(\mathbf{q}\mathbf{k}^\top / \sqrt{D_h}\right), \qquad A \in \mathbb{R}^{N \times N}
$$

$$
\text{SA}(\mathbf{z}) = A\,\mathbf{v}
$$

where:

- $$\mathbf{z}$$ is the (normalised) input, $$N \times D$$: 197 × 768 in ViT-B;
- $$\mathbf{U}_{qkv}$$ is one matrix of $$D \times 3D_h$$ that produces all three at once; $$\mathbf{q}, \mathbf{k}, \mathbf{v}$$ are each $$N \times D_h$$ (197 × 64). The released model stores it as three separate $$768 \times 768$$ matrices covering all 12 heads, which is the same numbers arranged differently, plus biases the equation omits;
- $$\mathbf{q}\mathbf{k}^\top$$ is $$N \times N$$: entry $$(i, j)$$ is the dot product of query $$i$$ with key $$j$$;
- $$\sqrt{D_h}$$ is 8 for $$D_h = 64$$; dividing keeps the scores from growing with the head size, so softmax does not saturate ([BERT Part 2](https://ishwarj.com/papers/bert/part-2-architecture-and-input/) measures why);
- softmax is applied to each row, so row $$i$$ holds the 197 weights of token $$i$$ and they add up to 1;
- $$A\,\mathbf{v}$$ is $$N \times D_h$$: row $$i$$ is the weighted average of all value rows, with token $$i$$'s weights.

This is Vaswani et al.'s Equation 1 with different letters:

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · §3.2.1, Equation (1)
> [![Section 3.2.1 of Attention Is All You Need: in practice we compute the attention function on a set of queries simultaneously, packed together into a matrix Q; the keys and values are also packed into matrices K and V; Attention of Q, K, V equals softmax of Q K transpose over square root of d k, times V](https://ishwarj.com/img/papers/vit/p2-ext-vaswani-eq1.pngext-vaswani-eq1.png)](/img/papers/vit/p2-ext-vaswani-eq1.png)
>
> **What it says:** queries are "packed together into a matrix $$Q$$", keys and values into $$K$$ and $$V$$, and $$\text{Attention}(Q, K, V) = \text{softmax}(QK^T/\sqrt{d_k})\,V$$.
>
> **What it means:**
>
> This is scaled dot-product attention as the original Transformer paper writes it. Only the letters differ from ViT's Appendix A.
>
> ViT's $$\mathbf{q}, \mathbf{k}, \mathbf{v}, D_h$$ are $$Q, K, V, d_k$$. The paper changed nothing here, as Section 3 promised when it said it follows the original Transformer "as closely as possible".

**A worked example small enough to check with a pencil.** Four tokens, $$D = 6$$, two heads, so $$D_h = 3$$, with small whole numbers chosen so the sums stay short ([`vit_part2_attn.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part2_attn.py)). Head 1:

```text
== Eq. 5 to 8 on a tiny example: N = 4 tokens, D = 6, k = 2 heads, D_h = D/k = 3 ==
  z (the input, one row per token)  shape (4, 6)
       t0      0      1      0      0      1      0
       t1      0      1      1      1      2      1
       t2      1      1      1      0      0      0
       t3      2      2      2      1      2      1
-- head 1 --
  U_qkv (head 1): D x 3 D_h  shape (6, 9)
               0      1      0      0      0      0     -1      1      0
               1     -1      1      0      1      0      0      0      1
               0      1      0     -1      1      1     -1      1      0
               1      1     -1     -1      0     -1     -1     -1      0
               1      0      1      1     -1      0      0     -1     -1
               0     -1     -1      1      0     -1      0      0     -1
  [q, k, v] = z U_qkv  (the first 3 columns are q, the next 3 k, the last 3 v)  shape (4, 9)
       t0      2     -1      2      1      0      0      0     -1      0
       t1      4      0      1      1      0     -1     -2     -2     -2
       t2      1      1      1     -1      2      1     -2      2      1
       t3      5      2      2      0      2      0     -5      1     -1
  q k^T / sqrt(D_h) = q k^T / 1.732  shape (4, 4)   (row i = query of token i, column j = key of token j)
       t0  1.155  0.000 -1.155 -1.155
       t1  2.309  1.732 -1.732  0.000
       t2  0.577  0.000  1.155  1.155
       t3  2.887  1.732  0.577  2.309
  A = softmax(row by row)  shape (4, 4)
       t0  0.661  0.208  0.066  0.066
       t1  0.596  0.335  0.010  0.059
       t2  0.195  0.110  0.348  0.348
       t3  0.506  0.160  0.050  0.284
    row sums: [1.0, 1.0, 1.0, 1.0]
  SA(z) = A v  (row i = weighted average of the value rows, weights = row i of A)  shape (4, 3)
       t0 -0.876 -0.880 -0.416
       t1 -0.986 -1.185 -0.718
       t2 -2.653  0.629 -0.219
       t3 -3.014  0.002 -0.995
```

Follow one number through. Row t0 of $$\mathbf{z}$$ is $$(0, 1, 0, 0, 1, 0)$$, so its query is the sum of rows 2 and 5 of $$\mathbf{U}_{qkv}$$'s first three columns: $$(1, -1, 1) + (1, 0, 1) = (2, -1, 2)$$, as printed. Its score against its own key $$(1, 0, 0)$$ is $$2 \cdot 1 + (-1) \cdot 0 + 2 \cdot 0 = 2$$; divided by $$\sqrt{3} = 1.732$$ that is 1.155. Softmax over the row $$(1.155, 0, -1.155, -1.155)$$:

$$
\frac{e^{1.155}}{e^{1.155} + e^{0} + e^{-1.155} + e^{-1.155}} = \frac{3.173}{3.173 + 1.000 + 0.315 + 0.315} = \frac{3.173}{4.803} = 0.661
$$

which the script checks the same way (`A[t0, t0] = 3.173 / 4.803 = 0.661`). And the first number of t0's output is $$0.661 \cdot 0 + 0.208 \cdot (-2) + 0.066 \cdot (-2) + 0.066 \cdot (-5) = -0.876$$: the weighted average of the first column of $$\mathbf{v}$$. Token t0 listens mostly to itself (0.661) and to t1 (0.208).

<figure class="fig"><svg viewBox="0 0 760 392" role="img" aria-label="The tiny worked example of Appendix A for head 1: the four by four scaled scores, the softmax weights whose rows sum to one, the four value rows of three numbers, and the output SA of z as the weighted sum of the value rows."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">Head 1 of the tiny example: scaled scores, softmax weights and SA(z) = A v</text><text class="t-title" x="20.0" y="56.0" text-anchor="start">q kᵀ / √3</text><text class="t-tick" x="72.0" y="76.0" text-anchor="middle">t0</text><text class="t-tick" x="116.0" y="76.0" text-anchor="middle">t1</text><text class="t-tick" x="160.0" y="76.0" text-anchor="middle">t2</text><text class="t-tick" x="204.0" y="76.0" text-anchor="middle">t3</text><text class="t-tick" x="42.0" y="102.0" text-anchor="end">t0</text><rect class="cell" x="51.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="72.0" y="102.0" text-anchor="middle">1.15</text><rect class="cell" x="95.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="116.0" y="102.0" text-anchor="middle">0.00</text><rect class="s2" x="139.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="160.0" y="102.0" text-anchor="middle">-1.15</text><rect class="s2" x="183.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="204.0" y="102.0" text-anchor="middle">-1.15</text><text class="t-tick" x="42.0" y="126.0" text-anchor="end">t1</text><rect class="cell" x="51.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.56"/><text class="t-cell" x="72.0" y="126.0" text-anchor="middle">2.31</text><rect class="cell" x="95.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.44"/><text class="t-cell" x="116.0" y="126.0" text-anchor="middle">1.73</text><rect class="s2" x="139.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.44"/><text class="t-cell" x="160.0" y="126.0" text-anchor="middle">-1.73</text><rect class="cell" x="183.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="204.0" y="126.0" text-anchor="middle">0.00</text><text class="t-tick" x="42.0" y="150.0" text-anchor="end">t2</text><rect class="cell" x="51.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="72.0" y="150.0" text-anchor="middle">0.58</text><rect class="cell" x="95.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="116.0" y="150.0" text-anchor="middle">0.00</text><rect class="cell" x="139.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="160.0" y="150.0" text-anchor="middle">1.15</text><rect class="cell" x="183.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="204.0" y="150.0" text-anchor="middle">1.15</text><text class="t-tick" x="42.0" y="174.0" text-anchor="end">t3</text><rect class="cell" x="51.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.68"/><text class="t-cell" x="72.0" y="174.0" text-anchor="middle">2.89</text><rect class="cell" x="95.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.44"/><text class="t-cell" x="116.0" y="174.0" text-anchor="middle">1.73</text><rect class="cell" x="139.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="160.0" y="174.0" text-anchor="middle">0.58</text><rect class="cell" x="183.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.56"/><text class="t-cell" x="204.0" y="174.0" text-anchor="middle">2.31</text><text class="t-title" x="260.0" y="56.0" text-anchor="start">A = softmax (rows sum to 1)</text><text class="t-tick" x="312.0" y="76.0" text-anchor="middle">t0</text><text class="t-tick" x="356.0" y="76.0" text-anchor="middle">t1</text><text class="t-tick" x="400.0" y="76.0" text-anchor="middle">t2</text><text class="t-tick" x="444.0" y="76.0" text-anchor="middle">t3</text><text class="t-tick" x="282.0" y="102.0" text-anchor="end">t0</text><rect class="cell" x="291.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.68"/><text class="t-cell" x="312.0" y="102.0" text-anchor="middle">0.66</text><rect class="cell" x="335.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.27"/><text class="t-cell" x="356.0" y="102.0" text-anchor="middle">0.21</text><rect class="cell" x="379.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.14"/><text class="t-cell" x="400.0" y="102.0" text-anchor="middle">0.07</text><rect class="cell" x="423.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.14"/><text class="t-cell" x="444.0" y="102.0" text-anchor="middle">0.07</text><text class="t-tick" x="282.0" y="126.0" text-anchor="end">t1</text><rect class="cell" x="291.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.62"/><text class="t-cell" x="312.0" y="126.0" text-anchor="middle">0.60</text><rect class="cell" x="335.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.38"/><text class="t-cell" x="356.0" y="126.0" text-anchor="middle">0.33</text><rect class="cell" x="379.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.09"/><text class="t-cell" x="400.0" y="126.0" text-anchor="middle">0.01</text><rect class="cell" x="423.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.13"/><text class="t-cell" x="444.0" y="126.0" text-anchor="middle">0.06</text><text class="t-tick" x="282.0" y="150.0" text-anchor="end">t2</text><rect class="cell" x="291.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.26"/><text class="t-cell" x="312.0" y="150.0" text-anchor="middle">0.20</text><rect class="cell" x="335.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.18"/><text class="t-cell" x="356.0" y="150.0" text-anchor="middle">0.11</text><rect class="cell" x="379.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.40"/><text class="t-cell" x="400.0" y="150.0" text-anchor="middle">0.35</text><rect class="cell" x="423.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.40"/><text class="t-cell" x="444.0" y="150.0" text-anchor="middle">0.35</text><text class="t-tick" x="282.0" y="174.0" text-anchor="end">t3</text><rect class="cell" x="291.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.54"/><text class="t-cell" x="312.0" y="174.0" text-anchor="middle">0.51</text><rect class="cell" x="335.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.22"/><text class="t-cell" x="356.0" y="174.0" text-anchor="middle">0.16</text><rect class="cell" x="379.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.13"/><text class="t-cell" x="400.0" y="174.0" text-anchor="middle">0.05</text><rect class="cell" x="423.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.34"/><text class="t-cell" x="444.0" y="174.0" text-anchor="middle">0.28</text><text class="t-title" x="500.0" y="56.0" text-anchor="start">v (value rows)</text><text class="t-tick" x="552.0" y="76.0" text-anchor="middle">v₀</text><text class="t-tick" x="596.0" y="76.0" text-anchor="middle">v₁</text><text class="t-tick" x="640.0" y="76.0" text-anchor="middle">v₂</text><text class="t-tick" x="522.0" y="102.0" text-anchor="end">t0</text><rect class="cell" x="531.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="552.0" y="102.0" text-anchor="middle">0</text><rect class="s2" x="575.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="596.0" y="102.0" text-anchor="middle">-1</text><rect class="cell" x="619.0" y="87.0" width="42" height="22" rx="3" style="fill-opacity:0.08"/><text class="t-cell" x="640.0" y="102.0" text-anchor="middle">0</text><text class="t-tick" x="522.0" y="126.0" text-anchor="end">t1</text><rect class="s2" x="531.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="552.0" y="126.0" text-anchor="middle">-2</text><rect class="s2" x="575.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="596.0" y="126.0" text-anchor="middle">-2</text><rect class="s2" x="619.0" y="111.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="640.0" y="126.0" text-anchor="middle">-2</text><text class="t-tick" x="522.0" y="150.0" text-anchor="end">t2</text><rect class="s2" x="531.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="552.0" y="150.0" text-anchor="middle">-2</text><rect class="cell" x="575.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.32"/><text class="t-cell" x="596.0" y="150.0" text-anchor="middle">2</text><rect class="cell" x="619.0" y="135.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="640.0" y="150.0" text-anchor="middle">1</text><text class="t-tick" x="522.0" y="174.0" text-anchor="end">t3</text><rect class="s2" x="531.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.68"/><text class="t-cell" x="552.0" y="174.0" text-anchor="middle">-5</text><rect class="cell" x="575.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="596.0" y="174.0" text-anchor="middle">1</text><rect class="s2" x="619.0" y="159.0" width="42" height="22" rx="3" style="fill-opacity:0.20"/><text class="t-cell" x="640.0" y="174.0" text-anchor="middle">-1</text><text class="t-title" x="500.0" y="196.0" text-anchor="start">SA(z) = A v</text><text class="t-tick" x="552.0" y="216.0" text-anchor="middle"></text><text class="t-tick" x="596.0" y="216.0" text-anchor="middle"></text><text class="t-tick" x="640.0" y="216.0" text-anchor="middle"></text><text class="t-tick" x="522.0" y="242.0" text-anchor="end">t0</text><rect class="s2" x="531.0" y="227.0" width="42" height="22" rx="3" style="fill-opacity:0.28"/><text class="t-cell" x="552.0" y="242.0" text-anchor="middle">-0.88</text><rect class="s2" x="575.0" y="227.0" width="42" height="22" rx="3" style="fill-opacity:0.28"/><text class="t-cell" x="596.0" y="242.0" text-anchor="middle">-0.88</text><rect class="s2" x="619.0" y="227.0" width="42" height="22" rx="3" style="fill-opacity:0.17"/><text class="t-cell" x="640.0" y="242.0" text-anchor="middle">-0.42</text><text class="t-tick" x="522.0" y="266.0" text-anchor="end">t1</text><rect class="s2" x="531.0" y="251.0" width="42" height="22" rx="3" style="fill-opacity:0.30"/><text class="t-cell" x="552.0" y="266.0" text-anchor="middle">-0.99</text><rect class="s2" x="575.0" y="251.0" width="42" height="22" rx="3" style="fill-opacity:0.35"/><text class="t-cell" x="596.0" y="266.0" text-anchor="middle">-1.18</text><rect class="s2" x="619.0" y="251.0" width="42" height="22" rx="3" style="fill-opacity:0.24"/><text class="t-cell" x="640.0" y="266.0" text-anchor="middle">-0.72</text><text class="t-tick" x="522.0" y="290.0" text-anchor="end">t2</text><rect class="s2" x="531.0" y="275.0" width="42" height="22" rx="3" style="fill-opacity:0.68"/><text class="t-cell" x="552.0" y="290.0" text-anchor="middle">-2.65</text><rect class="cell" x="575.0" y="275.0" width="42" height="22" rx="3" style="fill-opacity:0.22"/><text class="t-cell" x="596.0" y="290.0" text-anchor="middle">0.63</text><rect class="s2" x="619.0" y="275.0" width="42" height="22" rx="3" style="fill-opacity:0.13"/><text class="t-cell" x="640.0" y="290.0" text-anchor="middle">-0.22</text><text class="t-tick" x="522.0" y="314.0" text-anchor="end">t3</text><rect class="s2" x="531.0" y="299.0" width="42" height="22" rx="3" style="fill-opacity:0.50"/><text class="t-cell" x="552.0" y="314.0" text-anchor="middle">-1.84</text><rect class="s2" x="575.0" y="299.0" width="42" height="22" rx="3" style="fill-opacity:0.18"/><text class="t-cell" x="596.0" y="314.0" text-anchor="middle">-0.44</text><rect class="s2" x="619.0" y="299.0" width="42" height="22" rx="3" style="fill-opacity:0.21"/><text class="t-cell" x="640.0" y="314.0" text-anchor="middle">-0.55</text><text class="t-tick" x="20.0" y="334.0" text-anchor="start">row t0 of SA(z) = 0.661·v(t0) + 0.208·v(t1) + 0.066·v(t2) + 0.066·v(t3)</text><text class="t-tick" x="20.0" y="352.0" text-anchor="start">its first number: 0.661·0 + 0.208·(-2) + 0.066·(-2) + 0.066·(-5) = -0.876</text><text class="t-muted" x="20.0" y="370.0" text-anchor="start">Orange cells are negative numbers; the shade shows the size.</text><line class="edge" x1="236.0" y1="130.0" x2="254.0" y2="130.0" marker-end="url(#ah)"/><line class="edge" x1="476.0" y1="130.0" x2="494.0" y2="130.0" marker-end="url(#ah)"/></svg><figcaption>Head 1 of the tiny example as a picture. Left: the 4×4 scaled scores. Middle: softmax turns each row into weights that add up to 1. Right: the four value rows, and below them SA(z), each row a weighted average of the value rows. Orange cells are negative numbers.</figcaption></figure>

### Equation 8: many heads

> [!PAPER] · Appendix A
> [![The second half of Appendix A: multihead self-attention (MSA) is an extension of SA in which we run k self-attention operations, called heads, in parallel, and project their concatenated outputs. To keep compute and number of parameters constant when changing k, D h (Eq. 5) is typically set to D over k. Equation 8: MSA of z equals the concatenation of SA 1 of z to SA k of z, times U msa, with U msa in R k D h by D](https://ishwarj.com/img/papers/vit/p2-appendix-msa.pngp2-appendix-msa.png)](/img/papers/vit/p2-appendix-msa.png)
>
> **What it says:** MSA runs "$$k$$ self-attention operations, called 'heads', in parallel", and projects "their concatenated outputs". "To keep compute and number of parameters constant when changing $$k$$, $$D_h$$ (Eq. 5) is typically set to $$D/k$$." Then Equation 8.
>
> **What it means:**
>
> One head can learn one way of relating tokens, such as "look at your neighbour". Many heads side by side can learn many ways at once, like a team in which each member watches for something different.
>
> The rule $$D_h = D/k$$ keeps the size of the team fixed. That is why ViT-B has 12 heads of 64 numbers each: $$768/12 = 64$$. More heads means smaller heads, so the head count does not change the size of the model.

$$
\text{MSA}(\mathbf{z}) = [\text{SA}_1(\mathbf{z});\ \text{SA}_2(\mathbf{z});\ \cdots;\ \text{SA}_k(\mathbf{z})]\ \mathbf{U}_{msa}, \qquad \mathbf{U}_{msa} \in \mathbb{R}^{k \cdot D_h \times D}
$$

where:

- $$\text{SA}_h(\mathbf{z})$$ is head $$h$$'s output, $$N \times D_h$$, each head with its own $$\mathbf{U}_{qkv}$$;
- the brackets place the $$k$$ outputs side by side: $$N \times k D_h$$, which is $$N \times D$$ when $$D_h = D/k$$;
- $$\mathbf{U}_{msa}$$ mixes the heads, $$k D_h \times D$$ (768 × 768 in ViT-B; the released model adds a bias);
- the result is $$N \times D$$, the same shape as the input, ready for the residual addition of Equation 2.

Vaswani et al. draw it like this:

> [!PAPER] Vaswani et al. (2017), Attention Is All You Need · Figure 2
> [![Figure 2 of Attention Is All You Need. Left: scaled dot-product attention as a flow of MatMul, Scale, optional Mask, SoftMax, MatMul over Q, K, V. Right: multi-head attention, where V, K and Q each pass through h parallel linear layers into h scaled dot-product attention blocks whose outputs are concatenated and passed through a final linear layer](https://ishwarj.com/img/papers/vit/p2-ext-vaswani-fig2.pngxt-vaswani-fig2.png)](/img/papers/vit/p2-ext-vaswani-fig2.png)
>
> **What it says:** multi-head attention "consists of several attention layers running in parallel". Each head has its own linear projections of $$V, K, Q$$; the heads are concatenated and go through one more linear layer.
>
> **What it means:**
>
> This is the original drawing of one head (left) and of multi-head attention (right). The right half is Equation 8 as a picture: parallel heads, then a join, then one more linear layer.
>
> The "Mask (opt.)" box in the left half is the one piece ViT never uses. A model that writes text must hide the future words from each word. A picture has no future, so every patch may see every patch.

The tiny example's second head and the join:

```text
-- Eq. 8 --
  [SA_1(z); SA_2(z)]  (the two heads side by side: 4 x 6)  shape (4, 6)
       t0 -0.876 -0.880 -0.416  0.727  0.760  0.367
       t1 -0.986 -1.185 -0.718  0.673  0.910  0.288
       t2 -2.653  0.629 -0.219  0.950  0.210  0.615
       t3 -3.014  0.002 -0.995  0.884  0.769  0.653
  U_msa: k D_h x D = 6 x 6  shape (6, 6)
               0     -1      0      0      0      0
               0      1      0     -1     -1     -1
              -1     -1      0      1      1      0
               1      0      0      0     -1     -1
               0      0      0     -1     -1     -1
               1     -1      0      1      0      1
  MSA(z) = [SA_1(z); SA_2(z)] U_msa  shape (4, 6)
       t0  1.510  0.045  0.000  0.070 -1.023 -0.240
       t1  1.679  0.231  0.000 -0.155 -1.116 -0.110
       t2  1.784  2.885  0.000 -0.442 -2.007 -1.173
       t3  2.091  1.299  0.000 -0.228 -1.766 -0.560
```

The first three columns of the concatenation are head 1's output from above; the last three are head 2's. Multiplying by the $$6 \times 6$$ matrix $$\mathbf{U}_{msa}$$ gives four rows of six numbers, the same shape as the input $$\mathbf{z}$$. (The third column of our random $$\mathbf{U}_{msa}$$ happens to be all zeros, so the third output column is zero; a trained matrix would not do that.) In ViT-B the same picture has 197 rows, 12 heads of 64, and a $$768 \times 768$$ $$\mathbf{U}_{msa}$$; the shapes figure above shows it.

> [!TAKEAWAYS] Key takeaways
> - Section 2 lists five earlier ways of putting attention on pictures (local, sparse and axial attention, 2×2 patches, CNN + attention, iGPT on pixels). All fight the **quadratic cost** by changing attention. ViT changes the *sequence* instead: **196 patches** rather than 50,176 pixels, and keeps full attention and the standard encoder.
> - The closest prior work is **Cordonnier et al. (2020)**: 2×2 patches with full attention, on 32×32 images. ViT's claim to novelty is scale: 16×16 patches, medium-resolution images, and 14M to 300M training images.
> - A 224×224×3 picture becomes $$\mathbf{x}_p$$ of **196 × 768** ($$N = HW/P^2$$); one matrix $$\mathbf{E}$$ (768 × 768) projects every patch. The library's 16×16 stride-16 convolution is exactly this linear layer (difference $$5 \times 10^{-6}$$).
> - **Eq. 1**: prepend the learned **[class]** token and add 197 learned **position** rows; our $$\mathbf{z}_0$$ matched the library to $$5 \times 10^{-6}$$. Without positions, attention is permutation equivariant: shuffle the patches and the outputs shuffle identically.
> - **Eq. 2 and 3**: pre-norm LayerNorm, 12-head attention, residual; LayerNorm, MLP 768 → 3,072 → 768 with GELU, residual. Our hand-written layer 1 matched to $$6.7 \times 10^{-6}$$, all 12 layers to a relative $$10^{-6}$$, the final logits to $$2.9 \times 10^{-6}$$.
> - In layer 1 the class token mostly attends to itself (0.76 in head 1); the most outward-looking head puts at most 0.0146 on any patch. Attention to the object comes later (Part 5).
> - **Eq. 4**: $$\mathbf{y} = \text{LN}(\mathbf{z}_L^0)$$, then one linear layer in the fine-tuned checkpoint (768 × 1000) versus a dense + tanh pooler in the pre-trained one. On our picture: **Egyptian cat, 0.937**.
> - Parameter count by formula $$= $$ real count, exactly: **86,567,656** (ViT-B/16 with head), **304,326,632** (ViT-L/16), **632,045,800** (ViT-H/14). Base and Huge match Table 1's 86M and 632M; Large gives 304M against the table's 307M. Two thirds of the parameters are MLP, one third attention, under 1% embeddings.
> - **Appendix A**: $$[\mathbf{q}, \mathbf{k}, \mathbf{v}] = \mathbf{z}\mathbf{U}_{qkv}$$, $$A = \text{softmax}(\mathbf{q}\mathbf{k}^\top/\sqrt{D_h})$$, $$\text{SA} = A\mathbf{v}$$, heads concatenated and multiplied by $$\mathbf{U}_{msa}$$, with $$D_h = D/k = 64$$. Unchanged from Vaswani et al. (2017).

**Next, in Part 3:** how a pre-trained ViT is adapted to a new task. Section 3.2 removes the head, attaches a zero-initialised $$D \times K$$ layer and fine-tunes at a *higher* resolution, which means more patches than the position table has rows and a 2D interpolation of the position embeddings, worked by hand. Then Section 4.1: the datasets (ImageNet, ImageNet-21k, JFT-300M), Table 1 read properly, the ResNet baselines and the hybrids, and the training recipe of Tables 3 and 4. Continue to [Part 3](https://ishwarj.com/papers/vit/part-3-fine-tuning-and-setup/).

<details>
<summary>Run it yourself</summary>

Every number in this part comes from two scripts in [`code/papers/vit/`](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/vit). `vit_part2_math.py` downloads `google/vit-base-patch16-224` (about 350 MB) and `google/vit-base-patch16-224-in21k`, loads one picture from the `huggingface/cats-image` dataset, and recomputes every step of Section 3.1 and Appendix A by hand; ViT-L/16 and ViT-H/14 are built on the meta device from their configurations, so no large weights are needed. It runs on a CPU in about a minute. `vit_part2_attn.py` is the four-token example and runs in a second.

```bash
pip install torch transformers datasets pillow
python vit_part2_math.py      # prints everything below, writes results/part2_math.json
python vit_part2_attn.py      # the tiny Eq. 5 to 8 example, writes results/part2_attn.json
```

<figure><img src="/img/papers/vit/part2_math-run.png" alt="Terminal output of vit_part2_math.py: the picture and model configuration, the 196 patches and the first numbers of patch 0, the conv-equals-linear check, Equation 1 with the class token and position numbers, LayerNorm by hand, the attention of head 1 and head 8 of layer 1 for the class token, GELU values, the twelve layers compared with the library, Equation 4 and the top-5 classes, the head and pooler check and the parameter counts" loading="lazy" /><figcaption>The real output of vit_part2_math.py (first 60 lines).</figcaption></figure>

<figure><img src="/img/papers/vit/part2_attn-run.png" alt="Terminal output of vit_part2_attn.py: the input z, U qkv, q k v, the scores, the softmax weights and SA of z for two heads, the concatenation and U msa, the hand check of one softmax entry and the permutation equivariance test" loading="lazy" /><figcaption>The real output of vit_part2_attn.py (first 60 lines).</figcaption></figure>

</details>

## References

**The ViT paper**

1. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. [*An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*](https://arxiv.org/abs/2010.11929) (ViT). ICLR 2021.
2. Google Research. [vision_transformer](https://github.com/google-research/vision_transformer): the official code and pre-trained checkpoints.
3. Hugging Face model cards [google/vit-base-patch16-224](https://huggingface.co/google/vit-base-patch16-224) (fine-tuned on ImageNet) and [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) (pre-trained only), the checkpoints used in this part.

**Papers the ViT paper cites in this part (the five that matter most here)**

4. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin. [*Attention Is All You Need*](https://arxiv.org/abs/1706.03762) (the Transformer). NeurIPS 2017.
5. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova. [*BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*](https://arxiv.org/abs/1810.04805). NAACL 2019.
6. N. Parmar, A. Vaswani, J. Uszkoreit, Ł. Kaiser, N. Shazeer, A. Ku, D. Tran. [*Image Transformer*](https://arxiv.org/abs/1802.05751) (local self-attention). ICML 2018.
7. J.-B. Cordonnier, A. Loukas, M. Jaggi. [*On the Relationship between Self-Attention and Convolutional Layers*](https://arxiv.org/abs/1911.03584) (the 2×2 patch model). ICLR 2020.
8. A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, N. Houlsby. [*Big Transfer (BiT): General Visual Representation Learning*](https://arxiv.org/abs/1912.11370). ECCV 2020.

**Other sources used in this part**

9. J. L. Ba, J. R. Kiros, G. E. Hinton. [*Layer Normalization*](https://arxiv.org/abs/1607.06450). arXiv 2016.
10. Code for this part: [`vit_part2_math.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part2_math.py) and [`vit_part2_attn.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/vit_part2_attn.py); figures by [`figs_part2.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/figs_part2.py), paper excerpts by [`shots_part2.py`](https://github.com/ishwar6/ishwar-books/blob/main/code/papers/vit/shots_part2.py).

