# How an LLM Writes: Prefill and Decode

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/writings/llm-inference-1-prefill-and-decode/
Updated: 2026-10-22
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/writings/llm-inference-1-prefill-and-decode.md


You type a question into a chatbot and press enter. There is a short pause. Then the first word appears, and the rest of the answer streams out at a steady pace, word after word.

The pause and the stream are two different jobs. They use the same model, but they hit opposite limits inside the computer. Once you see why, most of what LLM serving systems do, from vLLM's design to the size of a GPU bill, starts to make sense.

> [!TIP] The whole idea in one minute
> An LLM writes one small piece of text per step, in a loop. It reads your whole question in one fast step, then writes the answer one piece per step. Every step must move the entire model from the GPU's memory to its calculators, and that move, not the arithmetic, takes most of the time. So the most useful trick is to write for many conversations in the same step: one move of the model then serves all of them. This page builds each of those ideas from zero, with measurements.

Every number here comes from a real model, Qwen2.5-0.5B, on the GPU of an Apple M5 Pro (64 GB) with PyTorch. It is a small model, so the absolute times are small too. The shapes are the same on a data-centre GPU, and where it matters I show the numbers for an NVIDIA H100.

## How a model writes one word

A language model does not see words. It sees **tokens**.

> [!DEFINITION] Token
> A small piece of text that the model treats as one unit: often a whole word, sometimes part of a word or a punctuation mark. The model has a fixed list of all the tokens it knows, called its vocabulary.

Here is how our model's tokenizer cuts two sentences. Common words are one token each. A long or rare word is cut into pieces the model has seen often.

<figure class="fig"><svg viewBox="0 0 760 290" role="img" aria-label="Two sentences cut into tokens by the Qwen2.5 tokenizer. Common words are one token; a rare word is cut into pieces."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="30.0" y="36.0" text-anchor="start">"The cat sat on the"</text><text class="t-tick" x="30.0" y="56.0" text-anchor="start">5 tokens</text><rect class="box-1" x="30.0" y="74.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="62.0" y="96.0" text-anchor="middle">The</text><text class="t-tick" x="62.0" y="126.0" text-anchor="middle">785</text><rect class="box-1" x="104.0" y="74.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="136.0" y="96.0" text-anchor="middle">·cat</text><text class="t-tick" x="136.0" y="126.0" text-anchor="middle">8,251</text><rect class="box-1" x="178.0" y="74.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="210.0" y="96.0" text-anchor="middle">·sat</text><text class="t-tick" x="210.0" y="126.0" text-anchor="middle">7,578</text><rect class="box-1" x="252.0" y="74.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="284.0" y="96.0" text-anchor="middle">·on</text><text class="t-tick" x="284.0" y="126.0" text-anchor="middle">389</text><rect class="box-1" x="326.0" y="74.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="358.0" y="96.0" text-anchor="middle">·the</text><text class="t-tick" x="358.0" y="126.0" text-anchor="middle">279</text><text class="t-title" x="30.0" y="156.0" text-anchor="start">"Tokenization is unbelievably useful"</text><text class="t-tick" x="30.0" y="176.0" text-anchor="start">6 tokens</text><rect class="box-1" x="30.0" y="194.0" width="73.5" height="34.0" rx="6"/><text class="t-title" x="66.8" y="216.0" text-anchor="middle">Token</text><text class="t-tick" x="66.8" y="246.0" text-anchor="middle">3,323</text><rect class="box-1" x="113.5" y="194.0" width="92.5" height="34.0" rx="6"/><text class="t-title" x="159.8" y="216.0" text-anchor="middle">ization</text><text class="t-tick" x="159.8" y="246.0" text-anchor="middle">2,022</text><rect class="box-1" x="216.0" y="194.0" width="64.0" height="34.0" rx="6"/><text class="t-title" x="248.0" y="216.0" text-anchor="middle">·is</text><text class="t-tick" x="248.0" y="246.0" text-anchor="middle">374</text><rect class="box-1" x="290.0" y="194.0" width="92.5" height="34.0" rx="6"/><text class="t-title" x="336.2" y="216.0" text-anchor="middle">·unbelie</text><text class="t-tick" x="336.2" y="246.0" text-anchor="middle">38,937</text><rect class="box-1" x="392.5" y="194.0" width="73.5" height="34.0" rx="6"/><text class="t-title" x="429.2" y="216.0" text-anchor="middle">vably</text><text class="t-tick" x="429.2" y="246.0" text-anchor="middle">88,134</text><rect class="box-1" x="476.0" y="194.0" width="83.0" height="34.0" rx="6"/><text class="t-title" x="517.5" y="216.0" text-anchor="middle">·useful</text><text class="t-tick" x="517.5" y="246.0" text-anchor="middle">5,390</text><text class="t-tick" x="30.0" y="274.0" text-anchor="start">The dot · marks a space. The number under each token is its id in the vocabulary of 151,665 tokens.</text></svg><figcaption>Two sentences cut into tokens by Qwen2.5-0.5B's tokenizer. "The cat sat on the" is five tokens; "Tokenization" and "unbelievably" are each cut into two pieces.</figcaption></figure>

The model's job is narrow. You give it a list of tokens. It runs them through a stack of layers (24 in our model). Out comes one thing: a score for every token in its vocabulary, saying how likely it is to come **next**. You pick one of the likely tokens, and you have the next piece of text.

<figure class="fig"><svg viewBox="0 0 760 262" role="img" aria-label="The generation loop: the text so far goes into the model, one new token comes out, it is appended, and the model runs again."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="40" y="70" width="300" height="64" rx="10"/><text class="t-strong" x="190" y="108" text-anchor="middle">The cat sat on the</text><rect class="box-1" x="420" y="52" width="150" height="100" rx="16"/><text class="t-strong" x="495" y="98" text-anchor="middle">the model</text><text class="t-tick" x="495" y="118" text-anchor="middle">24 layers</text><rect class="box-3" x="650" y="76" width="70" height="52" rx="10"/><text class="t-strong" x="685" y="108" text-anchor="middle">mat</text><line class="edge" x1="340" y1="102" x2="416" y2="102" marker-end="url(#ah)"/><line class="edge" x1="570" y1="102" x2="646" y2="102" marker-end="url(#ah)"/><path class="path" d="M685,132 C685,215 190,215 190,138" fill="none" marker-end="url(#ah-on)"/><text class="t-note" x="438" y="246" text-anchor="middle">append the new token, run the whole model again</text><text class="t-tick" x="190" y="58" text-anchor="middle">everything so far</text><text class="t-tick" x="685" y="64" text-anchor="middle">one new token</text></svg><figcaption>The generation loop. Everything so far goes in, one new token comes out, it is added to the end, and the whole model runs again.</figcaption></figure>

To write a whole answer, you repeat that. Add the new token to the end, run the model again, and get the next one. A 500-token answer needs 500 runs of the model, one after another. Each run needs the token from the run before it, so the runs cannot happen at the same time.

That loop is the whole of LLM inference. Every trick in this series makes the loop faster or makes one GPU run more loops at once.

> [!DEFINITION] Inference
> Using a trained model to produce output, as opposed to training it. For a language model, inference is the generation loop above.

## Two phases: reading and writing

The clearest way to see the two phases is to follow one request from start to finish. Say you ask:

> What is the capital of France?

The tokenizer cuts that question into 7 tokens: `What`, `is`, `the`, `capital`, `of`, `France`, `?`. Here is what the model does with them, step by step.

<figure class="fig"><svg viewBox="0 0 760 388" role="img" aria-label="One request step by step: prefill reads all seven prompt tokens together and produces the first word; each decode step feeds back one word and produces the next."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="90" y="22" text-anchor="middle">step 0 · prefill</text><text class="t-tick" x="90" y="40" text-anchor="middle">all 7 prompt tokens at once</text><rect class="box-1" x="40" y="54" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="69" text-anchor="middle">What</text><rect class="box-1" x="40" y="78" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="93" text-anchor="middle">is</text><rect class="box-1" x="40" y="102" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="117" text-anchor="middle">the</text><rect class="box-1" x="40" y="126" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="141" text-anchor="middle">capital</text><rect class="box-1" x="40" y="150" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="165" text-anchor="middle">of</text><rect class="box-1" x="40" y="174" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="189" text-anchor="middle">France</text><rect class="box-1" x="40" y="198" width="100" height="20" rx="4"/><text class="t-tick" x="90" y="213" text-anchor="middle">?</text><line class="edge" x1="90" y1="224" x2="90" y2="246" marker-end="url(#ah)"/><rect class="box" x="25" y="250" width="130" height="40" rx="8"/><text class="t-strong" x="90" y="275" text-anchor="middle">model</text><line class="edge" x1="90" y1="290" x2="90" y2="314" marker-end="url(#ah)"/><rect class="box-3" x="50" y="318" width="80" height="28" rx="6"/><text class="t-strong" x="90" y="337" text-anchor="middle">The</text><text class="t-title" x="247" y="22" text-anchor="middle">step 1 · decode</text><text class="t-tick" x="247" y="40" text-anchor="middle">1 token in, 1 out</text><rect class="box-2" x="207" y="200" width="80" height="22" rx="4"/><text class="t-tick" x="247" y="215" text-anchor="middle">The</text><line class="edge" x1="247" y1="224" x2="247" y2="246" marker-end="url(#ah)"/><rect class="box" x="192" y="250" width="110" height="40" rx="8"/><text class="t-strong" x="247" y="275" text-anchor="middle">model</text><line class="edge" x1="247" y1="290" x2="247" y2="314" marker-end="url(#ah)"/><rect class="box-3" x="207" y="318" width="80" height="28" rx="6"/><text class="t-strong" x="247" y="337" text-anchor="middle">capital</text><path class="path" d="M130,332 C170,332 182,211 203,211" fill="none" marker-end="url(#ah-on)"/><text class="t-title" x="387" y="22" text-anchor="middle">step 2 · decode</text><text class="t-tick" x="387" y="40" text-anchor="middle">1 token in, 1 out</text><rect class="box-2" x="347" y="200" width="80" height="22" rx="4"/><text class="t-tick" x="387" y="215" text-anchor="middle">capital</text><line class="edge" x1="387" y1="224" x2="387" y2="246" marker-end="url(#ah)"/><rect class="box" x="332" y="250" width="110" height="40" rx="8"/><text class="t-strong" x="387" y="275" text-anchor="middle">model</text><line class="edge" x1="387" y1="290" x2="387" y2="314" marker-end="url(#ah)"/><rect class="box-3" x="347" y="318" width="80" height="28" rx="6"/><text class="t-strong" x="387" y="337" text-anchor="middle">is</text><path class="path" d="M287,332 C327,332 322,211 343,211" fill="none" marker-end="url(#ah-on)"/><text class="t-title" x="527" y="22" text-anchor="middle">step 3 · decode</text><text class="t-tick" x="527" y="40" text-anchor="middle">1 token in, 1 out</text><rect class="box-2" x="487" y="200" width="80" height="22" rx="4"/><text class="t-tick" x="527" y="215" text-anchor="middle">is</text><line class="edge" x1="527" y1="224" x2="527" y2="246" marker-end="url(#ah)"/><rect class="box" x="472" y="250" width="110" height="40" rx="8"/><text class="t-strong" x="527" y="275" text-anchor="middle">model</text><line class="edge" x1="527" y1="290" x2="527" y2="314" marker-end="url(#ah)"/><rect class="box-3" x="487" y="318" width="80" height="28" rx="6"/><text class="t-strong" x="527" y="337" text-anchor="middle">Paris</text><path class="path" d="M427,332 C467,332 462,211 483,211" fill="none" marker-end="url(#ah-on)"/><text class="t-title" x="667" y="22" text-anchor="middle">step 4 · decode</text><text class="t-tick" x="667" y="40" text-anchor="middle">1 token in, 1 out</text><rect class="box-2" x="627" y="200" width="80" height="22" rx="4"/><text class="t-tick" x="667" y="215" text-anchor="middle">Paris</text><line class="edge" x1="667" y1="224" x2="667" y2="246" marker-end="url(#ah)"/><rect class="box" x="612" y="250" width="110" height="40" rx="8"/><text class="t-strong" x="667" y="275" text-anchor="middle">model</text><line class="edge" x1="667" y1="290" x2="667" y2="314" marker-end="url(#ah)"/><rect class="box-3" x="627" y="318" width="80" height="28" rx="6"/><text class="t-strong" x="667" y="337" text-anchor="middle">.</text><path class="path" d="M567,332 C607,332 602,211 623,211" fill="none" marker-end="url(#ah-on)"/><text class="t-tick" x="20" y="374" text-anchor="start">Each new token is fed back in as the next step's input. The answer: The capital is Paris.</text></svg><figcaption>One request, step by step. Step 0 takes in all seven prompt tokens together and produces the first word. Every later step takes in one word and produces the next.</figcaption></figure>

**Step 0 is special.** The whole question already exists, so nothing about it is unknown. The model can take in **all 7 tokens at the same time**, the way you read a short sentence at a glance. Inside the GPU, the 7 tokens travel through every layer side by side, as one block of numbers. Each layer handles all 7 in one operation.

At the end of step 0 the model has done two things. It has picked the first word of the answer, `The`. And while it read, it wrote down **notes** about each of the 7 tokens, so that it never has to read them again. Those notes matter a lot for memory, and they come back at the end of this page.

**Every step after that is different.** To produce the second word, the model needs the first. To produce the third, it needs the second. So each step takes in exactly **one** new token, the one it just wrote, and produces exactly one more.

| Step | Goes in | Comes out | Phase |
|---|---|---|---|
| 0 | `What is the capital of France?` (7 tokens, together) | `The` | prefill |
| 1 | `The` | `capital` | decode |
| 2 | `capital` | `is` | decode |
| 3 | `is` | `Paris` | decode |
| 4 | `Paris` | `.` | decode |

> [!DEFINITION] Prefill
> The first step of a request. The model reads the whole prompt in one pass, writes its notes on every prompt token, and produces the first token of the answer.

> [!DEFINITION] Decode
> Every later step: the model takes in the one token it wrote last and produces one more. A 500-token answer is 500 decode steps.

### Why can't the answer be written all at once too?

Because the answer does not exist yet. The prompt is complete the moment you press enter, so all of it can be handled together. The answer is invented one token at a time, and the model cannot know word 3 before it has chosen word 2. Reading a finished page is fast because you can take it in at a glance. Writing a sentence is slower because each word waits for the one before it.

### What the two phases look like in time

Here are the two phases drawn to scale, with real timings for a 512-token prompt.

<figure class="fig"><svg viewBox="0 0 760 210" role="img" aria-label="A request has two phases. Prefill reads the whole prompt in one pass and produces the first token. Decode then produces one token per step."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-tick" x="80.0" y="40.0" text-anchor="start">time</text><line class="edge" x1="114.0" y1="36.0" x2="200.0" y2="36.0" marker-end="url(#ah)"/><g class="mark"><title>prefill: all 512 prompt tokens in one pass, 27.4 ms</title><rect class="box-1" x="80.0" y="70.0" width="91.2" height="54.0" rx="6"/></g><text class="t-title" x="125.6" y="102.0" text-anchor="middle">prefill</text><g class="mark"><title>decode step 1: one new token, 10.0 ms</title><rect class="box-2" x="174.2" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 2: one new token, 10.0 ms</title><rect class="box-2" x="207.5" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 3: one new token, 10.0 ms</title><rect class="box-2" x="240.8" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 4: one new token, 10.0 ms</title><rect class="box-2" x="274.1" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 5: one new token, 10.0 ms</title><rect class="box-2" x="307.3" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 6: one new token, 10.0 ms</title><rect class="box-2" x="340.6" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 7: one new token, 10.0 ms</title><rect class="box-2" x="373.9" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 8: one new token, 10.0 ms</title><rect class="box-2" x="407.2" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 9: one new token, 10.0 ms</title><rect class="box-2" x="440.5" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 10: one new token, 10.0 ms</title><rect class="box-2" x="473.8" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 11: one new token, 10.0 ms</title><rect class="box-2" x="507.0" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 12: one new token, 10.0 ms</title><rect class="box-2" x="540.3" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 13: one new token, 10.0 ms</title><rect class="box-2" x="573.6" y="70.0" width="30.3" height="54.0" rx="4"/></g><g class="mark"><title>decode step 14: one new token, 10.0 ms</title><rect class="box-2" x="606.9" y="70.0" width="30.3" height="54.0" rx="4"/></g><text class="t-title" x="648.2" y="103.0" text-anchor="start">...</text><line class="axis" x1="80.0" y1="140.0" x2="171.2" y2="140.0"/><line class="axis" x1="80.0" y1="134.0" x2="80.0" y2="146.0"/><line class="axis" x1="171.2" y1="134.0" x2="171.2" y2="146.0"/><text class="t-note" x="80.0" y="166.0" text-anchor="start">time to first token (TTFT)</text><line class="axis" x1="307.3" y1="140.0" x2="337.6" y2="140.0"/><line class="axis" x1="307.3" y1="134.0" x2="307.3" y2="146.0"/><line class="axis" x1="337.6" y1="134.0" x2="337.6" y2="146.0"/><text class="t-note" x="307.3" y="166.0" text-anchor="start">time per output token (TPOT)</text><text class="t-tick" x="80.0" y="194.0" text-anchor="start">measured: 512-token prompt, Qwen2.5-0.5B on an Apple M5 Pro. Prefill 27 ms, each decode step 10 ms.</text></svg><figcaption>The two phases drawn to scale from real measurements: one wide prefill step, then a long run of narrow decode steps, one for each new token.</figcaption></figure>

The two phases match the two numbers that every serving dashboard shows, and the two things you feel as a user.

> [!DEFINITION] TTFT (time to first token)
> The time from sending a request to seeing the first token of the answer. It is mostly the prefill step, plus any time the request waits in a queue.

> [!DEFINITION] TPOT (time per output token)
> The average time between two tokens of the answer after the first one, also called inter-token latency. It is the length of one decode step.

| Metric | What you feel | Set by |
|---|---|---|
| TTFT | the pause before anything appears | prefill (step 0) |
| TPOT | how fast the words stream after that | decode (every later step) |

In the France example, TTFT is the length of step 0. TPOT is the average length of steps 1 to 4. In the 512-token run, TTFT is about 27 ms and TPOT is about 10 ms.

## The measurement that surprises everyone

Here is a simple experiment. Take a 512-token prompt and measure how long the model takes to **read** it (prefill). Then measure how long it takes to **write** 512 new tokens (decode). It is the same model and the same number of tokens.

<figure class="fig"><svg viewBox="0 0 760 150" role="img" aria-label="512 tokens in takes 27 ms, 512 tokens out takes about 5.2 seconds."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker></defs><g class="mark"><title>read 512 prompt tokens (prefill): 27.4 ms</title><rect class="hit-area" x="0" y="16.0" width="760" height="59.0"/><path class="s1 bar-mark" d="M260,33.5H260.0Q262.06849100123367,33.5 262.06849100123367,35.56849100123368V55.43150899876632Q262.06849100123367,57.5 260.0,57.5H260Z"/></g><text class="t-note" x="248" y="50.5" text-anchor="end">read 512 prompt tokens (prefill)</text><text class="t-val" x="270.06849100123367" y="50.5" text-anchor="start">27.4 ms</text><g class="mark"><title>write 512 new tokens (decode): 5,158 ms</title><rect class="hit-area" x="0" y="75.0" width="760" height="59.0"/><path class="s2 bar-mark" d="M260,92.5H646.0Q650.0,92.5 650.0,96.5V112.5Q650.0,116.5 646.0,116.5H260Z"/></g><text class="t-note" x="248" y="109.5" text-anchor="end">write 512 new tokens (decode)</text><text class="t-val" x="658.0" y="109.5" text-anchor="start">5,158 ms</text><line class="axis" x1="260" y1="16" x2="260" y2="134"/></svg><figcaption>Reading 512 tokens versus writing 512 tokens, measured on Qwen2.5-0.5B. The prefill bar is so short it is barely visible.</figcaption></figure>

Reading took **27 milliseconds**. Writing took **5.2 seconds**. That is about 190 times longer for the same number of tokens.

Nothing is broken. Every LLM on every GPU behaves like this. To see why, we need to look at what a GPU is made of, at the level that matters here.

## What a GPU is made of

First, what is the model, physically? It is a very large set of numbers called **weights**.

> [!DEFINITION] Weights (parameters)
> The numbers a model learned during training. Running the model means multiplying the input by these numbers, layer by layer. Our model has 494 million of them.

Each weight is stored in 16-bit format, which takes 2 bytes. So our small model is 494 million × 2 bytes = **0.99 GB** of numbers. Llama 3.1 8B is about 16 GB. The largest models are hundreds of gigabytes.

For this page, a GPU has two parts that matter: a place that **stores** numbers and a place that **calculates** with them.

> [!DEFINITION] HBM (GPU memory)
> High-bandwidth memory: the GPU's main memory, built from memory chips beside the processor. It holds the weights and everything else the model needs. An H100 has 80 GB of it.

> [!DEFINITION] Compute cores
> The parts of the GPU that do arithmetic. There are thousands of them, and they work in parallel. They can only calculate with numbers that have already arrived from memory.

Between the two runs a link with a fixed speed. So a GPU has two speed limits, and each one has its own unit.

> [!DEFINITION] FLOP and FLOP/s
> One floating-point operation: one multiplication or one addition on decimal numbers. FLOP/s counts how many of them the compute cores can do per second. An H100 can do about 989 trillion 16-bit FLOP/s (989 TFLOP/s).

> [!DEFINITION] Memory bandwidth
> How many bytes per second can move from memory to the compute cores. An H100 moves about 3.35 trillion bytes per second (3.35 TB/s). I measured our M5 Pro at 262 GB/s.

<figure class="fig"><svg viewBox="0 0 760 300" role="img" aria-label="A GPU, reduced to two parts: memory that holds the weights, compute cores that do the arithmetic, and a link between them with a fixed bandwidth."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="30.0" y="50.0" width="210.0" height="230.0" rx="14"/><text class="t-title" x="135.0" y="80.0" text-anchor="middle">memory (HBM)</text><text class="t-tick" x="135.0" y="100.0" text-anchor="middle">big, but far from the cores</text><rect class="box-2" x="60.0" y="116.0" width="150.0" height="26.0" rx="5"/><text class="t-tick" x="135.0" y="134.0" text-anchor="middle">weights</text><rect class="box-2" x="60.0" y="150.0" width="150.0" height="26.0" rx="5"/><text class="t-tick" x="135.0" y="168.0" text-anchor="middle">weights</text><rect class="box-2" x="60.0" y="184.0" width="150.0" height="26.0" rx="5"/><text class="t-tick" x="135.0" y="202.0" text-anchor="middle">weights</text><rect class="box-2" x="60.0" y="218.0" width="150.0" height="26.0" rx="5"/><text class="t-tick" x="135.0" y="236.0" text-anchor="middle">weights</text><text class="t-note" x="135.0" y="266.0" text-anchor="middle">H100: 80 GB</text><path class="path" d="M246,165 L470,165" marker-end="url(#ah-on)"/><text class="t-title" x="358.0" y="140.0" text-anchor="middle">memory bandwidth</text><text class="t-tick" x="358.0" y="196.0" text-anchor="middle">bytes per second through the link</text><text class="t-note" x="358.0" y="216.0" text-anchor="middle">H100: 3.35 TB/s</text><text class="t-note" x="358.0" y="236.0" text-anchor="middle">our M5 Pro: 262 GB/s (measured)</text><rect class="box" x="480.0" y="50.0" width="250.0" height="230.0" rx="14"/><text class="t-title" x="605.0" y="80.0" text-anchor="middle">compute cores</text><text class="t-tick" x="605.0" y="100.0" text-anchor="middle">thousands of multiply-add units</text><rect class="box-1" x="508.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="533.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="558.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="583.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="608.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="633.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="658.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="683.0" y="116.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="508.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="533.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="558.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="583.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="608.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="633.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="658.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="683.0" y="141.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="508.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="533.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="558.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="583.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="608.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="633.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="658.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="683.0" y="166.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="508.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="533.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="558.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="583.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="608.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="633.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="658.0" y="191.0" width="19.0" height="19.0" rx="3"/><rect class="box-1" x="683.0" y="191.0" width="19.0" height="19.0" rx="3"/><text class="t-tick" x="605.0" y="236.0" text-anchor="middle">FLOPs per second</text><text class="t-note" x="605.0" y="266.0" text-anchor="middle">H100: 989 trillion FLOP/s</text><text class="t-tick" x="380.0" y="26.0" text-anchor="middle">Two speeds decide everything: how fast the cores calculate, and how fast memory feeds them.</text></svg><figcaption>A GPU reduced to the two parts that matter here: memory that holds the weights, compute cores that do the arithmetic, and a link between them with a fixed bandwidth.</figcaption></figure>

Now compare the two speeds of the H100. In one second it can do 989 trillion operations, but it can move only 3.35 trillion bytes. Divide one by the other: in the time it takes to bring in **one byte**, the cores could do about **295 operations**. The calculators are far faster than the delivery.

### A kitchen with a narrow door

Picture a chef who cooks extremely fast. The chef works at a small counter with room for only a few jars. All the ingredients sit in a big pantry down the hall, behind a narrow door. To cook, the chef must bring jars through the door, use them, and send them back to make room for the next ones.

<figure class="fig"><svg viewBox="0 0 760 330" role="img" aria-label="The kitchen picture: a fast chef with a small counter, a pantry full of jars, and a narrow door between them."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box-2" x="30.0" y="40.0" width="200.0" height="200.0" rx="14"/><text class="t-title" x="130.0" y="68.0" text-anchor="middle">the pantry</text><rect class="box" x="52.0" y="86.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="92.0" y="86.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="132.0" y="86.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="172.0" y="86.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="52.0" y="126.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="92.0" y="126.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="132.0" y="126.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="172.0" y="126.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="52.0" y="166.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="92.0" y="166.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="132.0" y="166.0" width="30.0" height="30.0" rx="5"/><rect class="box" x="172.0" y="166.0" width="30.0" height="30.0" rx="5"/><text class="t-tick" x="130.0" y="222.0" text-anchor="middle">every ingredient jar</text><rect class="box-on" x="300.0" y="120.0" width="26.0" height="60.0" rx="4"/><text class="t-note" x="313.0" y="112.0" text-anchor="middle">narrow door</text><line class="path" x1="236.0" y1="150.0" x2="296.0" y2="150.0" marker-end="url(#ah-on)"/><line class="path" x1="330.0" y1="150.0" x2="400.0" y2="150.0" marker-end="url(#ah-on)"/><rect class="box-1" x="410.0" y="40.0" width="320.0" height="200.0" rx="14"/><text class="t-title" x="570.0" y="68.0" text-anchor="middle">the chef at a small counter</text><circle class="box" cx="490" cy="130" r="28"/><text class="t-title" x="490.0" y="135.0" text-anchor="middle">chef</text><rect class="box" x="540.0" y="112.0" width="160.0" height="40.0" rx="6"/><text class="t-tick" x="620.0" y="137.0" text-anchor="middle">room for a few jars</text><text class="t-tick" x="570.0" y="196.0" text-anchor="middle">cooks very fast, but only with</text><text class="t-tick" x="570.0" y="214.0" text-anchor="middle">what is on the counter</text><text class="t-title" x="30.0" y="274.0" text-anchor="start">pantry</text><text class="t-note" x="170.0" y="274.0" text-anchor="start">= GPU memory (HBM)</text><text class="t-title" x="390.0" y="274.0" text-anchor="start">ingredient jars</text><text class="t-note" x="530.0" y="274.0" text-anchor="start">= the model's weights</text><text class="t-title" x="30.0" y="300.0" text-anchor="start">the door</text><text class="t-note" x="170.0" y="300.0" text-anchor="start">= memory bandwidth</text><text class="t-title" x="390.0" y="300.0" text-anchor="start">chef's speed</text><text class="t-note" x="530.0" y="300.0" text-anchor="start">= FLOPs per second</text></svg><figcaption>The kitchen picture. A fast chef works at a small counter. The ingredient jars wait in a pantry, and everything must pass through a narrow door.</figcaption></figure>

Now the strange rule of this kitchen. Each round of cooking needs **every jar in the pantry**, once, in order.

With one dish per round, every jar comes through the door for a single dish. The chef waits at the door most of the time. If the chef cooks 512 dishes per round, each jar is used 512 times before it goes back. Now the chef is the busy one.

The mapping is exact. The pantry is HBM, and the jars are the weights. The door is the memory bandwidth, and the chef's speed is FLOP/s. The small counter is the small, fast memory beside the compute cores, which is far too small to keep a whole model. That is why every step must bring the weights in again.

## Why decode waits for memory

A layer of the model is mostly a **matrix multiplication**: a grid of weights times the numbers that describe each token.

> [!DEFINITION] Matrix multiplication
> Multiplying a grid of numbers (a matrix) by a list of numbers (a vector) or by another grid. Each output number is a sum of products: each weight is multiplied by an input number and added to a running total. This "multiply-add" is 2 FLOPs.

A tiny example shows the whole effect. Take a 3 × 3 weight matrix, which is 9 weights. At 2 bytes each, reading it costs 18 bytes. Multiply it by one token's 3 numbers: each weight gets one multiply-add, so the work is 9 × 2 = 18 FLOPs. That is **1 FLOP for every byte** read.

Now multiply the same matrix by four tokens at once. You still read the 9 weights once (18 bytes), but each weight now does four multiply-adds: 72 FLOPs, or **4 FLOPs per byte**.

<figure class="fig"><svg viewBox="0 0 760 250" role="img" aria-label="A 3 by 3 weight matrix multiplied by one token and by four tokens. The weights are read once in both cases; four tokens get four times the work out of them."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="30.0" y="26.0" text-anchor="start">A. decode, 1 token</text><rect class="box-2" x="30.0" y="50.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="43.5" y="69.0" text-anchor="middle">1</text><rect class="box-2" x="60.0" y="50.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="73.5" y="69.0" text-anchor="middle">2</text><rect class="box-2" x="90.0" y="50.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="103.5" y="69.0" text-anchor="middle">0</text><rect class="box-2" x="30.0" y="80.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="43.5" y="99.0" text-anchor="middle">0</text><rect class="box-2" x="60.0" y="80.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="73.5" y="99.0" text-anchor="middle">1</text><rect class="box-2" x="90.0" y="80.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="103.5" y="99.0" text-anchor="middle">3</text><rect class="box-2" x="30.0" y="110.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="43.5" y="129.0" text-anchor="middle">2</text><rect class="box-2" x="60.0" y="110.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="73.5" y="129.0" text-anchor="middle">0</text><rect class="box-2" x="90.0" y="110.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="103.5" y="129.0" text-anchor="middle">1</text><text class="t-tick" x="73.0" y="155.0" text-anchor="middle">weights W</text><text class="t-big" x="130.0" y="100.0" text-anchor="middle">×</text><rect class="box-1" x="150.0" y="50.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="163.5" y="69.0" text-anchor="middle">1</text><rect class="box-1" x="150.0" y="80.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="163.5" y="99.0" text-anchor="middle">2</text><rect class="box-1" x="150.0" y="110.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="163.5" y="129.0" text-anchor="middle">1</text><text class="t-tick" x="164.0" y="155.0" text-anchor="middle">token</text><text class="t-big" x="210.0" y="100.0" text-anchor="middle">=</text><rect class="box-3" x="230.0" y="50.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="243.5" y="69.0" text-anchor="middle">5</text><rect class="box-3" x="230.0" y="80.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="243.5" y="99.0" text-anchor="middle">5</text><rect class="box-3" x="230.0" y="110.0" width="27.0" height="27.0" rx="4"/><text class="t-title" x="243.5" y="129.0" text-anchor="middle">3</text><text class="t-tick" x="244.0" y="155.0" text-anchor="middle">output</text><text class="t-note" x="30.0" y="186.0" text-anchor="start">9 weights read = 18 bytes</text><text class="t-note" x="30.0" y="206.0" text-anchor="start">9 multiply-adds = 18 FLOPs</text><text class="t-title" x="30.0" y="230.0" text-anchor="start">18 FLOPs / 18 bytes = 1 FLOP per byte</text><text class="t-title" x="390.0" y="26.0" text-anchor="start">B. four tokens at once</text><rect class="box-2" x="390.0" y="50.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="402.5" y="69.0" text-anchor="middle">1</text><rect class="box-2" x="418.0" y="50.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="430.5" y="69.0" text-anchor="middle">2</text><rect class="box-2" x="446.0" y="50.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="458.5" y="69.0" text-anchor="middle">0</text><rect class="box-2" x="390.0" y="78.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="402.5" y="97.0" text-anchor="middle">0</text><rect class="box-2" x="418.0" y="78.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="430.5" y="97.0" text-anchor="middle">1</text><rect class="box-2" x="446.0" y="78.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="458.5" y="97.0" text-anchor="middle">3</text><rect class="box-2" x="390.0" y="106.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="402.5" y="125.0" text-anchor="middle">2</text><rect class="box-2" x="418.0" y="106.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="430.5" y="125.0" text-anchor="middle">0</text><rect class="box-2" x="446.0" y="106.0" width="25.0" height="25.0" rx="4"/><text class="t-title" x="458.5" y="125.0" text-anchor="middle">1</text><text class="t-tick" x="431.0" y="155.0" text-anchor="middle">the same W</text><text class="t-big" x="487.0" y="100.0" text-anchor="middle">×</text><rect class="box-1" x="502.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="514.0" y="69.0" text-anchor="middle">1</text><rect class="box-1" x="529.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="541.0" y="69.0" text-anchor="middle">0</text><rect class="box-1" x="556.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="568.0" y="69.0" text-anchor="middle">2</text><rect class="box-1" x="583.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="595.0" y="69.0" text-anchor="middle">1</text><rect class="box-1" x="502.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="514.0" y="96.0" text-anchor="middle">2</text><rect class="box-1" x="529.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="541.0" y="96.0" text-anchor="middle">1</text><rect class="box-1" x="556.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="568.0" y="96.0" text-anchor="middle">0</text><rect class="box-1" x="583.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="595.0" y="96.0" text-anchor="middle">1</text><rect class="box-1" x="502.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="514.0" y="123.0" text-anchor="middle">1</text><rect class="box-1" x="529.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="541.0" y="123.0" text-anchor="middle">2</text><rect class="box-1" x="556.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="568.0" y="123.0" text-anchor="middle">1</text><rect class="box-1" x="583.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="595.0" y="123.0" text-anchor="middle">0</text><text class="t-tick" x="554.0" y="155.0" text-anchor="middle">4 tokens</text><text class="t-big" x="620.0" y="100.0" text-anchor="middle">=</text><rect class="box-3" x="636.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="648.0" y="69.0" text-anchor="middle">5</text><rect class="box-3" x="663.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="675.0" y="69.0" text-anchor="middle">2</text><rect class="box-3" x="690.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="702.0" y="69.0" text-anchor="middle">2</text><rect class="box-3" x="717.0" y="50.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="729.0" y="69.0" text-anchor="middle">3</text><rect class="box-3" x="636.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="648.0" y="96.0" text-anchor="middle">5</text><rect class="box-3" x="663.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="675.0" y="96.0" text-anchor="middle">7</text><rect class="box-3" x="690.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="702.0" y="96.0" text-anchor="middle">3</text><rect class="box-3" x="717.0" y="77.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="729.0" y="96.0" text-anchor="middle">1</text><rect class="box-3" x="636.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="648.0" y="123.0" text-anchor="middle">3</text><rect class="box-3" x="663.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="675.0" y="123.0" text-anchor="middle">2</text><rect class="box-3" x="690.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="702.0" y="123.0" text-anchor="middle">5</text><rect class="box-3" x="717.0" y="104.0" width="24.0" height="24.0" rx="4"/><text class="t-title" x="729.0" y="123.0" text-anchor="middle">2</text><text class="t-tick" x="688.0" y="155.0" text-anchor="middle">4 outputs</text><text class="t-note" x="390.0" y="186.0" text-anchor="start">9 weights read = 18 bytes (once)</text><text class="t-note" x="390.0" y="206.0" text-anchor="start">4 × 9 multiply-adds = 72 FLOPs</text><text class="t-title" x="390.0" y="230.0" text-anchor="start">72 FLOPs / 18 bytes = 4 FLOPs per byte</text><line class="drop" x1="362.0" y1="40.0" x2="362.0" y2="236.0"/></svg><figcaption>A 3 × 3 weight matrix times one token, and the same matrix times four tokens. The weights are read once in both cases; four tokens get four times the work out of the same read.</figcaption></figure>

Decode is panel A, at a huge scale. It multiplies each weight matrix by **one** token, which engineers call a matrix-vector multiplication. Prefill is panel B with 512 columns instead of 4: a matrix-matrix multiplication. Both read every weight once per pass. Prefill gets 512 tokens of work out of that read. Decode gets one.

<figure class="fig"><svg viewBox="0 0 760 224" role="img" aria-label="Every forward pass reads all the weights from memory. Prefill uses that read for 512 tokens of work, decode for one."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker></defs><rect class="box" x="30" y="40" width="200" height="170" rx="14"/><text class="t-strong" x="130" y="72" text-anchor="middle">GPU memory</text><text class="t-note" x="130" y="94" text-anchor="middle">weights: 0.99 GB</text><rect class="box-1" x="56" y="110" width="148" height="11" rx="3"/><rect class="box-1" x="56" y="126" width="148" height="11" rx="3"/><rect class="box-1" x="56" y="142" width="148" height="11" rx="3"/><rect class="box-1" x="56" y="158" width="148" height="11" rx="3"/><rect class="box-1" x="56" y="174" width="148" height="11" rx="3"/><rect class="box-1" x="56" y="190" width="148" height="11" rx="3"/><rect class="box" x="530" y="40" width="200" height="170" rx="14"/><text class="t-strong" x="630" y="72" text-anchor="middle">compute cores</text><text class="t-note" x="630" y="94" text-anchor="middle">multiply and add</text><line class="edge" x1="230" y1="128" x2="526" y2="128" marker-end="url(#ah)"/><text class="t-tick" x="378" y="120" text-anchor="middle">read all 0.99 GB of weights</text><rect class="box-2" x="560" y="114" width="140" height="26" rx="6"/><text class="t-note" x="630" y="132" text-anchor="middle">decode: 1 token</text><line class="edge" x1="230" y1="176" x2="526" y2="176" marker-end="url(#ah)"/><text class="t-tick" x="378" y="168" text-anchor="middle">read all 0.99 GB of weights</text><rect class="box-1" x="560" y="162" width="140" height="26" rx="6"/><text class="t-note" x="630" y="180" text-anchor="middle">prefill: 512 tokens</text><text class="t-note" x="380" y="30" text-anchor="middle">memory bandwidth, measured: 262 GB/s</text></svg><figcaption>Both phases pay for the same trip through memory. Prefill gets 512 tokens of work out of it. Decode gets one.</figcaption></figure>

We can check this with our measurements. Our GPU moves 262 GB/s, and the weights are 0.99 GB. So reading the weights once takes at least 0.99 ÷ 262 = **3.8 ms**. That is a floor: no decode step can be faster than one read of the model. The arithmetic for one token is about 0.99 billion FLOPs, which the GPU's cores could finish in a small fraction of that time.

<figure class="fig"><svg viewBox="0 0 760 166" role="img" aria-label="A 10 ms decode step at batch 1: at least 3.8 ms reading weights, most of the rest fixed overhead, and almost no time doing arithmetic."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="40.0" y="30.0" text-anchor="start">one decode step, batch 1: 10.0 ms (measured)</text><g class="mark"><title>reading 0.99 GB of weights at 262 GB/s: at least 3.77 ms</title><rect class="box-2" x="40.0" y="50.0" width="256.4" height="46.0" rx="6"/></g><text class="t-title" x="168.2" y="78.0" text-anchor="middle">read the weights: 3.8 ms</text><g class="mark"><title>the rest of the step: about 6.2 ms</title><rect class="box" x="299.4" y="50.0" width="420.6" height="46.0" rx="6"/></g><text class="t-title" x="508.2" y="78.0" text-anchor="middle">everything else: about 6.2 ms</text><text class="t-tick" x="40.0" y="122.0" text-anchor="start">the floor: 0.99 GB ÷ 262 GB/s</text><text class="t-tick" x="304.4" y="122.0" text-anchor="start">starting hundreds of small GPU jobs one after another, and gaps between them</text><line class="axis" x1="40.0" y1="100.0" x2="40.0" y2="106.0"/><line class="axis" x1="176.0" y1="100.0" x2="176.0" y2="106.0"/><line class="axis" x1="312.0" y1="100.0" x2="312.0" y2="106.0"/><line class="axis" x1="448.0" y1="100.0" x2="448.0" y2="106.0"/><line class="axis" x1="584.0" y1="100.0" x2="584.0" y2="106.0"/><line class="axis" x1="720.0" y1="100.0" x2="720.0" y2="106.0"/><text class="t-tick" x="40.0" y="152.0" text-anchor="start">The arithmetic itself, 0.99 GFLOP, is a tiny part: at full speed an H100 would finish it in about 1 microsecond.</text></svg><figcaption>Where the 10 ms of one decode step go at batch 1: at least 3.8 ms reading weights, most of the rest fixed overhead, and almost no time on arithmetic.</figcaption></figure>

We measured a decode step at **10 ms**. The weight read is over a third of it. Most of the rest is a fixed cost per step. A small model like this one runs as hundreds of small GPU jobs, called kernels, one after another. Each one has a start-up cost, and the cores sit idle between them.

> [!DEFINITION] Kernel
> One function that runs on the GPU, such as one matrix multiplication or one normalisation. A forward pass of the model is a long sequence of kernels.

Neither part of the 10 ms is arithmetic. That is the central fact of decode.

> [!DEFINITION] Memory-bound and compute-bound
> A job is memory-bound when its speed is set by how fast data arrives from memory, so the compute cores wait. It is compute-bound when its speed is set by how fast the cores can calculate, so memory waits.

> [!NOTE] The one idea
> **Decode is limited by how fast the GPU can move the weights, not by how fast it can multiply.** Decode is memory-bound. Prefill, which uses each weight for many tokens, is compute-bound.

The vLLM paper puts the same fact at the centre of its introduction.

> [!PAPER] Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention · §1 · page 1
> [![The paragraph about autoregressive generation, with one at a time, repeated until the model outputs a termination token, and memory-bound, underutilizing the computation power of GPUs highlighted](https://ishwarj.com/img/papers/vllm/p1-intro-autoregressive.png-autoregressive.png)](/img/papers/vllm/p1-intro-autoregressive.png)
>
> **What it says:** the model generates tokens "one at a time", and this is "repeated until the model outputs a termination token". The result is a workload that is "memory-bound, underutilizing the computation power of GPUs".
>
> **What it means:**
>
> "Autoregressive" is the generation loop: each token is produced from all the tokens before it. A termination token is a special token that means "the answer is finished", so the loop stops there. The last sentence is the chef at the door. The GPU's calculators sit mostly idle because the loop feeds them one token at a time.
>
> This is the problem the whole vLLM paper sets out to fix. It cannot make one decode step do more arithmetic per byte for a single user. It can make each step serve many users, which is where this page ends.

Two lines on page 3 of the same paper name the two kinds of multiplication from the 3 × 3 example.

> [!PAPER] Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention · §2.2 · page 3
> [![The paragraph on the prompt phase, with The prompt phase and parallelized using matrix-matrix multiplication operations highlighted](https://ishwarj.com/img/papers/vllm/p2-prompt-phase.pngp2-prompt-phase.png)](/img/papers/vllm/p2-prompt-phase.png)
>
> **What it says:** in "the prompt phase", the work "can be parallelized using matrix-matrix multiplication operations", so it "can efficiently use the parallelism inherent in GPUs".
>
> **What it means:**
>
> The paper's "prompt phase" is our prefill. The symbols $$x_1, \ldots, x_n$$ are the $$n$$ prompt tokens, and $$P(x_{n+1} \mid x_1, \ldots, x_n)$$ means "the chance of each possible next token, given the prompt". The key vectors $$k$$ and value vectors $$v$$ are the notes the model writes on each prompt token.
>
> The point of the highlight is panel B of the 3 × 3 example. All prompt tokens are known, so they form a matrix with one column per token, and each weight is used once for every column.

> [!PAPER] Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention · §2.2 · page 3
> [![The paragraph on the autoregressive generation phase, with only the new key and value vector, maximum length, end-of-sequence token, matrix-vector multiplication and becomes memory-bound highlighted](https://ishwarj.com/img/papers/vllm/p2-gen-phase.pnglm/p2-gen-phase.png)](/img/papers/vllm/p2-gen-phase.png)
>
> **What it says:** the generation phase "often uses matrix-vector multiplication, which is less efficient", so it "becomes memory-bound" and is responsible for most of the time a single request takes.
>
> **What it means:**
>
> This is decode, panel A. At each step $$t$$ the model takes in one token and computes the notes for that one token only. The older notes were saved in earlier steps. The step multiplies every weight matrix by a single vector, so each weight gives only one multiply-add for the bytes it cost to read.
>
> The paper also says when decode stops: at a maximum length, or when the model writes the end-of-sequence token. Until then, every step reads the whole model again. That is why decode, and not prefill, fills most of the 5.2 seconds in our measurement.

## From a Python line to the GPU: CUDA and kernels

The 10 ms step had a large part that was neither reading weights nor arithmetic: the fixed cost of starting hundreds of kernels. To see where that cost comes from, follow one line of Python all the way into the GPU.

On NVIDIA GPUs, the software layer between your program and the chip is called **CUDA**.

> [!DEFINITION] CUDA
> NVIDIA's platform for running programs on its GPUs. It has a language based on C++, a driver, and libraries of ready-made kernels, such as matrix multiplication. PyTorch calls these libraries for you.

### An analogy: a foreman and a crew

Think of a building site. A foreman (the CPU) reads the plan, one line at a time. For each line, the foreman shouts one order to a huge crew (the GPU), such as "lay all these bricks". Then the crew splits into teams, and the teams work at the same time. The foreman does no building at all. The foreman only reads the plan and gives orders.

Two facts follow from this picture. Each order takes the foreman a moment to shout, however small the job. And if every job is small, the crew finishes each one before the next order arrives, so the crew stands and waits.

### What happens for `y = x @ W`

Here is the real sequence for one matrix multiplication in PyTorch.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="What happens for one line of PyTorch: the CPU launches a kernel, its blocks of threads run on the streaming multiprocessors, they read the inputs from HBM through the on-chip memories and write the result back."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">1. on the CPU</text><rect class="box-2" x="20.0" y="36.0" width="220.0" height="54.0" rx="8"/><text class="t-mono" x="130.0" y="58.0" text-anchor="middle">Python: y = x @ W</text><text class="t-tick" x="130.0" y="78.0" text-anchor="middle">PyTorch picks a GPU kernel</text><line class="edge" x1="130.0" y1="92.0" x2="130.0" y2="118.0" marker-end="url(#ah)"/><rect class="box-2" x="20.0" y="120.0" width="220.0" height="66.0" rx="8"/><text class="t-note" x="130.0" y="142.0" text-anchor="middle">launch the kernel:</text><text class="t-tick" x="130.0" y="160.0" text-anchor="middle">its code, its inputs and</text><text class="t-tick" x="130.0" y="176.0" text-anchor="middle">how many blocks of threads</text><path class="path" d="M242,153 L300,153" marker-end="url(#ah-on)"/><text class="t-tick" x="271.0" y="143.0" text-anchor="middle">launch</text><text class="t-title" x="310.0" y="24.0" text-anchor="start">2. on the GPU: the blocks of threads are spread over the SMs</text><rect class="box" x="305.0" y="36.0" width="440.0" height="184.0" rx="12"/><rect class="box-1" x="320.0" y="50.0" width="96.0" height="120.0" rx="8"/><text class="t-note" x="368.0" y="68.0" text-anchor="middle">SM 0</text><rect class="box-3" x="330.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="350.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="370.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="390.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="330.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="350.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="370.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="390.0" y="98.0" width="15.0" height="15.0" rx="2"/><text class="t-tick" x="368.0" y="134.0" text-anchor="middle">registers</text><text class="t-tick" x="368.0" y="150.0" text-anchor="middle">shared memory</text><text class="t-tick" x="368.0" y="164.0" text-anchor="middle">(on-chip SRAM)</text><rect class="box-1" x="426.0" y="50.0" width="96.0" height="120.0" rx="8"/><text class="t-note" x="474.0" y="68.0" text-anchor="middle">SM 1</text><rect class="box-3" x="436.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="456.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="476.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="496.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="436.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="456.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="476.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="496.0" y="98.0" width="15.0" height="15.0" rx="2"/><text class="t-tick" x="474.0" y="134.0" text-anchor="middle">registers</text><text class="t-tick" x="474.0" y="150.0" text-anchor="middle">shared memory</text><text class="t-tick" x="474.0" y="164.0" text-anchor="middle">(on-chip SRAM)</text><rect class="box-1" x="532.0" y="50.0" width="96.0" height="120.0" rx="8"/><text class="t-note" x="580.0" y="68.0" text-anchor="middle">SM 2</text><rect class="box-3" x="542.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="562.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="582.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="602.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="542.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="562.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="582.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="602.0" y="98.0" width="15.0" height="15.0" rx="2"/><text class="t-tick" x="580.0" y="134.0" text-anchor="middle">registers</text><text class="t-tick" x="580.0" y="150.0" text-anchor="middle">shared memory</text><text class="t-tick" x="580.0" y="164.0" text-anchor="middle">(on-chip SRAM)</text><rect class="box-1" x="638.0" y="50.0" width="96.0" height="120.0" rx="8"/><text class="t-note" x="686.0" y="68.0" text-anchor="middle">SM 131</text><rect class="box-3" x="648.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="668.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="688.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="708.0" y="78.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="648.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="668.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="688.0" y="98.0" width="15.0" height="15.0" rx="2"/><rect class="box-3" x="708.0" y="98.0" width="15.0" height="15.0" rx="2"/><text class="t-tick" x="686.0" y="134.0" text-anchor="middle">registers</text><text class="t-tick" x="686.0" y="150.0" text-anchor="middle">shared memory</text><text class="t-tick" x="686.0" y="164.0" text-anchor="middle">(on-chip SRAM)</text><text class="t-big" x="638.0" y="116.0" text-anchor="middle">...</text><rect class="box-4" x="320.0" y="182.0" width="414.0" height="26.0" rx="6"/><text class="t-note" x="527.0" y="200.0" text-anchor="middle">L2 cache, shared by all SMs</text><path class="path" d="M420,276 L420,224" marker-end="url(#ah-on)"/><text class="t-tick" x="428.0" y="262.0" text-anchor="start">3. read W and x</text><path class="path" d="M640,224 L640,276" marker-end="url(#ah-on)"/><text class="t-tick" x="632.0" y="262.0" text-anchor="end">4. write y back</text><rect class="box-2" x="305.0" y="280.0" width="440.0" height="46.0" rx="8"/><text class="t-note" x="525.0" y="300.0" text-anchor="middle">HBM: the GPU's main memory (the weights live here)</text><text class="t-tick" x="525.0" y="318.0" text-anchor="middle">H100: 80 GB at 3.35 TB/s</text><text class="t-tick" x="20.0" y="222.0" text-anchor="start">The CPU only gives orders.</text><text class="t-tick" x="20.0" y="240.0" text-anchor="start">All the arithmetic happens</text><text class="t-tick" x="20.0" y="258.0" text-anchor="start">on the GPU.</text><rect class="box-3" x="20.0" y="286.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="40.0" y="296.0" text-anchor="start">one block of threads</text><text class="t-tick" x="20.0" y="316.0" text-anchor="start">(many threads, run in warps of 32)</text></svg><figcaption>One line of PyTorch on an NVIDIA GPU. The CPU launches a kernel. Its blocks of threads are spread over the streaming multiprocessors (SMs). The threads read W and x from HBM through the L2 cache into their own fast on-chip memory, do the multiply-adds, and write y back to HBM.</figcaption></figure>

1. **The CPU picks a kernel.** PyTorch sees a matrix multiplication of two GPU tensors. It chooses a matrix-multiply kernel from NVIDIA's libraries, sized for these shapes.
2. **The CPU launches it.** It sends the GPU the kernel's code, the memory addresses of `x`, `W` and `y`, and how many **threads** to start. Then the CPU moves on at once. It does not wait for the answer.
3. **The GPU spreads the work.** The threads come in groups called **thread blocks**. The GPU hands the blocks out to its **streaming multiprocessors**. An H100 has 132 of them. Each block computes one tile of `y`.
4. **Each block reads, computes and writes.** Its threads copy their part of `W` and `x` from HBM into fast memory on the chip. They do the multiply-adds there, and they write their tile of `y` back to HBM.

> [!DEFINITION] Thread, warp and thread block
> A **thread** is one small sequence of instructions that runs on the GPU, usually working on a few numbers. The GPU runs threads in groups of 32 that move in step, called **warps**. A **thread block** is a group of threads, often a few hundred, that run on the same SM and can share data.

> [!DEFINITION] Streaming multiprocessor (SM)
> One of the GPU's processor units. Each SM has its own arithmetic units, including Tensor Cores for matrix multiplication, its own registers and its own small, fast shared memory. An H100 SXM has 132 SMs. The "compute cores" of this page live inside the SMs.

The CUDA programming guide says it in two sentences. A kernel, "when called", is "executed N times in parallel by N different CUDA threads". And when the CPU launches it, "the blocks of the grid are enumerated and distributed to multiprocessors with available execution capacity".

### The memory ladder

Step 4 hides an important detail. A GPU has several kinds of memory, and they differ in size by a factor of thousands. The kitchen's "small counter" from earlier is really a ladder of memories.

<figure class="fig"><svg viewBox="0 0 760 316" role="img" aria-label="The memory levels of an H100 drawn on a log scale. The on-chip memories add up to tens of megabytes; only HBM can hold a model of 16 GB."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">The memories of an H100, from the smallest and fastest to the largest (log scale)</text><text class="t-note" x="218.0" y="61.0" text-anchor="end">registers</text><text class="t-tick" x="218.0" y="77.0" text-anchor="end">256 KB per SM</text><g class="mark"><title>registers: 256 KB per SM</title><path class="s1 bar-mark" d="M230,44H326.9017171701682Q330.9017171701682,44 330.9017171701682,48V68Q330.9017171701682,72 326.9017171701682,72H230Z"/></g><text class="t-tick" x="338.9" y="63.0" text-anchor="start">34 MB in all, inside each SM, the fastest</text><text class="t-note" x="218.0" y="113.0" text-anchor="end">shared memory + L1</text><text class="t-tick" x="218.0" y="129.0" text-anchor="end">256 KB per SM</text><g class="mark"><title>shared memory + L1: 256 KB per SM</title><path class="s1 bar-mark" d="M230,96H326.9017171701682Q330.9017171701682,96 330.9017171701682,100V120Q330.9017171701682,124 326.9017171701682,124H230Z"/></g><text class="t-tick" x="338.9" y="115.0" text-anchor="start">34 MB in all, inside each SM</text><text class="t-note" x="218.0" y="165.0" text-anchor="end">L2 cache</text><text class="t-tick" x="218.0" y="181.0" text-anchor="end">50 MB</text><g class="mark"><title>L2 cache: 50 MB</title><path class="s1 bar-mark" d="M230,148H338.13202028617724Q342.13202028617724,148 342.13202028617724,152V172Q342.13202028617724,176 338.13202028617724,176H230Z"/></g><text class="t-tick" x="350.1" y="167.0" text-anchor="start">on the chip, shared by all 132 SMs</text><text class="t-note" x="218.0" y="217.0" text-anchor="end">HBM</text><text class="t-tick" x="218.0" y="233.0" text-anchor="end">80 GB</text><g class="mark"><title>HBM: 80 GB</title><path class="s2 bar-mark" d="M230,200H549.6039391414683Q553.6039391414683,200 553.6039391414683,204V224Q553.6039391414683,228 549.6039391414683,228H230Z"/></g><text class="t-tick" x="561.6" y="219.0" text-anchor="start">beside the chip, 3.35 TB/s</text><line class="drop" x1="507.6" y1="192.0" x2="507.6" y2="246.0"/><text class="t-tick" x="501.6" y="262.0" text-anchor="end">Llama 3.1 8B weights: 16.1 GB</text><text class="t-tick" x="230.0" y="282.0" text-anchor="middle">1 MB</text><text class="t-tick" x="362.0" y="282.0" text-anchor="middle">100 MB</text><text class="t-tick" x="494.0" y="282.0" text-anchor="middle">10 GB</text><text class="t-tick" x="560.0" y="282.0" text-anchor="middle">100 GB</text><text class="t-tick" x="20.0" y="304.0" text-anchor="start">Only HBM is big enough for the model, so every decode step must bring the weights from HBM again.</text></svg><figcaption>The memories of an H100 on a log scale. Registers and shared memory sit inside each SM and add up to tens of megabytes. The L2 cache is 50 MB. HBM, 80 GB, is the only level big enough to hold a model.</figcaption></figure>

| Memory | Where it is | Size on an H100 SXM |
|---|---|---|
| registers | inside each SM | 256 KB per SM |
| shared memory and L1 cache | inside each SM | 256 KB per SM, together |
| L2 cache | on the chip, shared by all SMs | 50 MB |
| HBM | memory chips beside the GPU | 80 GB, at 3.35 TB/s |

> [!DEFINITION] SRAM and cache
> **SRAM** is the fast memory built into the chip itself, such as registers and shared memory. A **cache** is SRAM that keeps copies of recently used data from HBM, so a second read of the same data is faster.

The on-chip memories are much faster than HBM, but they are tiny. All 132 SMs together have about 34 MB of registers and about 34 MB of shared memory. Llama 3.1 8B needs 16.1 GB. So the weights cannot stay on the chip between steps. Every decode step must pull them from HBM again, at 3.35 TB/s, which is the floor this page measured.

### Launch overhead: why small steps are slow

Now back to the foreman. A launch takes the CPU a few microseconds. The work of a kernel at batch 1 is often smaller than that. Then the GPU finishes a kernel and waits for the next order.

> [!DEFINITION] Launch overhead
> The fixed time it takes to start one kernel: the CPU work in PyTorch and the driver, plus the GPU's start-up of the kernel. It is paid once per kernel, whatever the size of the work.

We counted the operations in one decode step of our model. A step at batch 1 runs **880 GPU operations**, about 37 per layer. Then we timed a tiny operation, an add on 16 numbers, repeated 2,000 times. Each one took **3.4 microseconds**, with almost no arithmetic inside.

So the launches alone cost about 880 × 3.4 µs = **3.0 ms** per step. Add the 3.8 ms floor for reading the weights, and about 6.8 ms of the 10 ms step is explained. The rest is time that these two numbers do not cover. Some kernels cost more than a tiny add to start, and each one does some work too.

<figure class="fig"><svg viewBox="0 0 760 262" role="img" aria-label="Two timelines of one decode step. Without a CUDA graph the CPU launches every kernel in turn and the GPU waits between them; with a graph one launch replays all of them back to back."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">without a CUDA graph: one launch per kernel</text><text class="t-note" x="110.0" y="54.0" text-anchor="end">CPU</text><text class="t-note" x="110.0" y="90.0" text-anchor="end">GPU</text><rect class="box-2" x="124.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="142.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="174.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="192.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="224.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="242.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="274.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="292.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="324.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="342.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="374.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="392.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="424.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="442.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="474.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="492.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="524.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="542.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="574.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="592.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="624.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="642.0" y="76.0" width="34.0" height="22.0" rx="3"/><rect class="box-2" x="674.0" y="40.0" width="12.0" height="22.0" rx="2"/><rect class="box-1" x="692.0" y="76.0" width="34.0" height="22.0" rx="3"/><text class="t-tick" x="730.0" y="91.0" text-anchor="start">gaps</text><text class="t-title" x="20.0" y="124.0" text-anchor="start">with a CUDA graph: one launch replays the whole recorded step</text><text class="t-note" x="110.0" y="154.0" text-anchor="end">CPU</text><text class="t-note" x="110.0" y="190.0" text-anchor="end">GPU</text><rect class="box-2" x="124.0" y="140.0" width="60.0" height="22.0" rx="4"/><text class="t-tick" x="154.0" y="155.0" text-anchor="middle">replay</text><rect class="box-1" x="190.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="226.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="262.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="298.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="334.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="370.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="406.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="442.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="478.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="514.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="550.0" y="176.0" width="34.0" height="22.0" rx="3"/><rect class="box-1" x="586.0" y="176.0" width="34.0" height="22.0" rx="3"/><text class="t-tick" x="630.0" y="191.0" text-anchor="start">kernels back to back</text><rect class="box-2" x="20.0" y="214.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="38.0" y="224.0" text-anchor="start">CPU launch work</text><rect class="box-1" x="170.0" y="214.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="188.0" y="224.0" text-anchor="start">a kernel running on the GPU</text><text class="t-tick" x="20.0" y="248.0" text-anchor="start">Small kernels finish faster than the CPU can launch the next one, so the GPU waits in the gaps.</text></svg><figcaption>Two timelines of one decode step. Without a CUDA graph, the CPU launches each kernel in turn, and the GPU waits in the gaps. With a CUDA graph, one launch replays the whole recorded step, and the kernels run back to back.</figcaption></figure>

This cost does not grow with the batch. A batch of 64 runs the same 880 operations, each on more numbers. That is part of the "fixed part" of the step that you will meet in the batching section below.

> [!NOTE] Measured on Apple's GPU, not on CUDA
> Our measurements ran on the GPU of an Apple M5 Pro, through Apple's Metal framework (PyTorch's `mps` device), not through CUDA. Apple's GPU has its own names: GPU cores instead of SMs, for example. But the ideas are the same on both. There is one launch per operation and a fixed cost per launch. There is small, fast memory on the chip and big, slow memory beside it. The scripts run on an NVIDIA GPU if you change `mps` to `cuda`.

### Two fixes: CUDA graphs and kernel fusion

**CUDA graphs** remove most of the launch cost. The engine runs one decode step once while CUDA records every kernel and its inputs. That record is a **graph**.

From then on, the CPU launches the whole graph with one call, and the GPU runs the kernels back to back. A graph works only for fixed shapes, so engines record one graph for each of many batch sizes. By default, vLLM records full CUDA graphs for decode steps. For prefill and mixed steps, it uses graphs for the parts between the attention calls.

> [!DEFINITION] CUDA graph
> A recording of a fixed sequence of kernels and their inputs that the CPU can launch again with one call. It removes the launch cost of every kernel after the first, but it only fits the shapes it was recorded with.

**Kernel fusion** removes trips to HBM. A layer has many small steps, such as a multiply, an add and a normalisation. Each one reads its input from HBM and writes its output back. A fused kernel does all of them in one pass. The numbers in between stay in registers and never go back to HBM. It also means fewer kernels, so fewer launches.

<figure class="fig"><svg viewBox="0 0 760 198" role="img" aria-label="Kernel fusion. Three small kernels each read their input from HBM and write their output back; one fused kernel does all three steps with one read and one write."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">three small kernels</text><rect class="box-1" x="40.0" y="40.0" width="96.0" height="34.0" rx="6"/><text class="t-note" x="88.0" y="62.0" text-anchor="middle">multiply</text><line class="edge" x1="70.0" y1="120.0" x2="70.0" y2="78.0" marker-end="url(#ah)"/><line class="path" x1="106.0" y1="78.0" x2="106.0" y2="120.0" marker-end="url(#ah-on)"/><rect class="box-1" x="160.0" y="40.0" width="96.0" height="34.0" rx="6"/><text class="t-note" x="208.0" y="62.0" text-anchor="middle">add</text><line class="edge" x1="190.0" y1="120.0" x2="190.0" y2="78.0" marker-end="url(#ah)"/><line class="path" x1="226.0" y1="78.0" x2="226.0" y2="120.0" marker-end="url(#ah-on)"/><rect class="box-1" x="280.0" y="40.0" width="96.0" height="34.0" rx="6"/><text class="t-note" x="328.0" y="62.0" text-anchor="middle">normalise</text><line class="edge" x1="310.0" y1="120.0" x2="310.0" y2="78.0" marker-end="url(#ah)"/><line class="path" x1="346.0" y1="78.0" x2="346.0" y2="120.0" marker-end="url(#ah-on)"/><rect class="box-2" x="20.0" y="124.0" width="380.0" height="30.0" rx="6"/><text class="t-note" x="210.0" y="144.0" text-anchor="middle">HBM: 3 reads + 3 writes</text><text class="t-title" x="440.0" y="22.0" text-anchor="start">one fused kernel</text><rect class="box-on" x="460.0" y="40.0" width="260.0" height="34.0" rx="6"/><text class="t-note" x="590.0" y="62.0" text-anchor="middle">multiply, add, normalise</text><line class="edge" x1="540.0" y1="120.0" x2="540.0" y2="78.0" marker-end="url(#ah)"/><line class="path" x1="640.0" y1="78.0" x2="640.0" y2="120.0" marker-end="url(#ah-on)"/><rect class="box-2" x="440.0" y="124.0" width="300.0" height="30.0" rx="6"/><text class="t-note" x="590.0" y="144.0" text-anchor="middle">HBM: 1 read + 1 write</text><text class="t-tick" x="20.0" y="184.0" text-anchor="start">The fused kernel keeps the in-between numbers in registers, so they never make the trip to HBM.</text></svg><figcaption>Kernel fusion. Three small kernels make three reads and three writes to HBM. One fused kernel does the same three steps with one read and one write, because the in-between numbers stay in registers.</figcaption></figure>

> [!DEFINITION] Kernel fusion
> Joining several GPU operations into one kernel, so the data is read from memory once and the results in between stay on the chip.

Neither fix changes the floor. The weights must still come from HBM once per step. The fixes only cut the time on top of the floor. The rest of this page is about the floor itself, and about the one trick that spreads it over many users.

## Counting work per byte

The 3 × 3 example gives us a number to put on each job. Engineers call it **arithmetic intensity**.

> [!DEFINITION] Arithmetic intensity
> The number of FLOPs a job does for each byte it reads from memory. Low intensity means the job waits for memory. High intensity means it keeps the compute cores busy.

$$
\text{arithmetic intensity} = \frac{\text{FLOPs done}}{\text{bytes read from memory}}
$$

Read it left to right: count the arithmetic, count the bytes that had to come from memory, and divide. For a weight matrix the rule is short. Each weight costs 2 bytes and gives 2 FLOPs (one multiply-add) per token. So the intensity is about **1 FLOP per byte for each token** that uses the weights in the same pass.

Now a real matrix from our model. Each layer has matrices of 896 × 896 weights, which is 802,816 weights or 1.61 MB. One token through it is 1.61 million FLOPs: 1 FLOP per byte. Four tokens give 6.42 million FLOPs for the same 1.61 MB: 4 FLOPs per byte. A 512-token prefill gives 512 FLOPs per byte.

| Job | FLOPs per byte (weights) |
|---|---|
| decode, one sequence | about 1 |
| prefill, 512 tokens | about 512 |

Compare those with the H100's balance point of about 295, from the GPU section. Below 295 FLOPs per byte, memory cannot deliver bytes fast enough to keep the cores busy. Above it, memory is fast enough and the cores become the limit. Decode at 1 is far below the line. Prefill at 512 is above it.

## The roofline: one picture for both limits

A **roofline** turns those two limits into one picture. It answers a single question: for a job with a given work per byte, what is the most speed this GPU can give it?

> [!DEFINITION] Roofline
> A chart of the highest FLOP/s a processor can reach, plotted against arithmetic intensity. It has a sloped part set by memory bandwidth and a flat part set by peak compute.

Build it in three steps.

1. **The memory limit.** If memory delivers 3.35 trillion bytes per second, and each byte gives you $$I$$ FLOPs, you cannot do more than 3.35 × $$I$$ trillion FLOP/s. More work per byte means more speed, so this limit is a line that rises with $$I$$.
2. **The compute limit.** However many FLOPs per byte you have, the cores cannot go faster than 989 TFLOP/s. That is a flat line, the roof.
3. **The roofline.** The real limit is the lower of the two. It rises along the slope, then flattens at the roof. The bend, where the two lines meet, is the **ridge point**: 989 ÷ 3.35 ≈ 295 FLOPs per byte.

<figure class="fig"><svg viewBox="0 0 760 284" role="img" aria-label="The roofline built in three steps for an H100: a sloped memory limit, a flat compute limit, and the lower of the two."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="12.0" y="22.0" text-anchor="start">1. memory limit</text><line class="grid" x1="52.0" y1="206.0" x2="242.0" y2="206.0"/><text class="t-tick" x="44.0" y="210.0" text-anchor="end">1</text><line class="grid" x1="52.0" y1="156.0" x2="242.0" y2="156.0"/><text class="t-tick" x="44.0" y="160.0" text-anchor="end">10</text><line class="grid" x1="52.0" y1="106.0" x2="242.0" y2="106.0"/><text class="t-tick" x="44.0" y="110.0" text-anchor="end">100</text><line class="grid" x1="52.0" y1="56.0" x2="242.0" y2="56.0"/><text class="t-tick" x="44.0" y="60.0" text-anchor="end">1,000</text><text class="t-tick" x="52.0" y="224.0" text-anchor="middle">1</text><text class="t-tick" x="106.3" y="224.0" text-anchor="middle">10</text><text class="t-tick" x="160.6" y="224.0" text-anchor="middle">100</text><text class="t-tick" x="214.9" y="224.0" text-anchor="middle">1,000</text><line class="axis" x1="52.0" y1="206.0" x2="242.0" y2="206.0"/><polyline class="l2" points="52.0,179.7 240.8,47.2"/><text class="t-tick" x="242.0" y="166.0" text-anchor="end">speed = bandwidth</text><text class="t-tick" x="242.0" y="182.0" text-anchor="end">× FLOPs per byte</text><text class="t-tick" x="147.0" y="244.0" text-anchor="middle">FLOPs per byte</text><text class="t-title" x="257.0" y="22.0" text-anchor="start">2. compute limit</text><line class="grid" x1="297.0" y1="206.0" x2="487.0" y2="206.0"/><text class="t-tick" x="289.0" y="210.0" text-anchor="end">1</text><line class="grid" x1="297.0" y1="156.0" x2="487.0" y2="156.0"/><text class="t-tick" x="289.0" y="160.0" text-anchor="end">10</text><line class="grid" x1="297.0" y1="106.0" x2="487.0" y2="106.0"/><text class="t-tick" x="289.0" y="110.0" text-anchor="end">100</text><line class="grid" x1="297.0" y1="56.0" x2="487.0" y2="56.0"/><text class="t-tick" x="289.0" y="60.0" text-anchor="end">1,000</text><text class="t-tick" x="297.0" y="224.0" text-anchor="middle">1</text><text class="t-tick" x="351.3" y="224.0" text-anchor="middle">10</text><text class="t-tick" x="405.6" y="224.0" text-anchor="middle">100</text><text class="t-tick" x="459.9" y="224.0" text-anchor="middle">1,000</text><line class="axis" x1="297.0" y1="206.0" x2="487.0" y2="206.0"/><polyline class="l3" points="297.0,56.2 485.8,56.2"/><text class="t-tick" x="303.0" y="74.2" text-anchor="start">the cores cannot go faster</text><text class="t-tick" x="303.0" y="90.2" text-anchor="start">than 989 TFLOP/s</text><text class="t-tick" x="392.0" y="244.0" text-anchor="middle">FLOPs per byte</text><text class="t-title" x="502.0" y="22.0" text-anchor="start">3. the roofline: the lower of the two</text><line class="grid" x1="542.0" y1="206.0" x2="732.0" y2="206.0"/><text class="t-tick" x="534.0" y="210.0" text-anchor="end">1</text><line class="grid" x1="542.0" y1="156.0" x2="732.0" y2="156.0"/><text class="t-tick" x="534.0" y="160.0" text-anchor="end">10</text><line class="grid" x1="542.0" y1="106.0" x2="732.0" y2="106.0"/><text class="t-tick" x="534.0" y="110.0" text-anchor="end">100</text><line class="grid" x1="542.0" y1="56.0" x2="732.0" y2="56.0"/><text class="t-tick" x="534.0" y="60.0" text-anchor="end">1,000</text><text class="t-tick" x="542.0" y="224.0" text-anchor="middle">1</text><text class="t-tick" x="596.3" y="224.0" text-anchor="middle">10</text><text class="t-tick" x="650.6" y="224.0" text-anchor="middle">100</text><text class="t-tick" x="704.9" y="224.0" text-anchor="middle">1,000</text><line class="axis" x1="542.0" y1="206.0" x2="732.0" y2="206.0"/><polyline class="l1" points="542.0,179.7 676.1,56.2"/><polyline class="l1" points="676.1,56.2 730.8,56.2"/><line class="drop" x1="676.1" y1="56.2" x2="676.1" y2="206.0"/><text class="t-tick" x="680.1" y="198.0" text-anchor="start">ridge 295</text><text class="t-note" x="567.9" y="117.1" text-anchor="start">memory-bound</text><text class="t-note" x="685.3" y="74.2" text-anchor="start">compute-</text><text class="t-note" x="685.3" y="90.2" text-anchor="start">bound</text><text class="t-tick" x="637.0" y="244.0" text-anchor="middle">FLOPs per byte</text><text class="t-tick" x="12.0" y="270.0" text-anchor="start">Vertical axes: TFLOP/s the GPU can reach. Both axes use a log scale. Numbers: an NVIDIA H100 (BF16).</text></svg><figcaption>The roofline built in three steps for an H100: a sloped memory limit, a flat compute limit, and the lower of the two, which bends at the ridge point.</figcaption></figure>

$$
\text{highest speed} = \min\left(\text{peak FLOP/s},\ \text{bandwidth} \times I\right)
$$

Read it left to right. The speed a job can reach is the smaller of two numbers: the roof, and the bandwidth times the work per byte. A worked example for decode at $$I = 1$$ on an H100: 3.35 TB/s × 1 = 3.35 TFLOP/s, and the roof is 989. The smaller one is 3.35 TFLOP/s, which is **0.3%** of what the GPU can compute. For prefill at $$I = 512$$: 3.35 × 512 = 1,715, which is above the roof, so the limit is the full 989 TFLOP/s.

<figure class="fig"><svg viewBox="0 0 760 340" role="img" aria-label="Roofline for an H100: decode at batch 1 sits at the bottom of the memory slope; prefill sits on the compute roof."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="280.0" x2="730.0" y2="280.0"/><text class="t-tick" x="62.0" y="284.0" text-anchor="end">1</text><line class="grid" x1="70.0" y1="201.9" x2="730.0" y2="201.9"/><text class="t-tick" x="62.0" y="205.9" text-anchor="end">10</text><line class="grid" x1="70.0" y1="123.8" x2="730.0" y2="123.8"/><text class="t-tick" x="62.0" y="127.8" text-anchor="end">100</text><line class="grid" x1="70.0" y1="45.6" x2="730.0" y2="45.6"/><text class="t-tick" x="62.0" y="49.6" text-anchor="end">1,000</text><text class="t-tick" x="70.0" y="298.0" text-anchor="middle">1</text><text class="t-tick" x="180.0" y="298.0" text-anchor="middle">4</text><text class="t-tick" x="290.0" y="298.0" text-anchor="middle">16</text><text class="t-tick" x="400.0" y="298.0" text-anchor="middle">64</text><text class="t-tick" x="521.3" y="298.0" text-anchor="middle">295</text><text class="t-tick" x="620.0" y="298.0" text-anchor="middle">1,024</text><text class="t-tick" x="730.0" y="298.0" text-anchor="middle">4,096</text><line class="axis" x1="70.0" y1="280.0" x2="730.0" y2="280.0"/><polyline class="l1" points="70.0,239.0 521.3,46.0 730.0,46.0"/><line class="drop" x1="521.3" y1="46.0" x2="521.3" y2="280.0"/><text class="t-tick" x="730.0" y="36.0" text-anchor="end">compute roof: 989 TFLOP/s</text><text class="t-tick" x="212.2" y="200.2" text-anchor="start" transform="rotate(-23.2 212.2 200.2)">memory slope: 3.35 TB/s × FLOPs per byte</text><g class="mark"><title>decode, batch 1: about 1 FLOPs per byte, at most 3 TFLOP/s (0.3% of the roof)</title><circle class="s2 ring" cx="70.0" cy="239.0" r="6"/></g><text class="t-note" x="80.0" y="227.0" text-anchor="start">decode, batch 1</text><g class="mark"><title>prefill, 512 tokens: about 512 FLOPs per byte, at most 989 TFLOP/s (100.0% of the roof)</title><circle class="s1 ring" cx="565.0" cy="46.0" r="6"/></g><text class="t-note" x="575.0" y="72.0" text-anchor="start">prefill, 512 tokens</text><text class="t-tick" x="400.0" y="324.0" text-anchor="middle">FLOPs per byte read from memory (log scale)</text><text class="t-tick" x="10.0" y="16.0" text-anchor="start">TFLOP/s</text></svg><figcaption>A roofline for an H100. On the left slope, speed is capped by memory. On the flat roof, it is capped by compute. Decode at batch 1 sits at the bottom left; prefill sits on the roof.</figcaption></figure>

The roofline gives only a ceiling. Real jobs sit below it, because of the fixed costs per step that we saw above. Our M5 Pro shows the same pattern in what it really did:

| Job (measured on the M5 Pro) | FLOPs per byte | Real speed |
|---|---|---|
| decode, one sequence, 10.0 ms per step | 1 | 0.10 TFLOP/s |
| prefill, 512 tokens in 27.4 ms | about 371 | 13.4 TFLOP/s |

Prefill did over 100 times more arithmetic per second on the same chip. (The prefill intensity here is 371, not 512, because prefill only needs the large output layer for the last token. The script in the research notes counts both.) That gap is the 190x gap from the measurement above, seen as a picture.

<details>
<summary>Research notes: how the roofline numbers were counted</summary>

The FLOP counts use the usual rule of 2 FLOPs per weight per token. Our model has 494,032,768 weights. Of these, 136,134,656 are the token embedding, which the model also uses as its output layer. A decode step uses every weight once per token, so it does 0.988 GFLOP per token. Prefill keeps the output only for the last prompt token, so its other tokens skip the output layer: 0.716 GFLOP per token. The counts leave out attention, which is small at these lengths.

The H100 numbers come from NVIDIA's specification: 3.35 TB/s of memory bandwidth and 1,979 TFLOP/s of BF16 with sparsity. For the dense matrices that LLMs use, that is about 989 TFLOP/s. The M5 Pro's bandwidth, 262 GB/s, was measured for this series. The script `p1_roofline.py` prints every number in this section.

</details>

Prompt length changes the prefill side too. Here is prefill on our model for prompts from 16 to 8,192 tokens.

| Prompt length | Prefill time | Tokens per second |
|---|---|---|
| 16 | 10.8 ms | 1,484 |
| 256 | 17.7 ms | 14,483 |
| 1,024 | 47.9 ms | 21,371 |
| 2,048 | 95.0 ms | 21,551 |
| 8,192 | 450.5 ms | 18,185 |

Short prompts are slow per token. A 16-token prefill takes about as long as one decode step, because the fixed cost per step is larger than the work. As the prompt grows, the GPU gets enough work per weight to stay busy, and the rate climbs to about 21,500 tokens per second. At 8,192 tokens it dips a little.

Most of that dip comes from attention, the step where every token looks at every earlier token. That part of the work grows with the square of the length. Part 2 explains attention.

A paper on scheduling these two phases, Sarathi-Serve, states the same split in one sentence and draws it from its own measurements.

> [!PAPER] Agrawal et al. (2024), Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve · §1 · page 2
> [![The introduction's paragraph on the two phases, with The prefill phase is compute-bound, the decode phase is memory-bound and decodes benefit significantly from batching highlighted](https://ishwarj.com/img/writings/llm-inference/p1-sarathi-phases.png-sarathi-phases.png)](/img/writings/llm-inference/p1-sarathi-phases.png)
>
> **What it says:** "The prefill phase is compute-bound" because it handles all prompt tokens in parallel, while "the decode phase is memory-bound" because it handles one token per request at a time. "Therefore, decodes benefit significantly from batching".
>
> **What it means:**
>
> This is the roofline in words. Prefill is already on the roof, so giving it more work in the same pass gains little. Decode sits low on the slope, so anything that adds work per byte moves it up the slope.
>
> The last highlight names that thing: batching. The rest of this page explains what it is, why it works, and where it stops.

> [!PAPER] Agrawal et al. (2024), Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve · Figure 3 · page 5
> [![Figure 3 of Sarathi-Serve: prefill and decode throughput against batch size for Mistral-7B on one A100, with prefills are much more efficient than decode and batching boosts decode throughput almost linearly highlighted](https://ishwarj.com/img/writings/llm-inference/p1-sarathi-fig3.pngp1-sarathi-fig3.png)](/img/writings/llm-inference/p1-sarathi-fig3.png)
>
> **What it says:** for Mistral-7B on one A100 GPU, "prefills are much more efficient than decode", and "batching boosts decode throughput almost linearly but has a marginal effect on prefill throughput".
>
> **What it means:**
>
> Look at the two y-axes first. The prefill panel counts in thousands of tokens per second; the decode panel counts in hundreds. That is our 190x gap again, on a different model and a different GPU.
>
> Then look at the shapes. Doubling the decode batch roughly doubles decode's tokens per second. Adding prefills to a batch does almost nothing, because prefill is already on the roof. Our own measurements below show the same two shapes.

## The fix: write for many conversations at once

Decode wastes each trip to memory on a single token. The fix follows from the 3 × 3 example: put more tokens into each trip. A real server has many users at the same time, and each of them needs a next token. So instead of one decode step per user, run **one step for all of them together**.

> [!DEFINITION] Batch
> A group of sequences that the GPU processes in the same step. In decode, a batch of 64 means one forward pass of the model produces the next token for 64 different conversations.

> [!DEFINITION] Throughput
> How much useful output the whole system produces per second, here counted in tokens per second across all users.

### The bus

Think of a bus between two towns. The trip costs the same fuel and the same 10 minutes whether one passenger rides or forty. With one passenger, almost all of that cost is wasted. With forty, the same trip carries forty people.

<figure class="fig"><svg viewBox="0 0 760 222" role="img" aria-label="A bus trip costs about the same with one passenger or four. One read of the weights is the trip; each sequence is a passenger."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="30.0" y="36.0" text-anchor="start">one passenger</text><rect class="box" x="30.0" y="50.0" width="300.0" height="92.0" rx="18"/><rect class="box-1" x="48.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="s2" cx="65.0" cy="81.0" r="8"/><rect class="box-ghost" x="92.0" y="66.0" width="34.0" height="30.0" rx="5"/><rect class="box-ghost" x="136.0" y="66.0" width="34.0" height="30.0" rx="5"/><rect class="box-ghost" x="180.0" y="66.0" width="34.0" height="30.0" rx="5"/><rect class="box-ghost" x="224.0" y="66.0" width="34.0" height="30.0" rx="5"/><rect class="box-ghost" x="268.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="box" cx="90" cy="142" r="14"/><circle class="box" cx="270" cy="142" r="14"/><text class="t-note" x="180.0" y="124.0" text-anchor="middle">one trip: 10.0 ms</text><text class="t-title" x="420.0" y="36.0" text-anchor="start">four passengers</text><rect class="box" x="420.0" y="50.0" width="300.0" height="92.0" rx="18"/><rect class="box-1" x="438.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="s2" cx="455.0" cy="81.0" r="8"/><rect class="box-1" x="482.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="s2" cx="499.0" cy="81.0" r="8"/><rect class="box-1" x="526.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="s2" cx="543.0" cy="81.0" r="8"/><rect class="box-1" x="570.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="s2" cx="587.0" cy="81.0" r="8"/><rect class="box-ghost" x="614.0" y="66.0" width="34.0" height="30.0" rx="5"/><rect class="box-ghost" x="658.0" y="66.0" width="34.0" height="30.0" rx="5"/><circle class="box" cx="480" cy="142" r="14"/><circle class="box" cx="660" cy="142" r="14"/><text class="t-note" x="570.0" y="124.0" text-anchor="middle">one trip: 11.1 ms</text><text class="t-tick" x="30.0" y="190.0" text-anchor="start">the bus = one read of all the weights</text><text class="t-tick" x="30.0" y="208.0" text-anchor="start">a passenger = one sequence that needs its next token</text><text class="t-title" x="420.0" y="190.0" text-anchor="start">1 token per trip: 100 tokens/s</text><text class="t-title" x="420.0" y="208.0" text-anchor="start">4 tokens per trip: 361 tokens/s</text></svg><figcaption>A bus trip costs about the same with one passenger or four. One read of the weights is the trip; each sequence that needs a next token is a passenger.</figcaption></figure>

In decode, the trip is one read of all the weights from memory. The passengers are the sequences in the batch. The read costs about the same whether it serves one sequence or four, because, as we saw, the arithmetic is the cheap part.

### One decode step, batch 1 and batch 4

Here is what happens inside one step when four users share it. Each user is at a different point in a different conversation.

<figure class="fig"><svg viewBox="0 0 760 366" role="img" aria-label="One decode step for one sequence and for four. With four, the newest token of each is stacked, the weights are read once, and four next tokens come out."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">batch 1</text><text class="t-tick" x="20.0" y="42.0" text-anchor="start">10.0 ms</text><rect class="box" x="100.0" y="60.0" width="150.0" height="30.0" rx="6"/><text class="t-tick" x="175.0" y="80.0" text-anchor="middle">A: The capital is</text><line class="edge" x1="254.0" y1="75.0" x2="300.0" y2="75.0" marker-end="url(#ah)"/><rect class="box-2" x="304.0" y="54.0" width="170.0" height="42.0" rx="8"/><text class="t-title" x="389.0" y="72.0" text-anchor="middle">read all weights</text><text class="t-tick" x="389.0" y="88.0" text-anchor="middle">0.99 GB, once</text><line class="edge" x1="478.0" y1="75.0" x2="524.0" y2="75.0" marker-end="url(#ah)"/><rect class="box-3" x="528.0" y="60.0" width="90.0" height="30.0" rx="6"/><text class="t-big" x="573.0" y="81.0" text-anchor="middle">Paris</text><text class="t-note" x="630.0" y="80.0" text-anchor="start">1 token out</text><line class="drop" x1="20.0" y1="116.0" x2="740.0" y2="116.0"/><text class="t-title" x="20.0" y="146.0" text-anchor="start">batch 4</text><text class="t-tick" x="20.0" y="164.0" text-anchor="start">11.1 ms</text><text class="t-note" x="175.0" y="146.0" text-anchor="middle">1. stack the newest token</text><text class="t-note" x="175.0" y="162.0" text-anchor="middle">of each sequence</text><text class="t-note" x="389.0" y="146.0" text-anchor="middle">2. read each weight once,</text><text class="t-note" x="389.0" y="162.0" text-anchor="middle">use it for all four</text><text class="t-note" x="600.0" y="146.0" text-anchor="middle">3. each sequence gets</text><text class="t-note" x="600.0" y="162.0" text-anchor="middle">its own next token</text><rect class="box" x="100.0" y="180.0" width="150.0" height="30.0" rx="6"/><text class="t-tick" x="175.0" y="200.0" text-anchor="middle">A: The capital is</text><line class="edge" x1="254.0" y1="195.0" x2="300.0" y2="252.0" marker-end="url(#ah)"/><line class="edge" x1="478.0" y1="252.0" x2="524.0" y2="195.0" marker-end="url(#ah)"/><rect class="box-3" x="528.0" y="180.0" width="90.0" height="30.0" rx="6"/><text class="t-big" x="573.0" y="201.0" text-anchor="middle">Paris</text><rect class="box" x="100.0" y="218.0" width="150.0" height="30.0" rx="6"/><text class="t-tick" x="175.0" y="238.0" text-anchor="middle">B: The sky is</text><line class="edge" x1="254.0" y1="233.0" x2="300.0" y2="252.0" marker-end="url(#ah)"/><line class="edge" x1="478.0" y1="252.0" x2="524.0" y2="233.0" marker-end="url(#ah)"/><rect class="box-3" x="528.0" y="218.0" width="90.0" height="30.0" rx="6"/><text class="t-big" x="573.0" y="239.0" text-anchor="middle">blue</text><rect class="box" x="100.0" y="256.0" width="150.0" height="30.0" rx="6"/><text class="t-tick" x="175.0" y="276.0" text-anchor="middle">C: 2 + 2 =</text><line class="edge" x1="254.0" y1="271.0" x2="300.0" y2="252.0" marker-end="url(#ah)"/><line class="edge" x1="478.0" y1="252.0" x2="524.0" y2="271.0" marker-end="url(#ah)"/><rect class="box-3" x="528.0" y="256.0" width="90.0" height="30.0" rx="6"/><text class="t-big" x="573.0" y="277.0" text-anchor="middle">4</text><rect class="box" x="100.0" y="294.0" width="150.0" height="30.0" rx="6"/><text class="t-tick" x="175.0" y="314.0" text-anchor="middle">D: def hello</text><line class="edge" x1="254.0" y1="309.0" x2="300.0" y2="252.0" marker-end="url(#ah)"/><line class="edge" x1="478.0" y1="252.0" x2="524.0" y2="309.0" marker-end="url(#ah)"/><rect class="box-3" x="528.0" y="294.0" width="90.0" height="30.0" rx="6"/><text class="t-big" x="573.0" y="315.0" text-anchor="middle">(</text><rect class="box-2" x="304.0" y="226.0" width="170.0" height="54.0" rx="8"/><text class="t-title" x="389.0" y="248.0" text-anchor="middle">read all weights</text><text class="t-tick" x="389.0" y="266.0" text-anchor="middle">0.99 GB, once</text><text class="t-note" x="630.0" y="252.0" text-anchor="start">4 tokens out</text><text class="t-note" x="630.0" y="270.0" text-anchor="start">for 1.1 ms more</text><text class="t-tick" x="20.0" y="352.0" text-anchor="start">The four sequences do not need to be the same length or the same topic. They only share the trip to memory.</text></svg><figcaption>One decode step for one sequence and for four. With four, the newest token of each sequence is stacked into a small matrix, every weight is read once, and four next tokens come out.</figcaption></figure>

1. **Stack the newest tokens.** The step takes the latest token of each of the four sequences and puts their numbers side by side, one column per sequence. This is panel B of the 3 × 3 example.
2. **Read each weight once and use it four times.** Each weight matrix comes from memory once and is multiplied by all four columns. The arithmetic is four times larger, but it was the cheap part.
3. **Each sequence gets its own next token.** The output has one column per sequence. Each column becomes that sequence's next token, which is added to the end of its own text.

The sequences do not share any text. User A's answer about Paris never mixes with user B's answer about the sky. Each column is computed only from its own sequence. They share only the trip to memory.

Measured on our model, one step for one sequence takes **10.0 ms**. One step for four sequences takes **11.1 ms**. The GPU makes 4 tokens in 11.1 ms instead of 1 token in 10.0 ms. That is 361 tokens per second instead of 100, for 1.1 ms more per step.

The vLLM paper describes the same effect in its background section.

> [!PAPER] Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention · §2.3 · page 3
> [![The paragraph on batching, with amortized across the requests in a batch, arrive at different times, queueing delays, pad the inputs and wasting highlighted](https://ishwarj.com/img/papers/vllm/p2-batching.pngllm/p2-batching.png)](/img/papers/vllm/p2-batching.png)
>
> **What it says:** "Because the requests share the same model weights, the overhead of moving weights is amortized across the requests in a batch". Then two problems: requests "arrive at different times", and a simple batch would "pad the inputs" to the same length, "wasting" compute and memory.
>
> **What it means:**
>
> "Amortized" means the cost of one weight read is split across all requests in the batch. With 4 requests, each pays a quarter of the trip. With 64, each pays a sixty-fourth. This is the bus.
>
> The two problems are real, and this page leaves them for later. Users do not arrive together, and their answers have different lengths. A naive server would make early users wait for late ones, or fill short answers with empty padding. Part 3 shows how vLLM refills the batch at every step so that neither happens.

### The batch, measured from 1 to 128

Here is what batching does on our model, from 1 sequence to 128. Every sequence has a 512-token context.

<figure class="fig"><svg viewBox="0 0 760 320" role="img" aria-label="Decode throughput grows almost linearly with batch size because the weight read is shared."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="64" y1="264.0" x2="740" y2="264.0"/><text class="t-tick" x="56" y="268.0" text-anchor="end">0</text><line class="grid" x1="64" y1="217.2" x2="740" y2="217.2"/><text class="t-tick" x="56" y="221.2" text-anchor="end">1,000</text><line class="grid" x1="64" y1="170.4" x2="740" y2="170.4"/><text class="t-tick" x="56" y="174.39999999999998" text-anchor="end">2,000</text><line class="grid" x1="64" y1="123.6" x2="740" y2="123.6"/><text class="t-tick" x="56" y="127.6" text-anchor="end">3,000</text><line class="grid" x1="64" y1="76.8" x2="740" y2="76.8"/><text class="t-tick" x="56" y="80.79999999999998" text-anchor="end">4,000</text><line class="grid" x1="64" y1="30.0" x2="740" y2="30.0"/><text class="t-tick" x="56" y="34.0" text-anchor="end">5,000</text><line class="axis" x1="64" y1="264" x2="740" y2="264"/><g class="mark"><title>batch 1: 100 tokens/s, 10.0 ms per step</title><rect class="hit-area" x="64.0" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M94.25,264.0V263.34088819824257Q94.25,259.34088819824257 98.25,259.34088819824257H114.25Q118.25,259.34088819824257 118.25,263.34088819824257V264.0Z"/></g><text class="t-val" x="106.25" y="251.34088819824257" text-anchor="middle">99.6</text><text class="t-tick" x="106.25" y="284" text-anchor="middle">1</text><g class="mark"><title>batch 2: 190 tokens/s, 10.6 ms per step</title><rect class="hit-area" x="148.5" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M178.75,264.0V259.12841783897136Q178.75,255.12841783897136 182.75,255.12841783897136H198.75Q202.75,255.12841783897136 202.75,259.12841783897136V264.0Z"/></g><text class="t-val" x="190.75" y="247.12841783897136" text-anchor="middle">190</text><text class="t-tick" x="190.75" y="284" text-anchor="middle">2</text><g class="mark"><title>batch 4: 361 tokens/s, 11.1 ms per step</title><rect class="hit-area" x="233.0" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M263.25,264.0V251.08300029697827Q263.25,247.08300029697827 267.25,247.08300029697827H283.25Q287.25,247.08300029697827 287.25,251.08300029697827V264.0Z"/></g><text class="t-val" x="275.25" y="239.08300029697827" text-anchor="middle">361</text><text class="t-tick" x="275.25" y="284" text-anchor="middle">4</text><g class="mark"><title>batch 8: 670 tokens/s, 11.9 ms per step</title><rect class="hit-area" x="317.5" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M347.75,264.0V236.63128334359791Q347.75,232.63128334359791 351.75,232.63128334359791H367.75Q371.75,232.63128334359791 371.75,236.63128334359791V264.0Z"/></g><text class="t-val" x="359.75" y="224.63128334359791" text-anchor="middle">670</text><text class="t-tick" x="359.75" y="284" text-anchor="middle">8</text><g class="mark"><title>batch 16: 1,247 tokens/s, 12.8 ms per step</title><rect class="hit-area" x="402.0" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M432.25,264.0V209.66274263170567Q432.25,205.66274263170567 436.25,205.66274263170567H452.25Q456.25,205.66274263170567 456.25,209.66274263170567V264.0Z"/></g><text class="t-val" x="444.25" y="197.66274263170567" text-anchor="middle">1,247</text><text class="t-tick" x="444.25" y="284" text-anchor="middle">16</text><g class="mark"><title>batch 32: 2,120 tokens/s, 15.1 ms per step</title><rect class="hit-area" x="486.5" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M516.75,264.0V168.77820471286415Q516.75,164.77820471286415 520.75,164.77820471286415H536.75Q540.75,164.77820471286415 540.75,168.77820471286415V264.0Z"/></g><text class="t-val" x="528.75" y="156.77820471286415" text-anchor="middle">2,120</text><text class="t-tick" x="528.75" y="284" text-anchor="middle">32</text><g class="mark"><title>batch 64: 3,242 tokens/s, 19.7 ms per step</title><rect class="hit-area" x="571.0" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M601.25,264.0V116.27451800295245Q601.25,112.27451800295245 605.25,112.27451800295245H621.25Q625.25,112.27451800295245 625.25,116.27451800295245V264.0Z"/></g><text class="t-val" x="613.25" y="104.27451800295245" text-anchor="middle">3,242</text><text class="t-tick" x="613.25" y="284" text-anchor="middle">64</text><g class="mark"><title>batch 128: 4,367 tokens/s, 29.3 ms per step</title><rect class="hit-area" x="655.5" y="30" width="84.5" height="234"/><path class="s1 bar-mark" d="M685.75,264.0V63.61081165643168Q685.75,59.61081165643168 689.75,59.61081165643168H705.75Q709.75,59.61081165643168 709.75,63.61081165643168V264.0Z"/></g><text class="t-val" x="697.75" y="51.61081165643168" text-anchor="middle">4,367</text><text class="t-tick" x="697.75" y="284" text-anchor="middle">128</text><text class="t-tick" x="402.0" y="310" text-anchor="middle">sequences decoded together (batch size)</text><text class="t-tick" x="14" y="18" text-anchor="start">tokens per second</text></svg><figcaption>Tokens per second as more sequences are decoded together. Hover a bar to see the time per step.</figcaption></figure>

| Sequences at once | Time per step | Tokens per second (all users) | Tokens per second (each user) |
|---|---|---|---|
| 1 | 10.0 ms | 100 | 100 |
| 2 | 10.6 ms | 190 | 94 |
| 4 | 11.1 ms | 361 | 90 |
| 8 | 11.9 ms | 670 | 84 |
| 16 | 12.8 ms | 1,247 | 78 |
| 32 | 15.1 ms | 2,120 | 66 |
| 64 | 19.7 ms | 3,242 | 51 |
| 128 | 29.3 ms | 4,367 | 34 |

Read the table one row at a time.

- **1 to 2.** The step gets 0.6 ms longer, and the GPU makes almost twice as many tokens. The second passenger rides almost free.
- **2 to 8.** Each doubling adds less than a millisecond. Eight sequences make 670 tokens per second, 6.7 times one sequence, while each step is only 19% longer.
- **8 to 32.** The step grows from 11.9 to 15.1 ms. Tokens per second still more than triple, from 670 to 2,120.
- **32 to 64.** The step grows by 4.6 ms. Tokens per second rise by about half, to 3,242. Each new sequence now costs a little time.
- **64 to 128.** The step grows by 9.6 ms, to 29.3 ms. Doubling the batch adds only 35% more tokens per second, to 4,367.

From 1 to 128 sequences, each step became about 3 times slower but produced about 44 times more tokens. Each user's answer streams more slowly, from 100 to 34 tokens per second, but the GPU does far more useful work. Batching is the single biggest lever in LLM serving.

### Where the step time goes

The step times follow a simple pattern. Each step costs a **fixed part** of about 10.6 ms, plus about **0.146 ms for each sequence** in the batch. This line, fitted to the measurements from batch 8 to 128, matches every measured step from batch 4 to 128 within 0.2 ms.

<figure class="fig"><svg viewBox="0 0 760 310" role="img" aria-label="Decode step time split into a fixed part and a part that grows with each sequence. The line fits the measured step times within half a millisecond from batch 4 to 128."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="260.0" x2="710.0" y2="260.0"/><text class="t-tick" x="62.0" y="264.0" text-anchor="end">0 ms</text><line class="grid" x1="70.0" y1="191.2" x2="710.0" y2="191.2"/><text class="t-tick" x="62.0" y="195.2" text-anchor="end">10 ms</text><line class="grid" x1="70.0" y1="122.5" x2="710.0" y2="122.5"/><text class="t-tick" x="62.0" y="126.5" text-anchor="end">20 ms</text><line class="grid" x1="70.0" y1="53.8" x2="710.0" y2="53.8"/><text class="t-tick" x="62.0" y="57.8" text-anchor="end">30 ms</text><g class="mark"><title>batch 1: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M96.66666666666667,260.0V187.125Q96.66666666666667,187.125 96.66666666666667,187.125H150.0Q150.0,187.125 150.0,187.125V260.0Z"/></g><g class="mark"><title>batch 1: part per sequence 0.1 ms</title><path class="s1 bar-mark" d="M96.66666666666667,187.125V187.125Q96.66666666666667,186.12125 97.67041666666667,186.12125H148.99625Q150.0,186.12125 150.0,187.125V187.125Z"/></g><g class="mark"><title>batch 1: measured 10.0 ms</title><circle class="s3 ring" cx="123.3" cy="191.2" r="5"/></g><text class="t-val" x="123.3" y="176.1" text-anchor="middle">10.0 ms</text><text class="t-tick" x="123.3" y="278.0" text-anchor="middle">1</text><g class="mark"><title>batch 4: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M203.33333333333334,260.0V187.125Q203.33333333333334,187.125 203.33333333333334,187.125H256.6666666666667Q256.6666666666667,187.125 256.6666666666667,187.125V260.0Z"/></g><g class="mark"><title>batch 4: part per sequence 0.6 ms</title><path class="s1 bar-mark" d="M203.33333333333334,187.125V186.11Q203.33333333333334,183.11 206.33333333333334,183.11H253.66666666666669Q256.6666666666667,183.11 256.6666666666667,186.11V187.125Z"/></g><g class="mark"><title>batch 4: measured 11.1 ms</title><circle class="s3 ring" cx="230.0" cy="183.7" r="5"/></g><text class="t-val" x="230.0" y="173.1" text-anchor="middle">11.1 ms</text><text class="t-tick" x="230.0" y="278.0" text-anchor="middle">4</text><g class="mark"><title>batch 8: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M310.00000000000006,260.0V187.125Q310.00000000000006,187.125 310.00000000000006,187.125H363.33333333333337Q363.33333333333337,187.125 363.33333333333337,187.125V260.0Z"/></g><g class="mark"><title>batch 8: part per sequence 1.2 ms</title><path class="s1 bar-mark" d="M310.00000000000006,187.125V182.09500000000003Q310.00000000000006,179.09500000000003 313.00000000000006,179.09500000000003H360.33333333333337Q363.33333333333337,179.09500000000003 363.33333333333337,182.09500000000003V187.125Z"/></g><g class="mark"><title>batch 8: measured 11.9 ms</title><circle class="s3 ring" cx="336.7" cy="178.2" r="5"/></g><text class="t-val" x="336.7" y="168.2" text-anchor="middle">11.9 ms</text><text class="t-tick" x="336.7" y="278.0" text-anchor="middle">8</text><g class="mark"><title>batch 32: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M416.6666666666667,260.0V187.125Q416.6666666666667,187.125 416.6666666666667,187.125H470.0Q470.0,187.125 470.0,187.125V260.0Z"/></g><g class="mark"><title>batch 32: part per sequence 4.7 ms</title><path class="s1 bar-mark" d="M416.6666666666667,187.125V158.005Q416.6666666666667,155.005 419.6666666666667,155.005H467.0Q470.0,155.005 470.0,158.005V187.125Z"/></g><g class="mark"><title>batch 32: measured 15.1 ms</title><circle class="s3 ring" cx="443.3" cy="156.2" r="5"/></g><text class="t-val" x="443.3" y="145.0" text-anchor="middle">15.1 ms</text><text class="t-tick" x="443.3" y="278.0" text-anchor="middle">32</text><g class="mark"><title>batch 64: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M523.3333333333334,260.0V187.125Q523.3333333333334,187.125 523.3333333333334,187.125H576.6666666666667Q576.6666666666667,187.125 576.6666666666667,187.125V260.0Z"/></g><g class="mark"><title>batch 64: part per sequence 9.3 ms</title><path class="s1 bar-mark" d="M523.3333333333334,187.125V125.88500000000002Q523.3333333333334,122.88500000000002 526.3333333333334,122.88500000000002H573.6666666666667Q576.6666666666667,122.88500000000002 576.6666666666667,125.88500000000002V187.125Z"/></g><g class="mark"><title>batch 64: measured 19.7 ms</title><circle class="s3 ring" cx="550.0" cy="124.6" r="5"/></g><text class="t-val" x="550.0" y="112.9" text-anchor="middle">19.7 ms</text><text class="t-tick" x="550.0" y="278.0" text-anchor="middle">64</text><g class="mark"><title>batch 128: fixed part 10.6 ms</title><path class="s2 bar-mark" d="M630.0,260.0V187.125Q630.0,187.125 630.0,187.125H683.3333333333334Q683.3333333333334,187.125 683.3333333333334,187.125V260.0Z"/></g><g class="mark"><title>batch 128: part per sequence 18.7 ms</title><path class="s1 bar-mark" d="M630.0,187.125V61.64500000000001Q630.0,58.64500000000001 633.0,58.64500000000001H680.3333333333334Q683.3333333333334,58.64500000000001 683.3333333333334,61.64500000000001V187.125Z"/></g><g class="mark"><title>batch 128: measured 29.3 ms</title><circle class="s3 ring" cx="656.7" cy="58.6" r="5"/></g><text class="t-val" x="656.7" y="48.6" text-anchor="middle">29.3 ms</text><text class="t-tick" x="656.7" y="278.0" text-anchor="middle">128</text><line class="axis" x1="70.0" y1="260.0" x2="710.0" y2="260.0"/><text class="t-tick" x="390.0" y="300.0" text-anchor="middle">sequences in the batch</text><rect class="s2" x="80.0" y="8.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="98.0" y="18.0" text-anchor="start">fixed part, 10.6 ms (weights + overhead)</text><rect class="s1" x="380.0" y="8.0" width="12.0" height="12.0" rx="2"/><text class="t-tick" x="398.0" y="18.0" text-anchor="start">0.146 ms per sequence</text><circle class="s3" cx="556" cy="14" r="5"/><text class="t-tick" x="566.0" y="18.0" text-anchor="start">measured</text></svg><figcaption>Decode step time split into a fixed part and a part that grows with each sequence. The dots are the measured step times; the bars are the fitted line, 10.6 ms plus 0.146 ms per sequence.</figcaption></figure>

The fixed part is the trip: reading the weights plus the start-up costs of the kernels. Every step pays it once, whatever the batch size. The part per sequence is the work that belongs to one sequence: its arithmetic, and reading its own notes from memory. At batch 4 the fixed part is 95% of the step, so the extra sequences are nearly free. At batch 128 the part per sequence is 18.6 ms, which is larger than the fixed part.

The same line shows the end of the road. When the batch grows without limit, the fixed part stops mattering, and each token costs 0.146 ms. So this GPU can never make more than about **6,865 tokens per second** on this model, however large the batch. At 128 sequences it is already at 4,367.

### Batching on the roofline

On the roofline, batching moves decode to the right. Every sequence in the batch adds about one FLOP of work for each byte of weights read. So a batch of 64 has about 64 FLOPs per byte, and the highest speed rises with it.

<figure class="fig"><svg viewBox="0 0 760 356" role="img" aria-label="Decode climbs the memory slope of the H100 roofline as the batch grows: each sequence in the batch adds about one FLOP per byte."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><line class="grid" x1="70.0" y1="280.0" x2="730.0" y2="280.0"/><text class="t-tick" x="62.0" y="284.0" text-anchor="end">1</text><line class="grid" x1="70.0" y1="201.9" x2="730.0" y2="201.9"/><text class="t-tick" x="62.0" y="205.9" text-anchor="end">10</text><line class="grid" x1="70.0" y1="123.8" x2="730.0" y2="123.8"/><text class="t-tick" x="62.0" y="127.8" text-anchor="end">100</text><line class="grid" x1="70.0" y1="45.6" x2="730.0" y2="45.6"/><text class="t-tick" x="62.0" y="49.6" text-anchor="end">1,000</text><text class="t-tick" x="70.0" y="298.0" text-anchor="middle">1</text><text class="t-tick" x="180.0" y="298.0" text-anchor="middle">4</text><text class="t-tick" x="290.0" y="298.0" text-anchor="middle">16</text><text class="t-tick" x="400.0" y="298.0" text-anchor="middle">64</text><text class="t-tick" x="521.3" y="298.0" text-anchor="middle">295</text><text class="t-tick" x="620.0" y="298.0" text-anchor="middle">1,024</text><text class="t-tick" x="730.0" y="298.0" text-anchor="middle">4,096</text><line class="axis" x1="70.0" y1="280.0" x2="730.0" y2="280.0"/><polyline class="l1" points="70.0,239.0 521.3,46.0 730.0,46.0"/><line class="drop" x1="521.3" y1="46.0" x2="521.3" y2="280.0"/><text class="t-tick" x="730.0" y="36.0" text-anchor="end">compute roof: 989 TFLOP/s</text><text class="t-tick" x="212.2" y="200.2" text-anchor="start" transform="rotate(-23.2 212.2 200.2)">memory slope: 3.35 TB/s × FLOPs per byte</text><g class="mark"><title>batch 1: about 1 FLOPs per byte, at most 3 TFLOP/s (0.3% of the roof)</title><circle class="s2 ring" cx="70.0" cy="239.0" r="6"/></g><text class="t-note" x="80.0" y="227.0" text-anchor="start">batch 1</text><g class="mark"><title>batch 4: about 4 FLOPs per byte, at most 13 TFLOP/s (1.4% of the roof)</title><circle class="s2 ring" cx="180.0" cy="191.9" r="6"/></g><text class="t-note" x="118.0" y="181.9" text-anchor="start">batch 4</text><g class="mark"><title>batch 16: about 16 FLOPs per byte, at most 54 TFLOP/s (5.4% of the roof)</title><circle class="s2 ring" cx="290.0" cy="144.9" r="6"/></g><text class="t-note" x="220.0" y="134.9" text-anchor="start">batch 16</text><g class="mark"><title>batch 64: about 64 FLOPs per byte, at most 214 TFLOP/s (21.7% of the roof)</title><circle class="s2 ring" cx="400.0" cy="97.9" r="6"/></g><text class="t-note" x="330.0" y="87.9" text-anchor="start">batch 64</text><g class="mark"><title>batch 128: about 128 FLOPs per byte, at most 429 TFLOP/s (43.4% of the roof)</title><circle class="s2 ring" cx="455.0" cy="74.4" r="6"/></g><text class="t-note" x="377.0" y="64.4" text-anchor="start">batch 128</text><g class="mark"><title>prefill, 512 tokens: about 512 FLOPs per byte, at most 989 TFLOP/s (100.0% of the roof)</title><circle class="s1 ring" cx="565.0" cy="46.0" r="6"/></g><text class="t-note" x="575.0" y="72.0" text-anchor="start">prefill, 512 tokens</text><text class="t-tick" x="400.0" y="324.0" text-anchor="middle">FLOPs per byte read from memory (log scale)</text><text class="t-tick" x="10.0" y="16.0" text-anchor="start">TFLOP/s</text><text class="t-tick" x="70.0" y="348.0" text-anchor="start">Weights only. Each sequence also reads its own notes (Part 2), which pulls real points lower.</text></svg><figcaption>Decode climbs the memory slope of the H100 roofline as the batch grows. Each sequence in the batch adds about one FLOP per byte of weights.</figcaption></figure>

On an H100, batch 1 can use at most 0.3% of the compute roof. Batch 64 can reach 21.7%, and batch 128 can reach 43.4%. Decode only reaches the roof near batch 295. Real points sit lower than these ceilings, because each sequence also reads its own notes, which adds bytes without adding weight reuse.

## Real data-centre GPUs

So far the only big GPU on this page was the H100. A new inference engineer meets several others: the older A100, the H200, the newer B200 and AMD's MI300X. Their spec sheets list dozens of numbers. For decode, three of them matter, and you already know what each one means.

1. **Memory capacity (GB).** How big a model fits, and how much room is left for the notes of each conversation.
2. **Memory bandwidth (TB/s).** How fast one decode step can read the weights. This sets the floor of every step.
3. **Peak compute (TFLOP/s).** The roof of the roofline. Decode reaches it only with a very large batch.

> [!DEFINITION] Dense and sparse TFLOP/s
> Spec sheets often show two compute numbers. The "with sparsity" number counts a special mode that skips half of the weights, and it is twice the dense number. LLM serving uses dense matrices, so use the dense number. For the NVIDIA GPUs below, it is half of the sparsity figure.

Here are the five GPUs, from each vendor's own spec page or datasheet. BF16 (bfloat16) is the 16-bit number format that the weights on this page use. The ridge point is the compute divided by the bandwidth, as in the roofline section.

| GPU | Memory | Bandwidth | Dense BF16 compute | Ridge point (FLOPs per byte) |
|---|---|---|---|---|
| NVIDIA A100 80GB SXM | 80 GB | 2.04 TB/s | 312 TFLOP/s | 153 |
| NVIDIA H100 SXM | 80 GB | 3.35 TB/s | 989 TFLOP/s | 295 |
| NVIDIA H200 SXM | 141 GB | 4.8 TB/s | 989 TFLOP/s | 206 |
| NVIDIA B200, HGX | 180 GB | 7.7 TB/s | 2,250 TFLOP/s | 292 |
| AMD Instinct MI300X | 192 GB | 5.3 TB/s | 1,307 TFLOP/s | 247 |

### A worked example: one user on each GPU

Take Llama 3.1 8B in BF16, 16.1 GB of weights. At batch 1, every decode step must read all 16.1 GB. So the least time one step can take is the weight size divided by the bandwidth. On an H100: 16.1 GB ÷ 3.35 TB/s = **4.8 ms**. One step gives one token, so one user can get at most 1,000 ÷ 4.8 = **209 tokens per second**.

$$
\text{least time per step} = \frac{\text{weight bytes}}{\text{bandwidth}}, \qquad \text{most tokens/s for one user} = \frac{1}{\text{least time per step}}
$$

Read it left to right: bytes to move, divided by bytes per second, gives seconds. The same arithmetic on every GPU:

| GPU | Least time per decode step | Most tokens/s for one user |
|---|---|---|
| A100 80GB | 7.9 ms | 127 |
| H100 SXM | 4.8 ms | 209 |
| H200 SXM | 3.4 ms | 299 |
| B200 | 2.1 ms | 479 |
| MI300X | 3.0 ms | 330 |

<figure class="fig"><svg viewBox="0 0 760 286" role="img" aria-label="Left: the most tokens per second one user can get from Llama 3.1 8B at batch 1 on each GPU. Right: each GPU's ridge point, the batch size where decode stops being limited by memory."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="10.0" y="22.0" text-anchor="start">one user, Llama 3.1 8B, batch 1</text><text class="t-tick" x="10.0" y="40.0" text-anchor="start">the most tokens/s: bandwidth ÷ 16.1 GB</text><text class="t-note" x="112.0" y="76.0" text-anchor="end">A100</text><g class="mark"><title>A100 80GB SXM: 127 tok/s</title><path class="s2 bar-mark" d="M120,58H172.15600153271387Q176.15600153271387,58 176.15600153271387,62V80Q176.15600153271387,84 172.15600153271387,84H120Z"/></g><text class="t-val" x="182.2" y="76.0" text-anchor="start">127 tok/s</text><text class="t-note" x="112.0" y="116.0" text-anchor="end">H100 SXM</text><g class="mark"><title>H100 SXM: 209 tok/s</title><path class="s2 bar-mark" d="M120,98H208.2621898649296Q212.2621898649296,98 212.2621898649296,102V120Q212.2621898649296,124 208.2621898649296,124H120Z"/></g><text class="t-val" x="218.3" y="116.0" text-anchor="start">209 tok/s</text><text class="t-note" x="112.0" y="156.0" text-anchor="end">H200 SXM</text><g class="mark"><title>H200 SXM: 299 tok/s</title><path class="s2 bar-mark" d="M120,138H248.19657055273493Q252.19657055273493,138 252.19657055273493,142V160Q252.19657055273493,164 248.19657055273493,164H120Z"/></g><text class="t-val" x="258.2" y="156.0" text-anchor="start">299 tok/s</text><text class="t-note" x="112.0" y="196.0" text-anchor="end">B200</text><g class="mark"><title>B200 (HGX): 479 tok/s</title><path class="s2 bar-mark" d="M120,178H328.0653319283456Q332.0653319283456,178 332.0653319283456,182V200Q332.0653319283456,204 328.0653319283456,204H120Z"/></g><text class="t-val" x="338.1" y="196.0" text-anchor="start">479 tok/s</text><text class="t-note" x="112.0" y="236.0" text-anchor="end">MI300X</text><g class="mark"><title>MI300X: 330 tok/s</title><path class="s2 bar-mark" d="M120,218H261.9670466519782Q265.9670466519782,218 265.9670466519782,222V240Q265.9670466519782,244 261.9670466519782,244H120Z"/></g><text class="t-val" x="272.0" y="236.0" text-anchor="start">330 tok/s</text><text class="t-title" x="390.0" y="22.0" text-anchor="start">ridge point</text><text class="t-tick" x="390.0" y="40.0" text-anchor="start">FLOPs per byte = TFLOP/s ÷ TB/s</text><text class="t-note" x="492.0" y="76.0" text-anchor="end">A100</text><g class="mark"><title>A100 80GB SXM: 153</title><path class="s1 bar-mark" d="M500,58H605.980382540461Q609.980382540461,58 609.980382540461,62V80Q609.980382540461,84 605.980382540461,84H500Z"/></g><text class="t-val" x="616.0" y="76.0" text-anchor="start">153</text><text class="t-note" x="492.0" y="116.0" text-anchor="end">H100 SXM</text><g class="mark"><title>H100 SXM: 295</title><path class="s1 bar-mark" d="M500,98H708.1921641791045Q712.1921641791045,98 712.1921641791045,102V120Q712.1921641791045,124 708.1921641791045,124H500Z"/></g><text class="t-val" x="718.2" y="116.0" text-anchor="start">295</text><text class="t-note" x="492.0" y="156.0" text-anchor="end">H200 SXM</text><g class="mark"><title>H200 SXM: 206</title><path class="s1 bar-mark" d="M500,138H644.0924479166667Q648.0924479166667,138 648.0924479166667,142V160Q648.0924479166667,164 644.0924479166667,164H500Z"/></g><text class="t-val" x="654.1" y="156.0" text-anchor="start">206</text><text class="t-note" x="492.0" y="196.0" text-anchor="end">B200</text><g class="mark"><title>B200 (HGX): 292</title><path class="s1 bar-mark" d="M500,178H706.0243506493506Q710.0243506493506,178 710.0243506493506,182V200Q710.0243506493506,204 706.0243506493506,204H500Z"/></g><text class="t-val" x="716.0" y="196.0" text-anchor="start">292</text><text class="t-note" x="492.0" y="236.0" text-anchor="end">MI300X</text><g class="mark"><title>MI300X: 247</title><path class="s1 bar-mark" d="M500,218H673.3007075471698Q677.3007075471698,218 677.3007075471698,222V240Q677.3007075471698,244 673.3007075471698,244H500Z"/></g><text class="t-val" x="683.3" y="236.0" text-anchor="start">247</text><text class="t-tick" x="10.0" y="272.0" text-anchor="start">Vendor specs: dense BF16, no sparsity. Upper bounds from bandwidth alone; real steps are slower.</text></svg><figcaption>Left: the most tokens per second one user can get from Llama 3.1 8B at batch 1, which is set by bandwidth alone. Right: each GPU's ridge point, about the batch size where decode stops being limited by memory.</figcaption></figure>

These are upper limits. Real steps are slower, because of the launch costs and the other work from the CUDA section. But the order holds. For one user, the GPU with more bandwidth answers faster, and its compute hardly matters. The H200 has exactly the H100's compute, and it is 43% faster for one user only because its memory is faster.

### Why the ridge stays high

Look at the last column of the first table. From the A100 to the B200, compute grew 7.2 times, while bandwidth grew 3.8 times. So the ridge point almost doubled, from 153 to 292 FLOPs per byte. It does not rise with every chip. The H200 is an H100 with faster memory, so its ridge fell to 206. But over the generations, arithmetic has become cheaper faster than memory traffic.

Each sequence in the batch adds about one FLOP per byte of weights. So the ridge point is about the batch size at which decode stops waiting for memory. On an A100, that is a batch of about 150. On an H100 or a B200, it is about 300. A newer GPU leaves even more of its compute idle at a small batch, so batching matters more with each generation.

### Many GPUs in one server

Data-centre GPUs often come eight to a server, as in NVIDIA's DGX systems. The eight GPUs connect through **NVLink**, NVIDIA's fast link between GPUs, so that one GPU can read another's memory quickly.

<figure class="fig"><svg viewBox="0 0 760 206" role="img" aria-label="An 8-GPU server. Every GPU connects to NVLink switches, so any GPU can read from any other at high speed."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="22.0" text-anchor="start">one server: 8 GPUs joined by NVLink switches</text><rect class="box-1" x="30.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="68.0" y="64.0" text-anchor="middle">GPU 0</text><text class="t-tick" x="68.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="68.0" y1="98.0" x2="68.0" y2="128.0"/><rect class="box-1" x="120.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="158.0" y="64.0" text-anchor="middle">GPU 1</text><text class="t-tick" x="158.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="158.0" y1="98.0" x2="158.0" y2="128.0"/><rect class="box-1" x="210.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="248.0" y="64.0" text-anchor="middle">GPU 2</text><text class="t-tick" x="248.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="248.0" y1="98.0" x2="248.0" y2="128.0"/><rect class="box-1" x="300.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="338.0" y="64.0" text-anchor="middle">GPU 3</text><text class="t-tick" x="338.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="338.0" y1="98.0" x2="338.0" y2="128.0"/><rect class="box-1" x="390.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="428.0" y="64.0" text-anchor="middle">GPU 4</text><text class="t-tick" x="428.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="428.0" y1="98.0" x2="428.0" y2="128.0"/><rect class="box-1" x="480.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="518.0" y="64.0" text-anchor="middle">GPU 5</text><text class="t-tick" x="518.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="518.0" y1="98.0" x2="518.0" y2="128.0"/><rect class="box-1" x="570.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="608.0" y="64.0" text-anchor="middle">GPU 6</text><text class="t-tick" x="608.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="608.0" y1="98.0" x2="608.0" y2="128.0"/><rect class="box-1" x="660.0" y="40.0" width="76.0" height="56.0" rx="8"/><text class="t-note" x="698.0" y="64.0" text-anchor="middle">GPU 7</text><text class="t-tick" x="698.0" y="82.0" text-anchor="middle">80 GB</text><line class="edge-on" x1="698.0" y1="98.0" x2="698.0" y2="128.0"/><rect class="box-on" x="30.0" y="130.0" width="706.0" height="34.0" rx="8"/><text class="t-note" x="383.0" y="152.0" text-anchor="middle">NVLink switch: every GPU talks to every other GPU</text><text class="t-tick" x="20.0" y="192.0" text-anchor="start">H100: 900 GB/s of NVLink per GPU. B200: 1.8 TB/s. Both are far faster than the PCIe link to the CPU.</text></svg><figcaption>An 8-GPU H100 server. Every GPU connects to NVLink switches, so any GPU can exchange data with any other at high speed.</figcaption></figure>

> [!DEFINITION] NVLink
> NVIDIA's direct link between GPUs. An H100 SXM has 900 GB/s of NVLink bandwidth, and a B200 has 1.8 TB/s. That is far faster than PCIe, the standard link between a GPU and the CPU, but still slower than the GPU's own HBM.

A model that does not fit on one GPU is split across several of them. The GPUs then exchange partial results over NVLink in every layer. Llama 3.1 70B, for example, has 141 GB of weights in BF16, more than an 80 GB H100 can hold. [Part 7](https://ishwarj.com/writings/llm-inference-7-beyond-one-gpu/) shows how the split works and what it costs.

## Why the batch cannot grow forever

If one more passenger is almost free, why not run a thousand sequences in every step? Two limits stop you.

### Limit 1: every step gets slower

Every user in the batch waits for the whole step. At batch 1, a user gets a new token every 10.0 ms. At batch 128, every user waits 29.3 ms per token. That is still fast enough to read, but there is a cliff. Once the part per sequence is larger than the fixed part, each new sequence slows down everyone already in the batch.

On a large GPU the same thing happens at the roof. Past the ridge point, a step has more arithmetic than the memory read can hide. Then the step time grows in step with the batch, and tokens per second stop growing. The bus is full: one more passenger needs one more trip.

### Limit 2: every sequence needs its own notes

Remember the notes from prefill. Each sequence keeps notes on every token it has read or written, so that it never has to read those tokens again. Those notes are different for every conversation, so the batch cannot share them the way it shares the weights. They must all sit in GPU memory, next to the weights, for as long as the conversation runs.

> [!DEFINITION] KV cache
> The notes a model keeps for every token of a sequence: one pair of vectors, a key and a value, per token in every layer. It grows by one entry per token as the answer is written. Part 2 explains what the keys and values are and why they exist.

For our small model, each token's notes take 12 KiB. A batch of 128 sequences with 512 tokens each needs 128 × 512 × 12 KiB = 768 MiB of notes, which is **0.81 GB**. The weights take 0.99 GB. So at batch 128, the notes are already almost as large as the whole model.

For a larger model the notes are far larger. Llama 3.1 8B needs 128 KiB of notes per token. On an 80 GB H100, its weights take 16.1 GB, which leaves 63.9 GB. A 4,096-token conversation needs 512 MiB of notes, so at most 119 such conversations fit, even before any working memory. At the model's full 128K-token context, one conversation needs 16 GiB, and only 3 fit.

<figure class="fig"><svg viewBox="0 0 760 260" role="img" aria-label="GPU memory split between the weights and the notes of every sequence in the batch. The notes grow with the batch and with the length of each conversation."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="40.0" y="28.0" text-anchor="start">our run: Qwen2.5-0.5B, 128 sequences of 512 tokens</text><g class="mark"><title>weights: 0.99 GB</title><rect class="box-2" x="40.0" y="40.0" width="374.0" height="40.0" rx="3"/></g><g class="mark"><title>notes of 128 sequences: 0.81 GB</title><rect class="box-1" x="414.0" y="40.0" width="306.0" height="40.0" rx="3"/></g><rect class="box-ghost" x="40.0" y="40.0" width="680.0" height="40.0" rx="3"/><text class="t-tick" x="40.0" y="100.0" text-anchor="start"></text><text class="t-title" x="227.0" y="65.0" text-anchor="middle">weights 0.99 GB</text><text class="t-title" x="567.0" y="65.0" text-anchor="middle">notes 0.81 GB</text><text class="t-tick" x="40.0" y="100.0" text-anchor="start">Each token's notes take 12 KiB; 128 × 512 tokens × 12 KiB = 0.81 GB, almost as much as the model.</text><text class="t-title" x="40.0" y="148.0" text-anchor="start">Llama 3.1 8B on an 80 GB H100, conversations of 4,096 tokens</text><g class="mark"><title>weights: 16.1 GB</title><rect class="box-2" x="40.0" y="160.0" width="136.8" height="40.0" rx="3"/></g><g class="mark"><title>conversation 1: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="176.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 2: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="181.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 3: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="186.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 4: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="190.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 5: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="195.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 6: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="199.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 7: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="204.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 8: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="208.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 9: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="213.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 10: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="217.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 11: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="222.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 12: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="227.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 13: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="231.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 14: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="236.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 15: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="240.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 16: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="245.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 17: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="249.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 18: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="254.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 19: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="259.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 20: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="263.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 21: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="268.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 22: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="272.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 23: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="277.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 24: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="281.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 25: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="286.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 26: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="290.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 27: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="295.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 28: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="300.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 29: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="304.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 30: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="309.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 31: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="313.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 32: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="318.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 33: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="322.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 34: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="327.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 35: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="332.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 36: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="336.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 37: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="341.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 38: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="345.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 39: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="350.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 40: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="354.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 41: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="359.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 42: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="363.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 43: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="368.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 44: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="373.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 45: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="377.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 46: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="382.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 47: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="386.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 48: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="391.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 49: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="395.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 50: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="400.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 51: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="405.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 52: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="409.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 53: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="414.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 54: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="418.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 55: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="423.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 56: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="427.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 57: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="432.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 58: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="437.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 59: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="441.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 60: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="446.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 61: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="450.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 62: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="455.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 63: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="459.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 64: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="464.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 65: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="468.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 66: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="473.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 67: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="478.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 68: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="482.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 69: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="487.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 70: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="491.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 71: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="496.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 72: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="500.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 73: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="505.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 74: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="510.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 75: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="514.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 76: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="519.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 77: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="523.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 78: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="528.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 79: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="532.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 80: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="537.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 81: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="541.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 82: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="546.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 83: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="551.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 84: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="555.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 85: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="560.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 86: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="564.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 87: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="569.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 88: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="573.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 89: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="578.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 90: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="583.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 91: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="587.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 92: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="592.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 93: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="596.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 94: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="601.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 95: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="605.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 96: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="610.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 97: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="614.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 98: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="619.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 99: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="624.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 100: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="628.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 101: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="633.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 102: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="637.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 103: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="642.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 104: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="646.9" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 105: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="651.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 106: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="656.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 107: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="660.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 108: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="665.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 109: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="669.7" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 110: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="674.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 111: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="678.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 112: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="683.4" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 113: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="688.0" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 114: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="692.5" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 115: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="697.1" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 116: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="701.6" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 117: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="706.2" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 118: 4,096 tokens, 512 MiB of notes</title><rect class="box-3" x="710.8" y="160.0" width="4.6" height="40.0" rx="3"/></g><g class="mark"><title>conversation 119: 4,096 tokens, 512 MiB of notes</title><rect class="box-1" x="715.3" y="160.0" width="4.6" height="40.0" rx="3"/></g><rect class="box-ghost" x="40.0" y="160.0" width="680.0" height="40.0" rx="3"/><text class="t-tick" x="40.0" y="220.0" text-anchor="start"></text><text class="t-title" x="108.4" y="185.0" text-anchor="middle">weights</text><text class="t-tick" x="40.0" y="220.0" text-anchor="start">Each thin stripe is one conversation's notes (512 MiB). At most 119 fit, before any working memory.</text><text class="t-tick" x="40.0" y="240.0" text-anchor="start">At the full 128K context, one conversation needs 16 GiB of notes, and at most 3 fit.</text></svg><figcaption>GPU memory split between the weights and the notes of every sequence in the batch. The notes grow with the size of the batch and with the length of each conversation.</figcaption></figure>

So on a real server the batch is usually limited by memory for notes, long before the compute roof. The vLLM paper says this in one line.

> [!PAPER] Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention · §1 · page 2
> [![The next lines of the introduction, with the way the KV cache is managed is critical in determining the maximum batch size highlighted](https://ishwarj.com/img/papers/vllm/p1-intro-batch-limit.pngtro-batch-limit.png)](/img/papers/vllm/p1-intro-batch-limit.png)
>
> **What it says:** "the way the KV cache is managed is critical in determining the maximum batch size". Managed badly, the cache "can significantly limit the batch size and consequently the throughput".
>
> **What it means:**
>
> The weights are fixed, and the short-lived working numbers of a step are small. So the one part of memory that changes with the batch is the KV cache. Every byte wasted there is a sequence that cannot join the batch, and every missing sequence is lost throughput.
>
> That sentence links this page to the next two. Part 2 measures how big the notes get. Part 3 shows how vLLM stores them so that almost no memory is wasted.

> [!IMPORTANT] The rule of Part 1
> **To make a GPU productive, decode many sequences in the same step.** One read of the weights then serves the whole batch. The batch has two limits. Each step gets slower as the batch grows. And every sequence needs its own memory for its notes, the KV cache. On real servers, memory for the notes is usually the tighter limit.

## Latency or throughput: choose your trade

Batching exposes the main trade-off in serving. **Latency** is how fast one user gets their tokens. **Throughput** is how many tokens the whole GPU makes per second. The batch size moves one against the other.

<figure class="fig"><svg viewBox="0 0 760 270" role="img" aria-label="As the batch grows from 1 to 128, each user gets tokens about three times slower, while the GPU as a whole makes 44 times more."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-title" x="20.0" y="24.0" text-anchor="start">what one user sees: tokens/s</text><line class="grid" x1="60.0" y1="220.0" x2="360.0" y2="220.0"/><text class="t-tick" x="54.0" y="224.0" text-anchor="end">0</text><line class="grid" x1="60.0" y1="142.7" x2="360.0" y2="142.7"/><text class="t-tick" x="54.0" y="146.7" text-anchor="end">50</text><line class="grid" x1="60.0" y1="65.5" x2="360.0" y2="65.5"/><text class="t-tick" x="54.0" y="69.5" text-anchor="end">100</text><g class="mark"><title>batch 1: 100</title><path class="s2 bar-mark" d="M66.75,220.0V68.45454545454547Q66.75,65.45454545454547 69.75,65.45454545454547H87.75Q90.75,65.45454545454547 90.75,68.45454545454547V220.0Z"/></g><text class="t-tick" x="78.8" y="236.0" text-anchor="middle">1</text><g class="mark"><title>batch 2: 94</title><path class="s2 bar-mark" d="M104.25,220.0V77.20240137221268Q104.25,74.20240137221268 107.25,74.20240137221268H125.25Q128.25,74.20240137221268 128.25,77.20240137221268V220.0Z"/></g><text class="t-tick" x="116.2" y="236.0" text-anchor="middle">2</text><g class="mark"><title>batch 4: 90</title><path class="s2 bar-mark" d="M141.75,220.0V83.76986076986077Q141.75,80.76986076986077 144.75,80.76986076986077H162.75Q165.75,80.76986076986077 165.75,83.76986076986077V220.0Z"/></g><text class="t-tick" x="153.8" y="236.0" text-anchor="middle">4</text><g class="mark"><title>batch 8: 84</title><path class="s2 bar-mark" d="M179.25,220.0V93.12987012987014Q179.25,90.12987012987014 182.25,90.12987012987014H200.25Q203.25,90.12987012987014 203.25,93.12987012987014V220.0Z"/></g><text class="t-tick" x="191.2" y="236.0" text-anchor="middle">8</text><g class="mark"><title>batch 16: 78</title><path class="s2 bar-mark" d="M216.75,220.0V102.26136363636364Q216.75,99.26136363636364 219.75,99.26136363636364H237.75Q240.75,99.26136363636364 240.75,102.26136363636364V220.0Z"/></g><text class="t-tick" x="228.8" y="236.0" text-anchor="middle">16</text><g class="mark"><title>batch 32: 66</title><path class="s2 bar-mark" d="M254.25,220.0V120.65201685731485Q254.25,117.65201685731485 257.25,117.65201685731485H275.25Q278.25,117.65201685731485 278.25,120.65201685731485V220.0Z"/></g><text class="t-tick" x="266.2" y="236.0" text-anchor="middle">32</text><g class="mark"><title>batch 64: 51</title><path class="s2 bar-mark" d="M291.75,220.0V144.5505306875865Q291.75,141.5505306875865 294.75,141.5505306875865H312.75Q315.75,141.5505306875865 315.75,144.5505306875865V220.0Z"/></g><text class="t-tick" x="303.8" y="236.0" text-anchor="middle">64</text><g class="mark"><title>batch 128: 34</title><path class="s2 bar-mark" d="M329.25,220.0V170.25411107663666Q329.25,167.25411107663666 332.25,167.25411107663666H350.25Q353.25,167.25411107663666 353.25,170.25411107663666V220.0Z"/></g><text class="t-tick" x="341.2" y="236.0" text-anchor="middle">128</text><text class="t-val" x="78.8" y="59.5" text-anchor="middle">100</text><text class="t-val" x="341.2" y="161.3" text-anchor="middle">34</text><line class="axis" x1="60.0" y1="220.0" x2="360.0" y2="220.0"/><text class="t-tick" x="210.0" y="256.0" text-anchor="middle">sequences in the batch</text><text class="t-title" x="400.0" y="24.0" text-anchor="start">what the GPU makes: tokens/s</text><line class="grid" x1="440.0" y1="220.0" x2="740.0" y2="220.0"/><text class="t-tick" x="434.0" y="224.0" text-anchor="end">0</text><line class="grid" x1="440.0" y1="149.2" x2="740.0" y2="149.2"/><text class="t-tick" x="434.0" y="153.2" text-anchor="end">2,000</text><line class="grid" x1="440.0" y1="78.3" x2="740.0" y2="78.3"/><text class="t-tick" x="434.0" y="82.3" text-anchor="end">4,000</text><g class="mark"><title>batch 1: 100</title><path class="s1 bar-mark" d="M446.75,220.0V219.45833333333334Q446.75,216.45833333333334 449.75,216.45833333333334H467.75Q470.75,216.45833333333334 470.75,219.45833333333334V220.0Z"/></g><text class="t-tick" x="458.8" y="236.0" text-anchor="middle">1</text><g class="mark"><title>batch 2: 190</title><path class="s1 bar-mark" d="M484.25,220.0V216.27083333333334Q484.25,213.27083333333334 487.25,213.27083333333334H505.25Q508.25,213.27083333333334 508.25,216.27083333333334V220.0Z"/></g><text class="t-tick" x="496.2" y="236.0" text-anchor="middle">2</text><g class="mark"><title>batch 4: 361</title><path class="s1 bar-mark" d="M521.75,220.0V210.21458333333334Q521.75,207.21458333333334 524.75,207.21458333333334H542.75Q545.75,207.21458333333334 545.75,210.21458333333334V220.0Z"/></g><text class="t-tick" x="533.8" y="236.0" text-anchor="middle">4</text><g class="mark"><title>batch 8: 670</title><path class="s1 bar-mark" d="M559.25,220.0V199.27083333333334Q559.25,196.27083333333334 562.25,196.27083333333334H580.25Q583.25,196.27083333333334 583.25,199.27083333333334V220.0Z"/></g><text class="t-tick" x="571.2" y="236.0" text-anchor="middle">8</text><g class="mark"><title>batch 16: 1,247</title><path class="s1 bar-mark" d="M596.75,220.0V178.83541666666667Q596.75,175.83541666666667 599.75,175.83541666666667H617.75Q620.75,175.83541666666667 620.75,178.83541666666667V220.0Z"/></g><text class="t-tick" x="608.8" y="236.0" text-anchor="middle">16</text><g class="mark"><title>batch 32: 2,120</title><path class="s1 bar-mark" d="M634.25,220.0V147.91666666666669Q634.25,144.91666666666669 637.25,144.91666666666669H655.25Q658.25,144.91666666666669 658.25,147.91666666666669V220.0Z"/></g><text class="t-tick" x="646.2" y="236.0" text-anchor="middle">32</text><g class="mark"><title>batch 64: 3,242</title><path class="s1 bar-mark" d="M671.75,220.0V108.17916666666666Q671.75,105.17916666666666 674.75,105.17916666666666H692.75Q695.75,105.17916666666666 695.75,108.17916666666666V220.0Z"/></g><text class="t-tick" x="683.8" y="236.0" text-anchor="middle">64</text><g class="mark"><title>batch 128: 4,367</title><path class="s1 bar-mark" d="M709.25,220.0V68.33541666666667Q709.25,65.33541666666667 712.25,65.33541666666667H730.25Q733.25,65.33541666666667 733.25,68.33541666666667V220.0Z"/></g><text class="t-tick" x="721.2" y="236.0" text-anchor="middle">128</text><text class="t-val" x="458.8" y="210.5" text-anchor="middle">100</text><text class="t-val" x="721.2" y="59.3" text-anchor="middle">4,367</text><line class="axis" x1="440.0" y1="220.0" x2="740.0" y2="220.0"/><text class="t-tick" x="590.0" y="256.0" text-anchor="middle">sequences in the batch</text></svg><figcaption>As the batch grows from 1 to 128, each user receives tokens about three times more slowly, while the GPU as a whole makes 44 times more.</figcaption></figure>

> [!DEFINITION] Latency
> How long one user waits: for the first token (TTFT) or between tokens (TPOT).

| You care about | You want | What it costs |
|---|---|---|
| one user, fast | small batches | low TPOT, but an expensive GPU sits mostly idle |
| many users, cheap | large batches | high throughput per GPU, but each token arrives later |
| a quick first word | prefills that are not stuck behind other work | a scheduler that stops long prompts from blocking short ones (Part 3) |

Every serving system, vLLM included, manages this trade. It packs as many sequences into each step as memory allows, and it keeps every user's TTFT and TPOT within limits.

## A hint of a very useful trick

One measurement from earlier deserves a second look. A prefill of **16 tokens took 10.8 ms**. A decode step for **1 token took 10.0 ms**. Sixteen tokens in one pass cost almost the same as one.

<figure class="fig"><svg viewBox="0 0 760 160" role="img" aria-label="One decode step for 1 token takes 10.0 ms; one pass over 16 tokens takes 10.8 ms."><defs><marker id="ah" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="arrow" d="M0,0L10,5L0,10z"/></marker><marker id="ah-on" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-on" d="M0,0L10,5L0,10z"/></marker><marker id="ah-2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="6" markerHeight="6" orient="auto-start-reverse"><path class="arrow-2" d="M0,0L10,5L0,10z"/></marker></defs><text class="t-note" x="238.0" y="52.0" text-anchor="end">decode: 1 token</text><g class="mark"><title>decode: 1 token: 10.0 ms</title><path class="s2 bar-mark" d="M250,30H646.0Q650.0,30 650.0,34V56Q650.0,60 646.0,60H250Z"/></g><text class="t-val" x="658.0" y="51.0" text-anchor="start">10.0 ms</text><text class="t-note" x="238.0" y="102.0" text-anchor="end">prefill: 16 tokens</text><g class="mark"><title>prefill: 16 tokens: 10.8 ms</title><path class="s1 bar-mark" d="M250,80H678.0Q682.0,80 682.0,84V106Q682.0,110 678.0,110H250Z"/></g><text class="t-val" x="690.0" y="101.0" text-anchor="start">10.8 ms</text><text class="t-tick" x="250.0" y="146.0" text-anchor="start">Sixteen times the tokens for 8% more time.</text></svg><figcaption>One decode step for 1 token takes 10.0 ms; one pass over 16 tokens takes 10.8 ms. Sixteen times the tokens cost 8% more time.</figcaption></figure>

That suggests a clever move. A cheap helper model could guess the next few tokens. Then the big model checks all the guesses in a single pass, for about the cost of one decode step. If most guesses are right, you get several tokens for the price of one step.

This is **speculative decoding**, and it can be done so that the output is exactly what the big model would have written anyway. It works for the reason this page is about: decode leaves the compute cores idle. Part 4 builds it from scratch.

## Summary

> [!TAKEAWAYS]
> - An LLM writes by running the whole model once per token, in a loop.
> - **Prefill** reads the whole prompt in one pass and sets the **time to first token**. **Decode** writes one token per step and sets the **time per output token**.
> - A GPU has two speed limits: FLOP/s for arithmetic and bandwidth for moving bytes. An H100 can do about 295 FLOPs in the time it moves one byte.
> - Every step reads all the weights. Decode uses that read for 1 token (about 1 FLOP per byte), so it is **memory-bound**. Prefill uses it for 512 tokens, so it is **compute-bound**.
> - Measured: reading 512 tokens took **27 ms**; writing 512 tokens took **5.2 s**.
> - **Batching** shares one weight read across many sequences: 128 sequences gave **44x** the tokens per second for about **3x** the step time.
> - A line of PyTorch becomes a **kernel launch**: the CPU gives the order, and thousands of GPU threads in blocks run it on the **SMs**. At batch 1 our model ran **880** GPU operations per step, at about **3.4 µs** of launch cost each. **CUDA graphs** and **kernel fusion** cut that cost, but not the weight-read floor.
> - On real GPUs, one user of Llama 3.1 8B can get at most about **127** (A100) to **479** (B200) tokens per second, set by bandwidth. The ridge point is **150 to 300** FLOPs per byte, so a large batch is needed to use the compute.
> - The batch stops at two limits: steps get slower, and every sequence needs memory for its notes, the **KV cache**. On real servers, memory is usually the tighter one.

<details>
<summary>The benchmark script</summary>

```python
"""Prefill vs decode, measured. Qwen2.5-0.5B (BF16) on a GPU via PyTorch.
Random token ids: the content of the text does not change how long the math takes."""
import statistics, time
import torch
from transformers import AutoModelForCausalLM, DynamicCache

DEV = "mps"                      # "cuda" on an NVIDIA GPU
sync = torch.mps.synchronize     # torch.cuda.synchronize on NVIDIA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16).to(DEV).eval()
V = model.config.vocab_size

def ids(b, n):
    return torch.randint(0, V, (b, n), device=DEV)

@torch.inference_mode()
def prefill(x):
    out = model(input_ids=x, past_key_values=DynamicCache(), logits_to_keep=1)
    return out.logits[:, -1:].argmax(-1), out.past_key_values

@torch.inference_mode()
def decode_step_ms(batch, context, steps=24):
    tok, cache = prefill(ids(batch, context))
    times = []
    for _ in range(steps):
        sync(); t = time.perf_counter()
        out = model(input_ids=tok, past_key_values=cache, logits_to_keep=1)
        tok, cache = out.logits[:, -1:].argmax(-1), out.past_key_values
        sync(); times.append(time.perf_counter() - t)
    return statistics.median(times[3:]) * 1000

x = ids(1, 512)
prefill(x); sync()
t = time.perf_counter(); prefill(x); sync()
print(f"prefill 512 tokens: {(time.perf_counter() - t) * 1000:.1f} ms")
for b in (1, 8, 32, 64, 128):
    ms = decode_step_ms(b, 512)
    print(f"batch {b:3d}: {ms:6.2f} ms per step, {b / ms * 1000:7.0f} tokens/s")
```

The roofline, batching and memory numbers on this page come from `code/writings/llm-inference/p1_roofline.py`. It counts the model's weights and works out each number from the measured step times.

The kernel count and the launch cost come from `p1_kernels.py`, run on the same Apple GPU. The GPU table and the decode floors come from `p1_gpus.py`, which only does arithmetic on the vendor specs. Their outputs:

```text
device: mps, torch 2.14.1
one decode step (batch 1, 512-token context): 1328 PyTorch operations, 448 of them views or no-op type checks, 880 that do work on the GPU
most common working ops: mul 220, linear 169, add 146, cat 97, pow 49, mean 49, rsqrt 49, neg 48, scaled_dot_product_attention 24, silu 24
per layer (24 layers): about 37 working ops
tiny op 'add on 16 numbers': 3.4 microseconds per op (median of 7 runs of 2,000)
tiny op '16x16 matrix times a vector': 27.7 microseconds per op (median of 7 runs of 2,000)
880 working ops x 3.4 us = 3.0 ms of launch cost per decode step, if each op costs what a tiny op costs
decode step measured now: 15.0 ms (published Part 1 value: 10.0 ms)
```

```text
Llama 3.1 8B weights in BF16: 16.1 GB
GPU              HBM GB   TB/s  TFLOP/s  ridge  floor ms  max tok/s  weights % of HBM
A100 80GB SXM        80   2.04      312    153      7.88        127              20%
H100 SXM             80   3.35      989    295      4.79        209              20%
H200 SXM            141   4.80      989    206      3.35        299              11%
B200 (HGX)          180   7.70     2250    292      2.09        479               9%
MI300X              192   5.30     1307    247      3.03        330               8%
A100 -> B200: compute x7.2, bandwidth x3.8, ridge 153 -> 292
A100 -> H100: compute x3.2, bandwidth x1.6
H100 -> H200: same compute, bandwidth x1.43, ridge 295 -> 206
batch needed to reach the ridge (weights only, about 1 FLOP per byte per sequence): A100 80GB SXM 153, H100 SXM 295, H200 SXM 206, B200 (HGX) 292, MI300X 247
source A100 80GB SXM: nvidia.com/en-us/data-center/a100: 80GB HBM2e, 2,039 GB/s, BF16 312 TFLOPS (624 with sparsity)
source H100 SXM: nvidia.com/en-us/data-center/h100: 80GB, 3.35TB/s, BF16 1,979 TFLOPS with sparsity -> 989 dense
source H200 SXM: nvidia.com/en-us/data-center/h200: 141GB, 4.8TB/s, BF16 1,979 TFLOPS with sparsity -> 989 dense
source B200 (HGX): NVIDIA Blackwell datasheet, HGX B200 per GPU: 180 GB HBM3E, 7.7 TB/s, FP16/BF16 4.5 PFLOPS sparse -> 2,250 dense
source MI300X: amd.com MI300X page and data sheet: 192 GB HBM3, 5.3 TB/s, BF16 1307.4 TFLOPS dense
```

The kernel run measured a decode step of 15.0 ms, not 10.0 ms. The machine was busier during that run than during the earlier measurements, and the tables on this page keep the earlier values.

</details>

## References

1. NVIDIA, [H100 Tensor Core GPU specifications](https://www.nvidia.com/en-us/data-center/h100/) (3.35 TB/s memory bandwidth; 1,979 TFLOPS BF16 with sparsity, so about 989 dense).
2. S. Williams, A. Waterman, D. Patterson. *Roofline: an insightful visual performance model for multicore architectures.* Communications of the ACM, 2009.
3. W. Kwon et al. [*Efficient Memory Management for Large Language Model Serving with PagedAttention*](https://arxiv.org/abs/2309.06180). SOSP 2023.
4. A. Agrawal et al. [*Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve*](https://arxiv.org/abs/2403.02310). OSDI 2024.
5. NVIDIA, [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/archive/12.6.0/cuda-c-programming-guide/index.html) (kernels, thread blocks, warps of 32 threads).
6. NVIDIA, [NVIDIA Hopper Architecture In-Depth](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) (132 SMs in the H100 SXM5, 256 KB register file and 256 KB of shared memory and L1 per SM, 50 MB L2).
7. vLLM documentation: [CUDA Graphs](https://docs.vllm.ai/en/stable/design/cuda_graphs/) (the default mode, `FULL_AND_PIECEWISE`, uses full graphs for decode batches).
8. Specs used in the GPU table: NVIDIA [A100](https://www.nvidia.com/en-us/data-center/a100/) (80 GB, 2,039 GB/s, BF16 312 TFLOPS dense), [H200](https://www.nvidia.com/en-us/data-center/h200/) (141 GB, 4.8 TB/s, BF16 1,979 TFLOPS with sparsity), the [Blackwell datasheet](https://dam-cdn.nvd.orangelogic.com/AssetLink/y441155802qub41q118b2852i557jem5.pdf) (HGX B200 per GPU: 180 GB, 7.7 TB/s, BF16 4.5 PFLOPS with sparsity, dense is half), the [HGX platform page](https://www.nvidia.com/en-us/data-center/hgx/) (B200 NVLink 1.8 TB/s per GPU) and AMD [Instinct MI300X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html) (192 GB, 5.3 TB/s, BF16 1,307.4 TFLOPS dense). The H100's 900 GB/s of NVLink is on its spec page, reference 1.

**Next: [Part 2, The KV Cache](https://ishwarj.com/writings/llm-inference-2-kv-cache/).** Every sequence in the batch carries notes on every token it has seen. Part 2 shows what those notes are, how big they get, and why they decide how many people one GPU can serve.

