How an LLM Writes: Prefill and Decode

Why the first word takes a moment and the rest stream steadily. Prefill, decode, the GPU's two speeds, the roofline and batching, measured on a real model: reading 512 tokens takes 27 ms, writing 512 takes 5.2 s.

You type a question into a chatbot and press enter. There is a short pause. Then the first word appears, and the rest of the answer streams out at a steady pace, word after word.

The pause and the stream are two different jobs. They use the same model, but they hit opposite limits inside the computer. Once you see why, most of what LLM serving systems do, from vLLM's design to the size of a GPU bill, starts to make sense.

Every number here comes from a real model, Qwen2.5-0.5B, on the GPU of an Apple M5 Pro (64 GB) with PyTorch. It is a small model, so the absolute times are small too. The shapes are the same on a data-centre GPU, and where it matters I show the numbers for an NVIDIA H100.

How a model writes one word

A language model does not see words. It sees tokens.

Here is how our model's tokenizer cuts two sentences. Common words are one token each. A long or rare word is cut into pieces the model has seen often.

"The cat sat on the"5 tokensThe785·cat8,251·sat7,578·on389·the279"Tokenization is unbelievably useful"6 tokensToken3,323ization2,022·is374·unbelie38,937vably88,134·useful5,390The dot · marks a space. The number under each token is its id in the vocabulary of 151,665 tokens.
Two sentences cut into tokens by Qwen2.5-0.5B's tokenizer. "The cat sat on the" is five tokens; "Tokenization" and "unbelievably" are each cut into two pieces.

The model's job is narrow. You give it a list of tokens. It runs them through a stack of layers (24 in our model). Out comes one thing: a score for every token in its vocabulary, saying how likely it is to come next. You pick one of the likely tokens, and you have the next piece of text.

The cat sat on thethe model24 layersmatappend the new token, run the whole model againeverything so farone new token
The generation loop. Everything so far goes in, one new token comes out, it is added to the end, and the whole model runs again.

To write a whole answer, you repeat that. Add the new token to the end, run the model again, and get the next one. A 500-token answer needs 500 runs of the model, one after another. Each run needs the token from the run before it, so the runs cannot happen at the same time.

That loop is the whole of LLM inference. Every trick in this series makes the loop faster or makes one GPU run more loops at once.

Two phases: reading and writing

The clearest way to see the two phases is to follow one request from start to finish. Say you ask:

What is the capital of France?

The tokenizer cuts that question into 7 tokens: What, is, the, capital, of, France, ?. Here is what the model does with them, step by step.

step 0 · prefillall 7 prompt tokens at onceWhatisthecapitalofFrance?modelThestep 1 · decode1 token in, 1 outThemodelcapitalstep 2 · decode1 token in, 1 outcapitalmodelisstep 3 · decode1 token in, 1 outismodelParisstep 4 · decode1 token in, 1 outParismodel.Each new token is fed back in as the next step's input. The answer: The capital is Paris.
One request, step by step. Step 0 takes in all seven prompt tokens together and produces the first word. Every later step takes in one word and produces the next.

Step 0 is special. The whole question already exists, so nothing about it is unknown. The model can take in all 7 tokens at the same time, the way you read a short sentence at a glance. Inside the GPU, the 7 tokens travel through every layer side by side, as one block of numbers. Each layer handles all 7 in one operation.

At the end of step 0 the model has done two things. It has picked the first word of the answer, The. And while it read, it wrote down notes about each of the 7 tokens, so that it never has to read them again. Those notes matter a lot for memory, and they come back at the end of this page.

Every step after that is different. To produce the second word, the model needs the first. To produce the third, it needs the second. So each step takes in exactly one new token, the one it just wrote, and produces exactly one more.

StepGoes inComes outPhase
0What is the capital of France? (7 tokens, together)Theprefill
1Thecapitaldecode
2capitalisdecode
3isParisdecode
4Paris.decode

Why can't the answer be written all at once too?

Because the answer does not exist yet. The prompt is complete the moment you press enter, so all of it can be handled together. The answer is invented one token at a time, and the model cannot know word 3 before it has chosen word 2. Reading a finished page is fast because you can take it in at a glance. Writing a sentence is slower because each word waits for the one before it.

What the two phases look like in time

Here are the two phases drawn to scale, with real timings for a 512-token prompt.

timeprefill: all 512 prompt tokens in one pass, 27.4 msprefilldecode step 1: one new token, 10.0 msdecode step 2: one new token, 10.0 msdecode step 3: one new token, 10.0 msdecode step 4: one new token, 10.0 msdecode step 5: one new token, 10.0 msdecode step 6: one new token, 10.0 msdecode step 7: one new token, 10.0 msdecode step 8: one new token, 10.0 msdecode step 9: one new token, 10.0 msdecode step 10: one new token, 10.0 msdecode step 11: one new token, 10.0 msdecode step 12: one new token, 10.0 msdecode step 13: one new token, 10.0 msdecode step 14: one new token, 10.0 ms...time to first token (TTFT)time per output token (TPOT)measured: 512-token prompt, Qwen2.5-0.5B on an Apple M5 Pro. Prefill 27 ms, each decode step 10 ms.
The two phases drawn to scale from real measurements: one wide prefill step, then a long run of narrow decode steps, one for each new token.

The two phases match the two numbers that every serving dashboard shows, and the two things you feel as a user.

MetricWhat you feelSet by
TTFTthe pause before anything appearsprefill (step 0)
TPOThow fast the words stream after thatdecode (every later step)

In the France example, TTFT is the length of step 0. TPOT is the average length of steps 1 to 4. In the 512-token run, TTFT is about 27 ms and TPOT is about 10 ms.

The measurement that surprises everyone

Here is a simple experiment. Take a 512-token prompt and measure how long the model takes to read it (prefill). Then measure how long it takes to write 512 new tokens (decode). It is the same model and the same number of tokens.

read 512 prompt tokens (prefill): 27.4 msread 512 prompt tokens (prefill)27.4 mswrite 512 new tokens (decode): 5,158 mswrite 512 new tokens (decode)5,158 ms
Reading 512 tokens versus writing 512 tokens, measured on Qwen2.5-0.5B. The prefill bar is so short it is barely visible.

Reading took 27 milliseconds. Writing took 5.2 seconds. That is about 190 times longer for the same number of tokens.

Nothing is broken. Every LLM on every GPU behaves like this. To see why, we need to look at what a GPU is made of, at the level that matters here.

What a GPU is made of

First, what is the model, physically? It is a very large set of numbers called weights.

Each weight is stored in 16-bit format, which takes 2 bytes. So our small model is 494 million × 2 bytes = 0.99 GB of numbers. Llama 3.1 8B is about 16 GB. The largest models are hundreds of gigabytes.

For this page, a GPU has two parts that matter: a place that stores numbers and a place that calculates with them.

Between the two runs a link with a fixed speed. So a GPU has two speed limits, and each one has its own unit.

memory (HBM)big, but far from the coresweightsweightsweightsweightsH100: 80 GBmemory bandwidthbytes per second through the linkH100: 3.35 TB/sour M5 Pro: 262 GB/s (measured)compute coresthousands of multiply-add unitsFLOPs per secondH100: 989 trillion FLOP/sTwo speeds decide everything: how fast the cores calculate, and how fast memory feeds them.
A GPU reduced to the two parts that matter here: memory that holds the weights, compute cores that do the arithmetic, and a link between them with a fixed bandwidth.

Now compare the two speeds of the H100. In one second it can do 989 trillion operations, but it can move only 3.35 trillion bytes. Divide one by the other: in the time it takes to bring in one byte, the cores could do about 295 operations. The calculators are far faster than the delivery.

A kitchen with a narrow door

Picture a chef who cooks extremely fast. The chef works at a small counter with room for only a few jars. All the ingredients sit in a big pantry down the hall, behind a narrow door. To cook, the chef must bring jars through the door, use them, and send them back to make room for the next ones.

the pantryevery ingredient jarnarrow doorthe chef at a small counterchefroom for a few jarscooks very fast, but only withwhat is on the counterpantry= GPU memory (HBM)ingredient jars= the model's weightsthe door= memory bandwidthchef's speed= FLOPs per second
The kitchen picture. A fast chef works at a small counter. The ingredient jars wait in a pantry, and everything must pass through a narrow door.

Now the strange rule of this kitchen. Each round of cooking needs every jar in the pantry, once, in order.

With one dish per round, every jar comes through the door for a single dish. The chef waits at the door most of the time. If the chef cooks 512 dishes per round, each jar is used 512 times before it goes back. Now the chef is the busy one.

The mapping is exact. The pantry is HBM, and the jars are the weights. The door is the memory bandwidth, and the chef's speed is FLOP/s. The small counter is the small, fast memory beside the compute cores, which is far too small to keep a whole model. That is why every step must bring the weights in again.

Why decode waits for memory

A layer of the model is mostly a matrix multiplication: a grid of weights times the numbers that describe each token.

A tiny example shows the whole effect. Take a 3 × 3 weight matrix, which is 9 weights. At 2 bytes each, reading it costs 18 bytes. Multiply it by one token's 3 numbers: each weight gets one multiply-add, so the work is 9 × 2 = 18 FLOPs. That is 1 FLOP for every byte read.

Now multiply the same matrix by four tokens at once. You still read the 9 weights once (18 bytes), but each weight now does four multiply-adds: 72 FLOPs, or 4 FLOPs per byte.

A. decode, 1 token120013201weights W×121token=553output9 weights read = 18 bytes9 multiply-adds = 18 FLOPs18 FLOPs / 18 bytes = 1 FLOP per byteB. four tokens at once120013201the same W×1021210112104 tokens=5223573132524 outputs9 weights read = 18 bytes (once)4 × 9 multiply-adds = 72 FLOPs72 FLOPs / 18 bytes = 4 FLOPs per byte
A 3 × 3 weight matrix times one token, and the same matrix times four tokens. The weights are read once in both cases; four tokens get four times the work out of the same read.

Decode is panel A, at a huge scale. It multiplies each weight matrix by one token, which engineers call a matrix-vector multiplication. Prefill is panel B with 512 columns instead of 4: a matrix-matrix multiplication. Both read every weight once per pass. Prefill gets 512 tokens of work out of that read. Decode gets one.

GPU memoryweights: 0.99 GBcompute coresmultiply and addread all 0.99 GB of weightsdecode: 1 tokenread all 0.99 GB of weightsprefill: 512 tokensmemory bandwidth, measured: 262 GB/s
Both phases pay for the same trip through memory. Prefill gets 512 tokens of work out of it. Decode gets one.

We can check this with our measurements. Our GPU moves 262 GB/s, and the weights are 0.99 GB. So reading the weights once takes at least 0.99 ÷ 262 = 3.8 ms. That is a floor: no decode step can be faster than one read of the model. The arithmetic for one token is about 0.99 billion FLOPs, which the GPU's cores could finish in a small fraction of that time.

one decode step, batch 1: 10.0 ms (measured)reading 0.99 GB of weights at 262 GB/s: at least 3.77 msread the weights: 3.8 msthe rest of the step: about 6.2 mseverything else: about 6.2 msthe floor: 0.99 GB ÷ 262 GB/sstarting hundreds of small GPU jobs one after another, and gaps between themThe arithmetic itself, 0.99 GFLOP, is a tiny part: at full speed an H100 would finish it in about 1 microsecond.
Where the 10 ms of one decode step go at batch 1: at least 3.8 ms reading weights, most of the rest fixed overhead, and almost no time on arithmetic.

We measured a decode step at 10 ms. The weight read is over a third of it. Most of the rest is a fixed cost per step. A small model like this one runs as hundreds of small GPU jobs, called kernels, one after another. Each one has a start-up cost, and the cores sit idle between them.

Neither part of the 10 ms is arithmetic. That is the central fact of decode.

The vLLM paper puts the same fact at the centre of its introduction.

Two lines on page 3 of the same paper name the two kinds of multiplication from the 3 × 3 example.

From a Python line to the GPU: CUDA and kernels

The 10 ms step had a large part that was neither reading weights nor arithmetic: the fixed cost of starting hundreds of kernels. To see where that cost comes from, follow one line of Python all the way into the GPU.

On NVIDIA GPUs, the software layer between your program and the chip is called CUDA.

An analogy: a foreman and a crew

Think of a building site. A foreman (the CPU) reads the plan, one line at a time. For each line, the foreman shouts one order to a huge crew (the GPU), such as "lay all these bricks". Then the crew splits into teams, and the teams work at the same time. The foreman does no building at all. The foreman only reads the plan and gives orders.

Two facts follow from this picture. Each order takes the foreman a moment to shout, however small the job. And if every job is small, the crew finishes each one before the next order arrives, so the crew stands and waits.

What happens for y = x @ W

Here is the real sequence for one matrix multiplication in PyTorch.

1. on the CPUPython: y = x @ WPyTorch picks a GPU kernellaunch the kernel:its code, its inputs andhow many blocks of threadslaunch2. on the GPU: the blocks of threads are spread over the SMsSM 0registersshared memory(on-chip SRAM)SM 1registersshared memory(on-chip SRAM)SM 2registersshared memory(on-chip SRAM)SM 131registersshared memory(on-chip SRAM)...L2 cache, shared by all SMs3. read W and x4. write y backHBM: the GPU's main memory (the weights live here)H100: 80 GB at 3.35 TB/sThe CPU only gives orders.All the arithmetic happenson the GPU.one block of threads(many threads, run in warps of 32)
One line of PyTorch on an NVIDIA GPU. The CPU launches a kernel. Its blocks of threads are spread over the streaming multiprocessors (SMs). The threads read W and x from HBM through the L2 cache into their own fast on-chip memory, do the multiply-adds, and write y back to HBM.
  1. The CPU picks a kernel. PyTorch sees a matrix multiplication of two GPU tensors. It chooses a matrix-multiply kernel from NVIDIA's libraries, sized for these shapes.
  2. The CPU launches it. It sends the GPU the kernel's code, the memory addresses of x, W and y, and how many threads to start. Then the CPU moves on at once. It does not wait for the answer.
  3. The GPU spreads the work. The threads come in groups called thread blocks. The GPU hands the blocks out to its streaming multiprocessors. An H100 has 132 of them. Each block computes one tile of y.
  4. Each block reads, computes and writes. Its threads copy their part of W and x from HBM into fast memory on the chip. They do the multiply-adds there, and they write their tile of y back to HBM.

The CUDA programming guide says it in two sentences. A kernel, "when called", is "executed N times in parallel by N different CUDA threads". And when the CPU launches it, "the blocks of the grid are enumerated and distributed to multiprocessors with available execution capacity".

The memory ladder

Step 4 hides an important detail. A GPU has several kinds of memory, and they differ in size by a factor of thousands. The kitchen's "small counter" from earlier is really a ladder of memories.

The memories of an H100, from the smallest and fastest to the largest (log scale)registers256 KB per SMregisters: 256 KB per SM34 MB in all, inside each SM, the fastestshared memory + L1256 KB per SMshared memory + L1: 256 KB per SM34 MB in all, inside each SML2 cache50 MBL2 cache: 50 MBon the chip, shared by all 132 SMsHBM80 GBHBM: 80 GBbeside the chip, 3.35 TB/sLlama 3.1 8B weights: 16.1 GB1 MB100 MB10 GB100 GBOnly HBM is big enough for the model, so every decode step must bring the weights from HBM again.
The memories of an H100 on a log scale. Registers and shared memory sit inside each SM and add up to tens of megabytes. The L2 cache is 50 MB. HBM, 80 GB, is the only level big enough to hold a model.
MemoryWhere it isSize on an H100 SXM
registersinside each SM256 KB per SM
shared memory and L1 cacheinside each SM256 KB per SM, together
L2 cacheon the chip, shared by all SMs50 MB
HBMmemory chips beside the GPU80 GB, at 3.35 TB/s

The on-chip memories are much faster than HBM, but they are tiny. All 132 SMs together have about 34 MB of registers and about 34 MB of shared memory. Llama 3.1 8B needs 16.1 GB. So the weights cannot stay on the chip between steps. Every decode step must pull them from HBM again, at 3.35 TB/s, which is the floor this page measured.

Launch overhead: why small steps are slow

Now back to the foreman. A launch takes the CPU a few microseconds. The work of a kernel at batch 1 is often smaller than that. Then the GPU finishes a kernel and waits for the next order.

We counted the operations in one decode step of our model. A step at batch 1 runs 880 GPU operations, about 37 per layer. Then we timed a tiny operation, an add on 16 numbers, repeated 2,000 times. Each one took 3.4 microseconds, with almost no arithmetic inside.

So the launches alone cost about 880 × 3.4 µs = 3.0 ms per step. Add the 3.8 ms floor for reading the weights, and about 6.8 ms of the 10 ms step is explained. The rest is time that these two numbers do not cover. Some kernels cost more than a tiny add to start, and each one does some work too.

without a CUDA graph: one launch per kernelCPUGPUgapswith a CUDA graph: one launch replays the whole recorded stepCPUGPUreplaykernels back to backCPU launch worka kernel running on the GPUSmall kernels finish faster than the CPU can launch the next one, so the GPU waits in the gaps.
Two timelines of one decode step. Without a CUDA graph, the CPU launches each kernel in turn, and the GPU waits in the gaps. With a CUDA graph, one launch replays the whole recorded step, and the kernels run back to back.

This cost does not grow with the batch. A batch of 64 runs the same 880 operations, each on more numbers. That is part of the "fixed part" of the step that you will meet in the batching section below.

Two fixes: CUDA graphs and kernel fusion

CUDA graphs remove most of the launch cost. The engine runs one decode step once while CUDA records every kernel and its inputs. That record is a graph.

From then on, the CPU launches the whole graph with one call, and the GPU runs the kernels back to back. A graph works only for fixed shapes, so engines record one graph for each of many batch sizes. By default, vLLM records full CUDA graphs for decode steps. For prefill and mixed steps, it uses graphs for the parts between the attention calls.

Kernel fusion removes trips to HBM. A layer has many small steps, such as a multiply, an add and a normalisation. Each one reads its input from HBM and writes its output back. A fused kernel does all of them in one pass. The numbers in between stay in registers and never go back to HBM. It also means fewer kernels, so fewer launches.

three small kernelsmultiplyaddnormaliseHBM: 3 reads + 3 writesone fused kernelmultiply, add, normaliseHBM: 1 read + 1 writeThe fused kernel keeps the in-between numbers in registers, so they never make the trip to HBM.
Kernel fusion. Three small kernels make three reads and three writes to HBM. One fused kernel does the same three steps with one read and one write, because the in-between numbers stay in registers.

Neither fix changes the floor. The weights must still come from HBM once per step. The fixes only cut the time on top of the floor. The rest of this page is about the floor itself, and about the one trick that spreads it over many users.

Counting work per byte

The 3 × 3 example gives us a number to put on each job. Engineers call it arithmetic intensity.

arithmetic intensity=FLOPs donebytes read from memory\text{arithmetic intensity} = \frac{\text{FLOPs done}}{\text{bytes read from memory}}

Read it left to right: count the arithmetic, count the bytes that had to come from memory, and divide. For a weight matrix the rule is short. Each weight costs 2 bytes and gives 2 FLOPs (one multiply-add) per token. So the intensity is about 1 FLOP per byte for each token that uses the weights in the same pass.

Now a real matrix from our model. Each layer has matrices of 896 × 896 weights, which is 802,816 weights or 1.61 MB. One token through it is 1.61 million FLOPs: 1 FLOP per byte. Four tokens give 6.42 million FLOPs for the same 1.61 MB: 4 FLOPs per byte. A 512-token prefill gives 512 FLOPs per byte.

JobFLOPs per byte (weights)
decode, one sequenceabout 1
prefill, 512 tokensabout 512

Compare those with the H100's balance point of about 295, from the GPU section. Below 295 FLOPs per byte, memory cannot deliver bytes fast enough to keep the cores busy. Above it, memory is fast enough and the cores become the limit. Decode at 1 is far below the line. Prefill at 512 is above it.

The roofline: one picture for both limits

A roofline turns those two limits into one picture. It answers a single question: for a job with a given work per byte, what is the most speed this GPU can give it?

Build it in three steps.

  1. The memory limit. If memory delivers 3.35 trillion bytes per second, and each byte gives you II FLOPs, you cannot do more than 3.35 × II trillion FLOP/s. More work per byte means more speed, so this limit is a line that rises with II.
  2. The compute limit. However many FLOPs per byte you have, the cores cannot go faster than 989 TFLOP/s. That is a flat line, the roof.
  3. The roofline. The real limit is the lower of the two. It rises along the slope, then flattens at the roof. The bend, where the two lines meet, is the ridge point: 989 ÷ 3.35 ≈ 295 FLOPs per byte.
1. memory limit1101001,0001101001,000speed = bandwidth× FLOPs per byteFLOPs per byte2. compute limit1101001,0001101001,000the cores cannot go fasterthan 989 TFLOP/sFLOPs per byte3. the roofline: the lower of the two1101001,0001101001,000ridge 295memory-boundcompute-boundFLOPs per byteVertical axes: TFLOP/s the GPU can reach. Both axes use a log scale. Numbers: an NVIDIA H100 (BF16).
The roofline built in three steps for an H100: a sloped memory limit, a flat compute limit, and the lower of the two, which bends at the ridge point.
highest speed=min⁡(peak FLOP/s, bandwidth×I)\text{highest speed} = \min\left(\text{peak FLOP/s},\ \text{bandwidth} \times I\right)

Read it left to right. The speed a job can reach is the smaller of two numbers: the roof, and the bandwidth times the work per byte. A worked example for decode at I=1I = 1 on an H100: 3.35 TB/s × 1 = 3.35 TFLOP/s, and the roof is 989. The smaller one is 3.35 TFLOP/s, which is 0.3% of what the GPU can compute. For prefill at I=512I = 512: 3.35 × 512 = 1,715, which is above the roof, so the limit is the full 989 TFLOP/s.

1101001,0001416642951,0244,096compute roof: 989 TFLOP/smemory slope: 3.35 TB/s × FLOPs per bytedecode, batch 1: about 1 FLOPs per byte, at most 3 TFLOP/s (0.3% of the roof)decode, batch 1prefill, 512 tokens: about 512 FLOPs per byte, at most 989 TFLOP/s (100.0% of the roof)prefill, 512 tokensFLOPs per byte read from memory (log scale)TFLOP/s
A roofline for an H100. On the left slope, speed is capped by memory. On the flat roof, it is capped by compute. Decode at batch 1 sits at the bottom left; prefill sits on the roof.

The roofline gives only a ceiling. Real jobs sit below it, because of the fixed costs per step that we saw above. Our M5 Pro shows the same pattern in what it really did:

Job (measured on the M5 Pro)FLOPs per byteReal speed
decode, one sequence, 10.0 ms per step10.10 TFLOP/s
prefill, 512 tokens in 27.4 msabout 37113.4 TFLOP/s

Prefill did over 100 times more arithmetic per second on the same chip. (The prefill intensity here is 371, not 512, because prefill only needs the large output layer for the last token. The script in the research notes counts both.) That gap is the 190x gap from the measurement above, seen as a picture.

Research notes: how the roofline numbers were counted

The FLOP counts use the usual rule of 2 FLOPs per weight per token. Our model has 494,032,768 weights. Of these, 136,134,656 are the token embedding, which the model also uses as its output layer. A decode step uses every weight once per token, so it does 0.988 GFLOP per token. Prefill keeps the output only for the last prompt token, so its other tokens skip the output layer: 0.716 GFLOP per token. The counts leave out attention, which is small at these lengths.

The H100 numbers come from NVIDIA's specification: 3.35 TB/s of memory bandwidth and 1,979 TFLOP/s of BF16 with sparsity. For the dense matrices that LLMs use, that is about 989 TFLOP/s. The M5 Pro's bandwidth, 262 GB/s, was measured for this series. The script p1_roofline.py prints every number in this section.

Prompt length changes the prefill side too. Here is prefill on our model for prompts from 16 to 8,192 tokens.

Prompt lengthPrefill timeTokens per second
1610.8 ms1,484
25617.7 ms14,483
1,02447.9 ms21,371
2,04895.0 ms21,551
8,192450.5 ms18,185

Short prompts are slow per token. A 16-token prefill takes about as long as one decode step, because the fixed cost per step is larger than the work. As the prompt grows, the GPU gets enough work per weight to stay busy, and the rate climbs to about 21,500 tokens per second. At 8,192 tokens it dips a little.

Most of that dip comes from attention, the step where every token looks at every earlier token. That part of the work grows with the square of the length. Part 2 explains attention.

A paper on scheduling these two phases, Sarathi-Serve, states the same split in one sentence and draws it from its own measurements.

The fix: write for many conversations at once

Decode wastes each trip to memory on a single token. The fix follows from the 3 × 3 example: put more tokens into each trip. A real server has many users at the same time, and each of them needs a next token. So instead of one decode step per user, run one step for all of them together.

The bus

Think of a bus between two towns. The trip costs the same fuel and the same 10 minutes whether one passenger rides or forty. With one passenger, almost all of that cost is wasted. With forty, the same trip carries forty people.

one passengerone trip: 10.0 msfour passengersone trip: 11.1 msthe bus = one read of all the weightsa passenger = one sequence that needs its next token1 token per trip: 100 tokens/s4 tokens per trip: 361 tokens/s
A bus trip costs about the same with one passenger or four. One read of the weights is the trip; each sequence that needs a next token is a passenger.

In decode, the trip is one read of all the weights from memory. The passengers are the sequences in the batch. The read costs about the same whether it serves one sequence or four, because, as we saw, the arithmetic is the cheap part.

One decode step, batch 1 and batch 4

Here is what happens inside one step when four users share it. Each user is at a different point in a different conversation.

batch 110.0 msA: The capital isread all weights0.99 GB, onceParis1 token outbatch 411.1 ms1. stack the newest tokenof each sequence2. read each weight once,use it for all four3. each sequence getsits own next tokenA: The capital isParisB: The sky isblueC: 2 + 2 =4D: def hello(read all weights0.99 GB, once4 tokens outfor 1.1 ms moreThe four sequences do not need to be the same length or the same topic. They only share the trip to memory.
One decode step for one sequence and for four. With four, the newest token of each sequence is stacked into a small matrix, every weight is read once, and four next tokens come out.
  1. Stack the newest tokens. The step takes the latest token of each of the four sequences and puts their numbers side by side, one column per sequence. This is panel B of the 3 × 3 example.
  2. Read each weight once and use it four times. Each weight matrix comes from memory once and is multiplied by all four columns. The arithmetic is four times larger, but it was the cheap part.
  3. Each sequence gets its own next token. The output has one column per sequence. Each column becomes that sequence's next token, which is added to the end of its own text.

The sequences do not share any text. User A's answer about Paris never mixes with user B's answer about the sky. Each column is computed only from its own sequence. They share only the trip to memory.

Measured on our model, one step for one sequence takes 10.0 ms. One step for four sequences takes 11.1 ms. The GPU makes 4 tokens in 11.1 ms instead of 1 token in 10.0 ms. That is 361 tokens per second instead of 100, for 1.1 ms more per step.

The vLLM paper describes the same effect in its background section.

The batch, measured from 1 to 128

Here is what batching does on our model, from 1 sequence to 128. Every sequence has a 512-token context.

01,0002,0003,0004,0005,000batch 1: 100 tokens/s, 10.0 ms per step99.61batch 2: 190 tokens/s, 10.6 ms per step1902batch 4: 361 tokens/s, 11.1 ms per step3614batch 8: 670 tokens/s, 11.9 ms per step6708batch 16: 1,247 tokens/s, 12.8 ms per step1,24716batch 32: 2,120 tokens/s, 15.1 ms per step2,12032batch 64: 3,242 tokens/s, 19.7 ms per step3,24264batch 128: 4,367 tokens/s, 29.3 ms per step4,367128sequences decoded together (batch size)tokens per second
Tokens per second as more sequences are decoded together. Hover a bar to see the time per step.
Sequences at onceTime per stepTokens per second (all users)Tokens per second (each user)
110.0 ms100100
210.6 ms19094
411.1 ms36190
811.9 ms67084
1612.8 ms1,24778
3215.1 ms2,12066
6419.7 ms3,24251
12829.3 ms4,36734

Read the table one row at a time.

  • 1 to 2. The step gets 0.6 ms longer, and the GPU makes almost twice as many tokens. The second passenger rides almost free.
  • 2 to 8. Each doubling adds less than a millisecond. Eight sequences make 670 tokens per second, 6.7 times one sequence, while each step is only 19% longer.
  • 8 to 32. The step grows from 11.9 to 15.1 ms. Tokens per second still more than triple, from 670 to 2,120.
  • 32 to 64. The step grows by 4.6 ms. Tokens per second rise by about half, to 3,242. Each new sequence now costs a little time.
  • 64 to 128. The step grows by 9.6 ms, to 29.3 ms. Doubling the batch adds only 35% more tokens per second, to 4,367.

From 1 to 128 sequences, each step became about 3 times slower but produced about 44 times more tokens. Each user's answer streams more slowly, from 100 to 34 tokens per second, but the GPU does far more useful work. Batching is the single biggest lever in LLM serving.

Where the step time goes

The step times follow a simple pattern. Each step costs a fixed part of about 10.6 ms, plus about 0.146 ms for each sequence in the batch. This line, fitted to the measurements from batch 8 to 128, matches every measured step from batch 4 to 128 within 0.2 ms.

0 ms10 ms20 ms30 msbatch 1: fixed part 10.6 msbatch 1: part per sequence 0.1 msbatch 1: measured 10.0 ms10.0 ms1batch 4: fixed part 10.6 msbatch 4: part per sequence 0.6 msbatch 4: measured 11.1 ms11.1 ms4batch 8: fixed part 10.6 msbatch 8: part per sequence 1.2 msbatch 8: measured 11.9 ms11.9 ms8batch 32: fixed part 10.6 msbatch 32: part per sequence 4.7 msbatch 32: measured 15.1 ms15.1 ms32batch 64: fixed part 10.6 msbatch 64: part per sequence 9.3 msbatch 64: measured 19.7 ms19.7 ms64batch 128: fixed part 10.6 msbatch 128: part per sequence 18.7 msbatch 128: measured 29.3 ms29.3 ms128sequences in the batchfixed part, 10.6 ms (weights + overhead)0.146 ms per sequencemeasured
Decode step time split into a fixed part and a part that grows with each sequence. The dots are the measured step times; the bars are the fitted line, 10.6 ms plus 0.146 ms per sequence.

The fixed part is the trip: reading the weights plus the start-up costs of the kernels. Every step pays it once, whatever the batch size. The part per sequence is the work that belongs to one sequence: its arithmetic, and reading its own notes from memory. At batch 4 the fixed part is 95% of the step, so the extra sequences are nearly free. At batch 128 the part per sequence is 18.6 ms, which is larger than the fixed part.

The same line shows the end of the road. When the batch grows without limit, the fixed part stops mattering, and each token costs 0.146 ms. So this GPU can never make more than about 6,865 tokens per second on this model, however large the batch. At 128 sequences it is already at 4,367.

Batching on the roofline

On the roofline, batching moves decode to the right. Every sequence in the batch adds about one FLOP of work for each byte of weights read. So a batch of 64 has about 64 FLOPs per byte, and the highest speed rises with it.

1101001,0001416642951,0244,096compute roof: 989 TFLOP/smemory slope: 3.35 TB/s × FLOPs per bytebatch 1: about 1 FLOPs per byte, at most 3 TFLOP/s (0.3% of the roof)batch 1batch 4: about 4 FLOPs per byte, at most 13 TFLOP/s (1.4% of the roof)batch 4batch 16: about 16 FLOPs per byte, at most 54 TFLOP/s (5.4% of the roof)batch 16batch 64: about 64 FLOPs per byte, at most 214 TFLOP/s (21.7% of the roof)batch 64batch 128: about 128 FLOPs per byte, at most 429 TFLOP/s (43.4% of the roof)batch 128prefill, 512 tokens: about 512 FLOPs per byte, at most 989 TFLOP/s (100.0% of the roof)prefill, 512 tokensFLOPs per byte read from memory (log scale)TFLOP/sWeights only. Each sequence also reads its own notes (Part 2), which pulls real points lower.
Decode climbs the memory slope of the H100 roofline as the batch grows. Each sequence in the batch adds about one FLOP per byte of weights.

On an H100, batch 1 can use at most 0.3% of the compute roof. Batch 64 can reach 21.7%, and batch 128 can reach 43.4%. Decode only reaches the roof near batch 295. Real points sit lower than these ceilings, because each sequence also reads its own notes, which adds bytes without adding weight reuse.

Real data-centre GPUs

So far the only big GPU on this page was the H100. A new inference engineer meets several others: the older A100, the H200, the newer B200 and AMD's MI300X. Their spec sheets list dozens of numbers. For decode, three of them matter, and you already know what each one means.

  1. Memory capacity (GB). How big a model fits, and how much room is left for the notes of each conversation.
  2. Memory bandwidth (TB/s). How fast one decode step can read the weights. This sets the floor of every step.
  3. Peak compute (TFLOP/s). The roof of the roofline. Decode reaches it only with a very large batch.

Here are the five GPUs, from each vendor's own spec page or datasheet. BF16 (bfloat16) is the 16-bit number format that the weights on this page use. The ridge point is the compute divided by the bandwidth, as in the roofline section.

GPUMemoryBandwidthDense BF16 computeRidge point (FLOPs per byte)
NVIDIA A100 80GB SXM80 GB2.04 TB/s312 TFLOP/s153
NVIDIA H100 SXM80 GB3.35 TB/s989 TFLOP/s295
NVIDIA H200 SXM141 GB4.8 TB/s989 TFLOP/s206
NVIDIA B200, HGX180 GB7.7 TB/s2,250 TFLOP/s292
AMD Instinct MI300X192 GB5.3 TB/s1,307 TFLOP/s247

A worked example: one user on each GPU

Take Llama 3.1 8B in BF16, 16.1 GB of weights. At batch 1, every decode step must read all 16.1 GB. So the least time one step can take is the weight size divided by the bandwidth. On an H100: 16.1 GB ÷ 3.35 TB/s = 4.8 ms. One step gives one token, so one user can get at most 1,000 ÷ 4.8 = 209 tokens per second.

least time per step=weight bytesbandwidth,most tokens/s for one user=1least time per step\text{least time per step} = \frac{\text{weight bytes}}{\text{bandwidth}}, \qquad \text{most tokens/s for one user} = \frac{1}{\text{least time per step}}

Read it left to right: bytes to move, divided by bytes per second, gives seconds. The same arithmetic on every GPU:

GPULeast time per decode stepMost tokens/s for one user
A100 80GB7.9 ms127
H100 SXM4.8 ms209
H200 SXM3.4 ms299
B2002.1 ms479
MI300X3.0 ms330
one user, Llama 3.1 8B, batch 1the most tokens/s: bandwidth ÷ 16.1 GBA100A100 80GB SXM: 127 tok/s127 tok/sH100 SXMH100 SXM: 209 tok/s209 tok/sH200 SXMH200 SXM: 299 tok/s299 tok/sB200B200 (HGX): 479 tok/s479 tok/sMI300XMI300X: 330 tok/s330 tok/sridge pointFLOPs per byte = TFLOP/s ÷ TB/sA100A100 80GB SXM: 153153H100 SXMH100 SXM: 295295H200 SXMH200 SXM: 206206B200B200 (HGX): 292292MI300XMI300X: 247247Vendor specs: dense BF16, no sparsity. Upper bounds from bandwidth alone; real steps are slower.
Left: the most tokens per second one user can get from Llama 3.1 8B at batch 1, which is set by bandwidth alone. Right: each GPU's ridge point, about the batch size where decode stops being limited by memory.

These are upper limits. Real steps are slower, because of the launch costs and the other work from the CUDA section. But the order holds. For one user, the GPU with more bandwidth answers faster, and its compute hardly matters. The H200 has exactly the H100's compute, and it is 43% faster for one user only because its memory is faster.

Why the ridge stays high

Look at the last column of the first table. From the A100 to the B200, compute grew 7.2 times, while bandwidth grew 3.8 times. So the ridge point almost doubled, from 153 to 292 FLOPs per byte. It does not rise with every chip. The H200 is an H100 with faster memory, so its ridge fell to 206. But over the generations, arithmetic has become cheaper faster than memory traffic.

Each sequence in the batch adds about one FLOP per byte of weights. So the ridge point is about the batch size at which decode stops waiting for memory. On an A100, that is a batch of about 150. On an H100 or a B200, it is about 300. A newer GPU leaves even more of its compute idle at a small batch, so batching matters more with each generation.

Many GPUs in one server

Data-centre GPUs often come eight to a server, as in NVIDIA's DGX systems. The eight GPUs connect through NVLink, NVIDIA's fast link between GPUs, so that one GPU can read another's memory quickly.

one server: 8 GPUs joined by NVLink switchesGPU 080 GBGPU 180 GBGPU 280 GBGPU 380 GBGPU 480 GBGPU 580 GBGPU 680 GBGPU 780 GBNVLink switch: every GPU talks to every other GPUH100: 900 GB/s of NVLink per GPU. B200: 1.8 TB/s. Both are far faster than the PCIe link to the CPU.
An 8-GPU H100 server. Every GPU connects to NVLink switches, so any GPU can exchange data with any other at high speed.

A model that does not fit on one GPU is split across several of them. The GPUs then exchange partial results over NVLink in every layer. Llama 3.1 70B, for example, has 141 GB of weights in BF16, more than an 80 GB H100 can hold. Part 7 shows how the split works and what it costs.

Why the batch cannot grow forever

If one more passenger is almost free, why not run a thousand sequences in every step? Two limits stop you.

Limit 1: every step gets slower

Every user in the batch waits for the whole step. At batch 1, a user gets a new token every 10.0 ms. At batch 128, every user waits 29.3 ms per token. That is still fast enough to read, but there is a cliff. Once the part per sequence is larger than the fixed part, each new sequence slows down everyone already in the batch.

On a large GPU the same thing happens at the roof. Past the ridge point, a step has more arithmetic than the memory read can hide. Then the step time grows in step with the batch, and tokens per second stop growing. The bus is full: one more passenger needs one more trip.

Limit 2: every sequence needs its own notes

Remember the notes from prefill. Each sequence keeps notes on every token it has read or written, so that it never has to read those tokens again. Those notes are different for every conversation, so the batch cannot share them the way it shares the weights. They must all sit in GPU memory, next to the weights, for as long as the conversation runs.

For our small model, each token's notes take 12 KiB. A batch of 128 sequences with 512 tokens each needs 128 × 512 × 12 KiB = 768 MiB of notes, which is 0.81 GB. The weights take 0.99 GB. So at batch 128, the notes are already almost as large as the whole model.

For a larger model the notes are far larger. Llama 3.1 8B needs 128 KiB of notes per token. On an 80 GB H100, its weights take 16.1 GB, which leaves 63.9 GB. A 4,096-token conversation needs 512 MiB of notes, so at most 119 such conversations fit, even before any working memory. At the model's full 128K-token context, one conversation needs 16 GiB, and only 3 fit.

our run: Qwen2.5-0.5B, 128 sequences of 512 tokensweights: 0.99 GBnotes of 128 sequences: 0.81 GBweights 0.99 GBnotes 0.81 GBEach token's notes take 12 KiB; 128 × 512 tokens × 12 KiB = 0.81 GB, almost as much as the model.Llama 3.1 8B on an 80 GB H100, conversations of 4,096 tokensweights: 16.1 GBconversation 1: 4,096 tokens, 512 MiB of notesconversation 2: 4,096 tokens, 512 MiB of notesconversation 3: 4,096 tokens, 512 MiB of notesconversation 4: 4,096 tokens, 512 MiB of notesconversation 5: 4,096 tokens, 512 MiB of notesconversation 6: 4,096 tokens, 512 MiB of notesconversation 7: 4,096 tokens, 512 MiB of notesconversation 8: 4,096 tokens, 512 MiB of notesconversation 9: 4,096 tokens, 512 MiB of notesconversation 10: 4,096 tokens, 512 MiB of notesconversation 11: 4,096 tokens, 512 MiB of notesconversation 12: 4,096 tokens, 512 MiB of notesconversation 13: 4,096 tokens, 512 MiB of notesconversation 14: 4,096 tokens, 512 MiB of notesconversation 15: 4,096 tokens, 512 MiB of notesconversation 16: 4,096 tokens, 512 MiB of notesconversation 17: 4,096 tokens, 512 MiB of notesconversation 18: 4,096 tokens, 512 MiB of notesconversation 19: 4,096 tokens, 512 MiB of notesconversation 20: 4,096 tokens, 512 MiB of notesconversation 21: 4,096 tokens, 512 MiB of notesconversation 22: 4,096 tokens, 512 MiB of notesconversation 23: 4,096 tokens, 512 MiB of notesconversation 24: 4,096 tokens, 512 MiB of notesconversation 25: 4,096 tokens, 512 MiB of notesconversation 26: 4,096 tokens, 512 MiB of notesconversation 27: 4,096 tokens, 512 MiB of notesconversation 28: 4,096 tokens, 512 MiB of notesconversation 29: 4,096 tokens, 512 MiB of notesconversation 30: 4,096 tokens, 512 MiB of notesconversation 31: 4,096 tokens, 512 MiB of notesconversation 32: 4,096 tokens, 512 MiB of notesconversation 33: 4,096 tokens, 512 MiB of notesconversation 34: 4,096 tokens, 512 MiB of notesconversation 35: 4,096 tokens, 512 MiB of notesconversation 36: 4,096 tokens, 512 MiB of notesconversation 37: 4,096 tokens, 512 MiB of notesconversation 38: 4,096 tokens, 512 MiB of notesconversation 39: 4,096 tokens, 512 MiB of notesconversation 40: 4,096 tokens, 512 MiB of notesconversation 41: 4,096 tokens, 512 MiB of notesconversation 42: 4,096 tokens, 512 MiB of notesconversation 43: 4,096 tokens, 512 MiB of notesconversation 44: 4,096 tokens, 512 MiB of notesconversation 45: 4,096 tokens, 512 MiB of notesconversation 46: 4,096 tokens, 512 MiB of notesconversation 47: 4,096 tokens, 512 MiB of notesconversation 48: 4,096 tokens, 512 MiB of notesconversation 49: 4,096 tokens, 512 MiB of notesconversation 50: 4,096 tokens, 512 MiB of notesconversation 51: 4,096 tokens, 512 MiB of notesconversation 52: 4,096 tokens, 512 MiB of notesconversation 53: 4,096 tokens, 512 MiB of notesconversation 54: 4,096 tokens, 512 MiB of notesconversation 55: 4,096 tokens, 512 MiB of notesconversation 56: 4,096 tokens, 512 MiB of notesconversation 57: 4,096 tokens, 512 MiB of notesconversation 58: 4,096 tokens, 512 MiB of notesconversation 59: 4,096 tokens, 512 MiB of notesconversation 60: 4,096 tokens, 512 MiB of notesconversation 61: 4,096 tokens, 512 MiB of notesconversation 62: 4,096 tokens, 512 MiB of notesconversation 63: 4,096 tokens, 512 MiB of notesconversation 64: 4,096 tokens, 512 MiB of notesconversation 65: 4,096 tokens, 512 MiB of notesconversation 66: 4,096 tokens, 512 MiB of notesconversation 67: 4,096 tokens, 512 MiB of notesconversation 68: 4,096 tokens, 512 MiB of notesconversation 69: 4,096 tokens, 512 MiB of notesconversation 70: 4,096 tokens, 512 MiB of notesconversation 71: 4,096 tokens, 512 MiB of notesconversation 72: 4,096 tokens, 512 MiB of notesconversation 73: 4,096 tokens, 512 MiB of notesconversation 74: 4,096 tokens, 512 MiB of notesconversation 75: 4,096 tokens, 512 MiB of notesconversation 76: 4,096 tokens, 512 MiB of notesconversation 77: 4,096 tokens, 512 MiB of notesconversation 78: 4,096 tokens, 512 MiB of notesconversation 79: 4,096 tokens, 512 MiB of notesconversation 80: 4,096 tokens, 512 MiB of notesconversation 81: 4,096 tokens, 512 MiB of notesconversation 82: 4,096 tokens, 512 MiB of notesconversation 83: 4,096 tokens, 512 MiB of notesconversation 84: 4,096 tokens, 512 MiB of notesconversation 85: 4,096 tokens, 512 MiB of notesconversation 86: 4,096 tokens, 512 MiB of notesconversation 87: 4,096 tokens, 512 MiB of notesconversation 88: 4,096 tokens, 512 MiB of notesconversation 89: 4,096 tokens, 512 MiB of notesconversation 90: 4,096 tokens, 512 MiB of notesconversation 91: 4,096 tokens, 512 MiB of notesconversation 92: 4,096 tokens, 512 MiB of notesconversation 93: 4,096 tokens, 512 MiB of notesconversation 94: 4,096 tokens, 512 MiB of notesconversation 95: 4,096 tokens, 512 MiB of notesconversation 96: 4,096 tokens, 512 MiB of notesconversation 97: 4,096 tokens, 512 MiB of notesconversation 98: 4,096 tokens, 512 MiB of notesconversation 99: 4,096 tokens, 512 MiB of notesconversation 100: 4,096 tokens, 512 MiB of notesconversation 101: 4,096 tokens, 512 MiB of notesconversation 102: 4,096 tokens, 512 MiB of notesconversation 103: 4,096 tokens, 512 MiB of notesconversation 104: 4,096 tokens, 512 MiB of notesconversation 105: 4,096 tokens, 512 MiB of notesconversation 106: 4,096 tokens, 512 MiB of notesconversation 107: 4,096 tokens, 512 MiB of notesconversation 108: 4,096 tokens, 512 MiB of notesconversation 109: 4,096 tokens, 512 MiB of notesconversation 110: 4,096 tokens, 512 MiB of notesconversation 111: 4,096 tokens, 512 MiB of notesconversation 112: 4,096 tokens, 512 MiB of notesconversation 113: 4,096 tokens, 512 MiB of notesconversation 114: 4,096 tokens, 512 MiB of notesconversation 115: 4,096 tokens, 512 MiB of notesconversation 116: 4,096 tokens, 512 MiB of notesconversation 117: 4,096 tokens, 512 MiB of notesconversation 118: 4,096 tokens, 512 MiB of notesconversation 119: 4,096 tokens, 512 MiB of notesweightsEach thin stripe is one conversation's notes (512 MiB). At most 119 fit, before any working memory.At the full 128K context, one conversation needs 16 GiB of notes, and at most 3 fit.
GPU memory split between the weights and the notes of every sequence in the batch. The notes grow with the size of the batch and with the length of each conversation.

So on a real server the batch is usually limited by memory for notes, long before the compute roof. The vLLM paper says this in one line.

Latency or throughput: choose your trade

Batching exposes the main trade-off in serving. Latency is how fast one user gets their tokens. Throughput is how many tokens the whole GPU makes per second. The batch size moves one against the other.

what one user sees: tokens/s050100batch 1: 1001batch 2: 942batch 4: 904batch 8: 848batch 16: 7816batch 32: 6632batch 64: 5164batch 128: 3412810034sequences in the batchwhat the GPU makes: tokens/s02,0004,000batch 1: 1001batch 2: 1902batch 4: 3614batch 8: 6708batch 16: 1,24716batch 32: 2,12032batch 64: 3,24264batch 128: 4,3671281004,367sequences in the batch
As the batch grows from 1 to 128, each user receives tokens about three times more slowly, while the GPU as a whole makes 44 times more.
You care aboutYou wantWhat it costs
one user, fastsmall batcheslow TPOT, but an expensive GPU sits mostly idle
many users, cheaplarge batcheshigh throughput per GPU, but each token arrives later
a quick first wordprefills that are not stuck behind other worka scheduler that stops long prompts from blocking short ones (Part 3)

Every serving system, vLLM included, manages this trade. It packs as many sequences into each step as memory allows, and it keeps every user's TTFT and TPOT within limits.

A hint of a very useful trick

One measurement from earlier deserves a second look. A prefill of 16 tokens took 10.8 ms. A decode step for 1 token took 10.0 ms. Sixteen tokens in one pass cost almost the same as one.

decode: 1 tokendecode: 1 token: 10.0 ms10.0 msprefill: 16 tokensprefill: 16 tokens: 10.8 ms10.8 msSixteen times the tokens for 8% more time.
One decode step for 1 token takes 10.0 ms; one pass over 16 tokens takes 10.8 ms. Sixteen times the tokens cost 8% more time.

That suggests a clever move. A cheap helper model could guess the next few tokens. Then the big model checks all the guesses in a single pass, for about the cost of one decode step. If most guesses are right, you get several tokens for the price of one step.

This is speculative decoding, and it can be done so that the output is exactly what the big model would have written anyway. It works for the reason this page is about: decode leaves the compute cores idle. Part 4 builds it from scratch.

Summary

The benchmark script
python
"""Prefill vs decode, measured. Qwen2.5-0.5B (BF16) on a GPU via PyTorch.
Random token ids: the content of the text does not change how long the math takes."""
import statistics, time
import torch
from transformers import AutoModelForCausalLM, DynamicCache

DEV = "mps"                      # "cuda" on an NVIDIA GPU
sync = torch.mps.synchronize     # torch.cuda.synchronize on NVIDIA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16).to(DEV).eval()
V = model.config.vocab_size

def ids(b, n):
    return torch.randint(0, V, (b, n), device=DEV)

@torch.inference_mode()
def prefill(x):
    out = model(input_ids=x, past_key_values=DynamicCache(), logits_to_keep=1)
    return out.logits[:, -1:].argmax(-1), out.past_key_values

@torch.inference_mode()
def decode_step_ms(batch, context, steps=24):
    tok, cache = prefill(ids(batch, context))
    times = []
    for _ in range(steps):
        sync(); t = time.perf_counter()
        out = model(input_ids=tok, past_key_values=cache, logits_to_keep=1)
        tok, cache = out.logits[:, -1:].argmax(-1), out.past_key_values
        sync(); times.append(time.perf_counter() - t)
    return statistics.median(times[3:]) * 1000

x = ids(1, 512)
prefill(x); sync()
t = time.perf_counter(); prefill(x); sync()
print(f"prefill 512 tokens: {(time.perf_counter() - t) * 1000:.1f} ms")
for b in (1, 8, 32, 64, 128):
    ms = decode_step_ms(b, 512)
    print(f"batch {b:3d}: {ms:6.2f} ms per step, {b / ms * 1000:7.0f} tokens/s")

The roofline, batching and memory numbers on this page come from code/writings/llm-inference/p1_roofline.py. It counts the model's weights and works out each number from the measured step times.

The kernel count and the launch cost come from p1_kernels.py, run on the same Apple GPU. The GPU table and the decode floors come from p1_gpus.py, which only does arithmetic on the vendor specs. Their outputs:

plain text
device: mps, torch 2.14.1
one decode step (batch 1, 512-token context): 1328 PyTorch operations, 448 of them views or no-op type checks, 880 that do work on the GPU
most common working ops: mul 220, linear 169, add 146, cat 97, pow 49, mean 49, rsqrt 49, neg 48, scaled_dot_product_attention 24, silu 24
per layer (24 layers): about 37 working ops
tiny op 'add on 16 numbers': 3.4 microseconds per op (median of 7 runs of 2,000)
tiny op '16x16 matrix times a vector': 27.7 microseconds per op (median of 7 runs of 2,000)
880 working ops x 3.4 us = 3.0 ms of launch cost per decode step, if each op costs what a tiny op costs
decode step measured now: 15.0 ms (published Part 1 value: 10.0 ms)
plain text
Llama 3.1 8B weights in BF16: 16.1 GB
GPU              HBM GB   TB/s  TFLOP/s  ridge  floor ms  max tok/s  weights % of HBM
A100 80GB SXM        80   2.04      312    153      7.88        127              20%
H100 SXM             80   3.35      989    295      4.79        209              20%
H200 SXM            141   4.80      989    206      3.35        299              11%
B200 (HGX)          180   7.70     2250    292      2.09        479               9%
MI300X              192   5.30     1307    247      3.03        330               8%
A100 -> B200: compute x7.2, bandwidth x3.8, ridge 153 -> 292
A100 -> H100: compute x3.2, bandwidth x1.6
H100 -> H200: same compute, bandwidth x1.43, ridge 295 -> 206
batch needed to reach the ridge (weights only, about 1 FLOP per byte per sequence): A100 80GB SXM 153, H100 SXM 295, H200 SXM 206, B200 (HGX) 292, MI300X 247
source A100 80GB SXM: nvidia.com/en-us/data-center/a100: 80GB HBM2e, 2,039 GB/s, BF16 312 TFLOPS (624 with sparsity)
source H100 SXM: nvidia.com/en-us/data-center/h100: 80GB, 3.35TB/s, BF16 1,979 TFLOPS with sparsity -> 989 dense
source H200 SXM: nvidia.com/en-us/data-center/h200: 141GB, 4.8TB/s, BF16 1,979 TFLOPS with sparsity -> 989 dense
source B200 (HGX): NVIDIA Blackwell datasheet, HGX B200 per GPU: 180 GB HBM3E, 7.7 TB/s, FP16/BF16 4.5 PFLOPS sparse -> 2,250 dense
source MI300X: amd.com MI300X page and data sheet: 192 GB HBM3, 5.3 TB/s, BF16 1307.4 TFLOPS dense

The kernel run measured a decode step of 15.0 ms, not 10.0 ms. The machine was busier during that run than during the earlier measurements, and the tables on this page keep the earlier values.

References

  1. NVIDIA, H100 Tensor Core GPU specifications (3.35 TB/s memory bandwidth; 1,979 TFLOPS BF16 with sparsity, so about 989 dense).
  2. S. Williams, A. Waterman, D. Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 2009.
  3. W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
  4. A. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
  5. NVIDIA, CUDA C++ Programming Guide (kernels, thread blocks, warps of 32 threads).
  6. NVIDIA, NVIDIA Hopper Architecture In-Depth (132 SMs in the H100 SXM5, 256 KB register file and 256 KB of shared memory and L1 per SM, 50 MB L2).
  7. vLLM documentation: CUDA Graphs (the default mode, FULL_AND_PIECEWISE, uses full graphs for decode batches).
  8. Specs used in the GPU table: NVIDIA A100 (80 GB, 2,039 GB/s, BF16 312 TFLOPS dense), H200 (141 GB, 4.8 TB/s, BF16 1,979 TFLOPS with sparsity), the Blackwell datasheet (HGX B200 per GPU: 180 GB, 7.7 TB/s, BF16 4.5 PFLOPS with sparsity, dense is half), the HGX platform page (B200 NVLink 1.8 TB/s per GPU) and AMD Instinct MI300X (192 GB, 5.3 TB/s, BF16 1,307.4 TFLOPS dense). The H100's 900 GB/s of NVLink is on its spec page, reference 1.

Next: Part 2, The KV Cache. Every sequence in the batch carries notes on every token it has seen. Part 2 shows what those notes are, how big they get, and why they decide how many people one GPU can serve.