How an LLM Writes: Prefill and Decode

Why the first word takes a moment and the rest stream steadily. The two phases of LLM inference, built from the ground up and measured on a real model: a 512-token prompt is read in 27 ms, but writing 512 tokens takes 5.2 seconds.

You type a question into a chatbot and press enter. There is a short pause. Then the first word appears, and the rest of the answer streams out at a steady pace, word after word.

That pause and that stream are not the same thing happening at two speeds. They are two completely different jobs, with opposite physics, done by the same model. Once you see why, almost everything about how LLMs are served, from vLLM's design to why your GPU bill looks the way it does, starts to make sense.

This series builds that picture from the ground up. No prior knowledge of GPUs needed.

Every number in this piece was measured on a real model, Qwen2.5-0.5B, running on an Apple M5 Pro GPU with PyTorch. It is a small model on a laptop, so the absolute numbers are small too. The shapes are the same on a data-centre GPU, and where it matters I will show the equivalent numbers for an NVIDIA H100.

How a model writes one word#

A language model does not see words. It sees tokens: chunks of text, usually a word or part of a word. "The cat sat on the" is five tokens.

What the model does is simple to state. Give it a list of tokens, and it runs them through a stack of layers (24 in our model) and produces one thing at the end: a probability for every token in its vocabulary of being the next one. Pick one, usually a likely one, and you have the next word.

To write a whole answer, you repeat that. Append the new token, run the model again, get the next one. A 500-token answer means 500 trips through the model, one after another. Each trip needs the token from the trip before it, so they cannot run in parallel.

That loop is the entire story of LLM inference. Everything else is making it faster.

Two phases: reading and writing#

The easiest way to see what happens is to follow one real request from start to finish. Say you ask:

What is the capital of France?

That question is about 7 tokens: What, is, the, capital, of, France, ?. Here is what the model does with it, step by step.

Step 0 is special. The whole question already exists. Nothing about it is unknown. So the model can take in all 7 tokens at the same moment, the way you take in a short sentence at a glance instead of letter by letter. Inside the GPU, the 7 tokens travel through every layer side by side, as one block of numbers, and each layer processes all of them in a single operation.

At the end of step 0 the model has done two things. It has predicted the first word of the answer, The. And, while it read, it wrote down a set of notes about each of the 7 tokens, so it never has to read them again. (Those notes are the KV cache. They are the subject of Part 2.)

Every step after that is different. To produce the second word, the model needs to know the first. To produce the third, it needs the second. So each step takes in exactly one new token, the one it just wrote, and produces exactly one more:

That split has names:

• Prefill is step 0: read the whole prompt at once and produce the first token.
• Decode is every step after it: produce one token per step, each one depending on the last.

Why can't the answer be written all at once too?#

Because it does not exist yet. The prompt is known the moment you press enter, so all of it can be processed together. The answer is being invented one word at a time, and the model cannot know word 3 until it has chosen word 2. It is like the difference between reading a finished page, which you can scan in one look, and writing a sentence, which you can only write one word after another.

What the two phases look like in time#

Here is the same idea drawn to scale, using real timings for a 512-token prompt:

These two phases map onto the two numbers every serving dashboard shows, and onto what you feel as a user:

In the France example, TTFT is how long step 0 takes. TPOT is the average length of steps 1 to 4.

The measurement that surprises everyone#

Here is a simple experiment. Take a 512-token prompt. Measure how long the model takes to read it (prefill). Then measure how long it takes to write 512 tokens (decode). Same model, same number of tokens.

Reading took 27 milliseconds. Writing took 5.2 seconds. That is about 190 times longer, for the same number of tokens.

Nothing is broken here. This is how every LLM behaves, on every GPU. And the reason is the most important idea in this whole series.

The model is heavy, the math is light#

Here is an analogy first, then the real numbers.

Picture a chef in a kitchen. To cook anything, the chef needs a huge recipe book, but the book is kept in a storeroom down the hall. And there is a strange rule: every time the chef cooks, they must carry the entire book from the storeroom and read every page, even to make a single dish.

• If 512 orders are all waiting at once, that is fine. One trip to the storeroom, one read of the book, and the chef cooks all 512 dishes while the book is open. That is prefill.
• But if each dish can only be started once the previous one is done, the chef has to make the whole trip once per dish. The cooking takes a moment; the walking takes most of the time. That is decode.

The chef is the GPU's compute cores. The storeroom is the GPU's memory. The recipe book is the model's weights. Now the real numbers.

A model is, physically, a very large pile of numbers: its weights. Our small model has 494 million of them, stored in 16-bit format. That is 0.99 GB. Llama 3.1 8B is 16 GB. The big frontier models are hundreds of gigabytes.

Those weights live in the GPU's memory. The part of the GPU that does arithmetic, the compute cores, sits next to that memory and has to pull the weights in to use them. Every single trip through the model, it has to pull in all of them, every layer, every matrix.

Now think about what each phase does with that trip:

• Prefill pulls the weights in once and uses them for all 512 prompt tokens at the same time.
• Decode pulls the weights in once and uses them for one token. Then it does it all again for the next token.

So decode spends most of its time not calculating, but waiting for weights to arrive, like the chef walking to the storeroom. We can check this with numbers. I measured this GPU's memory speed at 262 GB/s. Reading 0.99 GB at that speed takes at least 3.8 ms. That is the floor: no decode step can be faster than the time it takes to read the model once.

We measured a decode step at 10 ms. So even the floor, the pure weight-reading time, is over a third of each step. Most of the rest is fixed per-step overhead: a small model like this one runs as hundreds of tiny GPU operations launched one after another, and launching each one costs time. You will see the proof of that further down, in the batching measurements: decoding two sequences at once costs only half a millisecond more than one.

Counting work per byte#

There is a clean way to put a number on this. It is called arithmetic intensity: how many calculations you do for each byte you read from memory.

Every weight gets used in about two calculations (a multiply and an add) for each token it processes. A weight is two bytes. So:

• Decode, one sequence: about 2 calculations per 2 bytes, so roughly 1 calculation per byte.
• Prefill, 512 tokens: the same weights are used for 512 tokens, so roughly 512 calculations per byte.

Every GPU has a speed limit for each resource. An NVIDIA H100 can do about 989 trillion 16-bit calculations per second, and read about 3.35 trillion bytes per second from memory. Divide one by the other and you get about 295: below that many calculations per byte, the GPU cannot be kept busy, because the memory cannot feed it fast enough.

That boundary is best drawn as a picture called a roofline:

Decode for a single sequence sits at about 1 calculation per byte, far down the slope. It uses well under 1% of what the GPU can compute. Prefill sits on the roof. That is the 190x gap from the measurement above, explained in one picture.

There is one more detail in the prefill numbers worth noticing:

Short prompts are inefficient (16 tokens take about as long as one decode step, because the fixed overhead dominates). Throughput climbs as the prompt gets long enough to keep the GPU busy, then dips slightly at 8,192 tokens. Most of that dip comes from attention: every token looks at every earlier token, so that part of the work grows with the square of the length. We will meet attention properly in Part 2.

The trick: write for many people at once#

If decode wastes the weight trip on a single token, the fix almost suggests itself: use each trip for many sequences. Instead of one conversation, decode 8, or 64, or 128 at the same time. The weights are pulled in once per step and every sequence in the batch gets its next token.

Here is what that does, measured:

Going from 1 sequence to 128 made each step about 3 times slower but produced about 44 times more tokens. Each user's answer streams a little slower; the GPU does vastly more useful work. This is batching, and it is the single biggest lever in LLM serving.

Why not batch a thousand sequences, then? Two things stop you:

1. Each step gets slower. Past a point, you move off the memory slope and onto the compute roof, and every user waits longer for every token.
2. Each sequence needs its own memory. Every conversation carries notes about everything said so far, and those notes have to sit in GPU memory next to the weights. On a real server, that memory, not compute, is what usually caps the batch.

Those notes are the KV cache. They are what Part 2 is about.

Latency or throughput: pick your trade#

Batching exposes the central trade-off in serving: latency (how fast one user gets their answer) against throughput (how many tokens the whole GPU produces per second).

Every serving system, vLLM included, is a machine for managing this trade: packing as many sequences into each step as memory allows, without letting any one user's experience fall apart.

A hint of a very useful trick#

One measurement from earlier deserves a second look. Prefilling 16 tokens took 10.8 ms. Decoding 1 token took 10.0 ms. Processing 16 tokens at once cost almost the same as processing one.

That suggests a clever move. What if a cheap helper guessed the next several tokens, and the big model checked all the guesses in a single pass? If most guesses are right, you get several tokens for the price of one step. This is speculative decoding, and it can be done so that the output is exactly what the big model would have written anyway. It gets all of Part 4, and it works for precisely the reason this article is about: decode leaves the GPU's compute idle.

Summary#

• An LLM writes by running the whole model once per token, in a loop.
• Prefill reads the whole prompt in one pass. It reuses every weight for many tokens, so it is compute-bound and fast per token. It sets time to first token.
• Decode writes one token per step. Each step reads all the weights for very little work, so it is memory-bound and slow per token. It sets time per output token, and it is where most of the time goes.
• Measured: reading 512 tokens took 27 ms; writing 512 took 5.2 s.
• Batching decodes many sequences per weight trip: 128 sequences gave 44x the throughput for 3x the step time.
• What limits the batch is usually memory for each sequence's notes, the KV cache. That is Part 2.

"""Prefill vs decode, measured. Qwen2.5-0.5B (BF16) on a GPU via PyTorch.
Random token ids: the content of the text does not change how long the math takes."""
import statistics, time
import torch
from transformers import AutoModelForCausalLM, DynamicCache

DEV = "mps"                      # "cuda" on an NVIDIA GPU
sync = torch.mps.synchronize     # torch.cuda.synchronize on NVIDIA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16).to(DEV).eval()
V = model.config.vocab_size

def ids(b, n):
    return torch.randint(0, V, (b, n), device=DEV)

@torch.inference_mode()
def prefill(x):
    out = model(input_ids=x, past_key_values=DynamicCache(), logits_to_keep=1)
    return out.logits[:, -1:].argmax(-1), out.past_key_values

@torch.inference_mode()
def decode_step_ms(batch, context, steps=24):
    tok, cache = prefill(ids(batch, context))
    times = []
    for _ in range(steps):
        sync(); t = time.perf_counter()
        out = model(input_ids=tok, past_key_values=cache, logits_to_keep=1)
        tok, cache = out.logits[:, -1:].argmax(-1), out.past_key_values
        sync(); times.append(time.perf_counter() - t)
    return statistics.median(times[3:]) * 1000

x = ids(1, 512)
prefill(x); sync()
t = time.perf_counter(); prefill(x); sync()
print(f"prefill 512 tokens: {(time.perf_counter() - t) * 1000:.1f} ms")
for b in (1, 8, 32, 64, 128):
    ms = decode_step_ms(b, 512)
    print(f"batch {b:3d}: {ms:6.2f} ms per step, {b / ms * 1000:7.0f} tokens/s")

References#

1. NVIDIA, H100 Tensor Core GPU specifications (3.35 TB/s memory bandwidth; 1,979 TFLOPS BF16 with sparsity, so about 989 dense).
2. S. Williams, A. Waterman, D. Patterson. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 2009.
3. A. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. ("Prefill iterations have high latency but saturate GPU compute... decode iterations have low latency but also low compute utilization.")