How vLLM Serves Thousands: Paging, Batching and Caching
How vLLM stores the KV cache in small pages, shares and reuses them, and schedules every step: from 48 to 415 requests on one GPU, pauses cut from 463 ms to 40 ms, a 7x faster first token.
One GPU can hold the notes of a few hundred conversations, or of a few dozen. The hardware is the same in both cases. The difference is only in how the notes are stored.
Part 1 showed that decode is limited by memory traffic. So a GPU earns its keep only when it decodes many sequences in one step, as a batch. Part 2 showed what limits the batch. Every sequence keeps a KV cache, and it grows with every token. Storing it as one reserved slab wastes most of the memory. The vLLM paper measured older systems using only 20.4% to 38.2% of their KV memory for real tokens.
This part shows how vLLM fixes that, one idea at a time. The first idea is about 60 years old, and it comes from the operating system on your laptop.
A lesson from operating systems
Your laptop runs dozens of programs at once. Each one asks for memory while it runs, and none of them knows in advance how much it will need. A browser tab can use 50 MB or 2 GB. This is the same problem that Part 2 ended with: something grows, nobody knows how far, and the memory is shared.
Operating systems solved it in the 1960s with virtual memory and paging. Before the mechanism, an analogy.
Think of a ring binder with loose pages and a contents page at the front. The report's pages do not need to sit next to each other in the binder. Page 1 of the report can be the 5th sheet, and page 2 can be the 2nd. The contents page tells you where each one is. When the report gets longer, you take any blank sheet, write on it, and add one line to the contents page. You never had to set aside 200 blank sheets on the first day.
That is paging. Memory is cut into small pieces of one fixed size, called pages. Each program sees its own pages in a neat order: page 0, page 1, page 2. In the real memory chips, each page can sit anywhere. A page table, one per program, records where each page really is.
Now the important part: what happens when a program needs more memory. It asks, and the operating system takes any free frame. The frame becomes the program's next page, and the page table gets one more row. The program does not reserve its maximum up front. It gets memory one page at a time, at the moment it needs it.
Look at the three kinds of waste from Part 2 with this picture in mind. A program never holds memory for data it might write later, because it gets a page only when it writes. Its last page can be partly empty, but that is at most one page. And because every page and every frame is the same size, any free frame fits any request, so small useless gaps cannot form. Keep these three facts in mind. The rest of this section moves them, one to one, onto the GPU.
The same idea for the KV cache
In 2023 a team at UC Berkeley noticed that the KV cache has the same shape of problem. They applied paging to it and called the result PagedAttention. They built a serving engine around it and called it vLLM.
The KV cache is cut into blocks. One block holds the keys and values of a fixed number of tokens: 16 by default. Each request sees its own blocks in order, called logical blocks. Block 0 has tokens 1 to 16, block 1 has tokens 17 to 32, and so on. In GPU memory, each one is stored in a physical block that can be anywhere. A block table, one per request, records which physical block holds each logical one.
At startup, vLLM measures how much GPU memory is left after the model's weights and working memory. It cuts all of that into physical blocks and puts them in a free pool. Take the simulation further down: Llama 3.1 8B on one 80 GB GPU. Its KV memory budget is 51.94 GB, or 24,766 blocks of 16 tokens.
One request, token by token
Now follow one request from its first token. Say the prompt is 30 tokens. Prefill (Part 1) computes the keys and values of all 30 at once. They need room for 30 tokens, so the request gets two blocks from the free pool, say blocks 7 and 1. The first holds tokens 1 to 16. The second holds tokens 17 to 30 and has 2 empty slots.
Decode then adds one token per step. Token 31 goes into an empty slot of block 1, and token 32 fills it. Token 33 has no room, so the request gets one more block from the pool, block 3, and one more entry in its block table. Nothing else moves. Tokens 34 to 48 fill block 3, and token 49 gets block 12.
These are the real outputs of our small allocator, p3_paging.py, which follows the same rule:
== 4. A request with a 30-token prompt grows to 50 tokens (block size 16)
tokens 30: block table [7, 1], last block filled 14/16, empty slots 2
tokens 31: block table [7, 1], last block filled 15/16, empty slots 1
tokens 32: block table [7, 1], last block filled 16/16, empty slots 0
tokens 33: block table [7, 1, 3], last block filled 1/16, empty slots 15
tokens 48: block table [7, 1, 3], last block filled 16/16, empty slots 0
tokens 49: block table [7, 1, 3, 12], last block filled 1/16, empty slots 15
tokens 50: block table [7, 1, 3, 12], last block filled 2/16, empty slots 14The paper walks through the same steps with a block size of 4, so that the picture fits on a page.
Two requests at once look the same. Each has its own block table, and their blocks mix in the same physical pool.
Attention, block by block
There is one catch. Attention (Part 2) compares the new token's query with the key of every earlier token, then mixes their values. A normal attention kernel expects those keys and values in one unbroken array. With paging, they are in pieces.
So vLLM wrote its own attention kernel, which reads the cache through the block table. For each logical block in order, it looks up the physical block number, reads that block's keys, and computes the scores of those 16 tokens. When all blocks are done, it applies the softmax over all the scores together and mixes the values in the same way.
We can check that claim. Our script stores the keys and values of 37 tokens in a pool of 8 blocks, using the block table [7, 1, 3]. Method 1 is our own blockwise attention, following the paper. Method 2 is PyTorch's built-in attention on the same keys and values in one ordinary array:
def paged_attention(q, pool_k, pool_v, block_table, n_tok):
"""Method 1: the paper's blockwise form. Visit blocks in logical order, look up each physical block."""
scores, values = [], []
for j, phys in enumerate(block_table):
filled = min(B, n_tok - j * B) # the last block may be partly filled
Kj = pool_k[phys, :filled] # (filled, d): one lookup, one block
scores.append(Kj @ q / math.sqrt(d)) # q . k for the tokens of this block
values.append(pool_v[phys, :filled])
s = torch.cat(scores)
a = torch.exp(s - s.max()); a = a / a.sum() # softmax over all 37 tokens
return a @ torch.cat(values)== 6. Attention read block by block through a block table equals ordinary attention (CPU, FP32)
block table [7, 1, 3], 37 tokens, block size 16
Method 1 (blocks through the table) vs Method 2 (PyTorch on one contiguous array): max difference 1.5e-07
block lookups for this token: 3; contiguous reads: 1The difference is 0.00000015, which is rounding noise in 32-bit numbers. The last line is the cost we come back to below: three lookups instead of one read.
The three wastes, one at a time
Now we can return to the three kinds of waste from Part 2 and look at each one with blocks. We use one running example. A request has 37 tokens so far. Its answer will end when it reaches 300 tokens, but nobody knows that yet. An older system reserves a slab of 2,048 slots for it, the model's maximum.
Waste 1: reserved slots
The problem. A slot that is reserved now, for a token that will be written later, is empty until then. Other requests cannot use it in the meantime. In our example the slab has 2,048 slots and holds 37 tokens. The answer will reach 300 tokens, so 263 of the empty slots are reserved slots: they will be used, but not yet. Some of them stay empty for hundreds of steps.
With blocks. vLLM gives the request 3 blocks, 48 slots, for its 37 tokens. 11 slots are empty, and the next 11 tokens fill them. The 4th block comes only at token 49. At any moment, a request holds at most one block that is not full.
In bytes, for Llama 3.1 8B, the difference right now is 256 MiB for the slab against 6 MiB for three blocks.
Waste 2: internal fragmentation
The problem. When the answer ends, any slot it never reached was wasted for the whole life of the request. This is internal fragmentation: waste inside a request's own reservation. In our example the answer ends at 300 tokens, so 1,748 of the 2,048 slots were never used. Only 14.6% of the slab ever held data.
With blocks. The request ends with 19 blocks, 304 slots, for 300 tokens. Only the last block has empty slots: 4 of them. The worst case is 15 empty slots, if a request ends with one token in its last block. That is a fixed limit, no matter how long the answer is.
== 3. One request of 37 tokens: a 2,048-slot reservation vs 16-token blocks
contiguous: reserved 2,048 slots, used 37, empty 2,011 (1.8% used)
paged: 3 blocks x 16 = 48 slots, used 37, empty 11 (77.1% used)
if the request will end at 300 tokens: of the 2,011 empty slots now, 263 are reserved slots (filled later) and 1,748 are internal fragmentation (never filled)
at the end, contiguous: 1,748 slots never used (14.6% used); paged: 19 blocks = 304 slots, 4 never used (98.7% used)The paper states this limit in one sentence.
Waste 3: external fragmentation
The problem. Older systems needed each request's slab in one piece. Over time, as requests of different sizes arrive and finish, the free memory breaks up into gaps between slabs. A new request can fail to fit even when the total free memory is large enough.
Here is a tiny memory of 16 slots. Three requests, A, B and C, are still running, and 7 slots are free, in gaps of 2, 3 and 2. Request D needs 6 slots in one piece, and no gap is big enough.
With blocks. Cut the same memory into blocks of 2 slots. D needs 3 blocks, and there are 3 free blocks: 2, 5 and 7. They are not next to each other, but that does not matter, because D's block table records where they are. Every block has the same size, so a free block always fits.
== 5. External fragmentation: 16 slots of memory, three requests finished, one new request
memory AAAA..BBB...CC.. free slots 7, free gaps [2, 3, 2], largest gap 3
a new request needs 6 slots in one piece: fits contiguously? False
with blocks of 2 slots: free blocks 3, the request needs 3: fits? TrueLook again at the bottom of the figure. B has 3 tokens, so its second block holds one token and one empty slot. That empty slot is internal fragmentation, limited to one block, as above. Paging does not make waste zero. It moves all of it into the last block of each request, where it is small and bounded.
All three together
To see the effect at the scale of a real server, I simulated 200,000 requests. The server is one 80 GB GPU with Llama 3.1 8B (128 KiB of cache per token, from Part 2). Prompt and answer lengths came from wide, realistic spreads, and each request was caught at a random moment in its life. The question: of the memory each policy sets aside, how much holds tokens, and how many requests fit at once?
| Allocation policy | Memory holding real tokens | Requests that fit |
|---|---|---|
| reserve the model's maximum length (8,192 tokens) | 11.6% | 48 |
reserve prompt + the request's max_tokens | 33.7% | 141 |
| reserve the exact final length (impossible: needs the future) | 83.5% | 349 |
| paged, 16-token blocks | 99.2% | 415 |
Paging beats even the "oracle" that magically knows every answer's length. The oracle still has to reserve the space for tokens that it has not written yet: waste 1. My simulation also leaves out external fragmentation, so the three slab policies look better here than they would on a real server. With blocks, the last block of a request had 7.5 empty slots on average, out of 16.
The real system agrees. The paper measured the memory of real serving runs.
More useful memory means more requests per step, which means larger batches. Across their tests, the authors report 2 to 4 times the throughput of the best systems of the time, "with the same level of latency".
Why 16 tokens per block?
A smaller block wastes less in the last block, but the kernel then does more lookups for less data each time. A larger block is the reverse. The authors tried sizes from 1 to 256.
The price of paging
Nothing is free. The block table adds a step to every read of the cache. This section shows what that step costs, and why vLLM pays it.
Start with an analogy. You read a bound book straight through, page after page. Now the same book is in loose pages, spread over several rooms, with an index on the front door: "pages 1 to 16 in room 7, pages 17 to 32 in room 1". To read it, you check the index, walk to room 7, read, come back, check again, walk to room 1. The words are the same. The reading takes longer because of the index and the walking.
A GPU reads memory fastest in long, regular sweeps. With one contiguous array, the kernel knows the start address and the length, and its thousands of threads read straight through. With blocks, it must first read the block table, compute where each block starts, and treat the last, partly filled block in a different way. In one batch, every request also has a different length and a different table.
The paper measured this cost directly. It compared the attention kernel of vLLM with the attention kernel of FasterTransformer, a fast NVIDIA library that keeps the cache in one contiguous array.
So attention is slower. Why is the whole system still much faster? There are two reasons.
First, attention is only part of a step. A decode step also runs the large matrix multiplications of every layer, which read all the model's weights (Part 1). Paging does not touch that work. If attention were, for example, a quarter of a step, a 26% slower attention would make the step about 6.5% slower. That share is an assumption for the arithmetic only. The real share depends on the model, the batch and the context length.
Second, paging buys a much larger batch. In our simulation the same GPU fits 415 requests instead of 48 when it reserves the maximum length. Part 1 measured what batch size is worth. Going from 1 to 128 sequences gave 44 times more tokens per second, for a step only about 3 times slower. A small tax on one part of each step buys many more sequences in that step.
The authors also reduced the cost with care. They merged the block lookup into the attention kernel itself, so the gather and the arithmetic happen in one pass.
Today vLLM runs attention through several kernel libraries, and all of them read the cache through block tables. The trade stayed the same: a small tax on attention for a large gain in memory.
Sharing blocks
Blocks bring a second gift. Two requests can point at the same physical block. Nothing in the block table forbids it.
Where does this help? Some requests ask for several answers to one prompt: "give me three versions of this email" (called parallel sampling). Beam search, used in translation, keeps several candidate answers and extends the best ones. In both cases, many sequences start with the same tokens. Their keys and values are the same too, so they need to be stored only once.
The analogy is two students sharing one textbook. That works well while both only read it. When one student wants to write notes in the margin of page 12, they photocopy page 12 and write on the copy. The rest of the book stays shared. Operating systems do exactly this when a program splits into two (called a fork), and the name of the trick is copy-on-write.
Here is the trick on a small example: two samples of one 20-token prompt. The prompt fills block 7 and 4 slots of block 1. Both samples point at both blocks, so each block has a reference count of 2. Then each sample writes its own token 21 into block 1.
== 7. Copy-on-write: two samples of one 20-token prompt (block size 16)
after prefill: tables {'sample 1': [7, 1], 'sample 2': [7, 1]}, ref counts {7: 2, 1: 2}, blocks stored 2 (without sharing: 4)
sample 1 writes token 21: block 1 is shared (ref 2), copy it to block 3, ref of 1 -> 1
sample 2 writes token 21: block 1 has ref 1, write in place
now: tables {'sample 1': [7, 3], 'sample 2': [7, 1]}, ref counts {7: 2, 1: 1, 3: 1}, blocks stored 3Prefix caching: reuse across time
Sharing among the requests running right now is good. Sharing across time is better. Many requests to one service start with the same long text: a system prompt with instructions, or the same document with a different question. If a thousand users send requests that start with the same 2,000-token system prompt, why compute its keys and values a thousand times?
The analogy is a coffee shop that sells one drink 500 times a morning. It does not grind fresh beans for each cup from zero. It keeps ground coffee ready, and only the last steps happen per order.
The paper describes the idea for a prefix that the service provider declares in advance.
How does vLLM find a match without being told? It gives every full block a name: a hash of the 16 tokens in the block and of the name of the block before it. Each name includes the name before it. So a block's name stands for the whole text from the start to the end of that block. Two requests that start with the same 32 tokens get the same first two names.
When a request arrives, vLLM computes the names of its full blocks, in order, and looks each one up in a table of cached blocks. It reuses every block it finds until the first miss. Prefill then runs only for the tokens after that point.
== 8. Prefix caching: each full block is named by a hash of its tokens and of the block before it
request 1: 50 tokens, full-block hashes ['9266d336', 'e9d16e46', 'df7fc873'], reused from cache 0 blocks = 0 tokens, prefill 50 tokens
request 2: 49 tokens, full-block hashes ['9266d336', 'e9d16e46', '3953b78c'], reused from cache 2 blocks = 32 tokens, prefill 17 tokens
request 3: 50 tokens, full-block hashes ['5b33b966', '255d9ee6', '9925b8e7'], reused from cache 0 blocks = 0 tokens, prefill 50 tokensRequest 3 shows why the name must include the block before it. Its second block has the same 16 tokens as request 1's second block. But their keys and values are different. In a transformer, the keys and values of a token depend on every token before it (Part 2). Reusing that block would give wrong answers, and the chained name prevents it.
How much does it save? I measured a 2,048-token shared system prompt followed by a 64-token question:
| Time to first token | |
|---|---|
| full prefill of prompt + question | 98.3 ms |
| system prompt reused from cache, prefill only the question | 14.2 ms |
That is 7 times faster, and the next token is identical: the largest difference in any output score between the two runs was exactly 0.0. The saved prefill is not an approximation. It is the same arithmetic, done once instead of twice.
What happens to cached blocks when memory gets tight? When a request finishes, its blocks go to the free pool but keep their names and contents. If a new request matches them, they come back out of the pool. If the pool needs a block, it takes the one used least recently. So the cache costs no memory that anyone else needs.
Continuous batching
So far the topic was memory. The next ideas are about time: who runs in each step.
Batching (Part 1) helps only while the batch stays full. The simple way to batch is static: gather 64 requests, run them together until all are finished, then take the next 64. But answers have very different lengths. A request that ends after 20 tokens leaves its slot empty while the batch waits for one that runs to 2,000.
The analogy is a bus that leaves only when every passenger from the last trip has got off at the final stop. Most seats ride empty for most of the trip. A better bus lets someone off at every stop and lets the next person in the queue get on.
Continuous batching decides who is in the batch at every step. It was introduced by the Orca system (Yu et al., OSDI 2022), under the name iteration-level scheduling. When a request finishes, its slot goes to the next one in line at the next step.
I simulated 4,096 requests with realistic answer lengths, served through 64 slots:
| Steps to finish all requests | Slots doing useful work | |
|---|---|---|
| static batching | 114,566 | 21% |
| continuous batching | 25,127 | 96% |
Same requests, same slots, 4.6 times fewer steps. The exact ratio depends on how uneven the answer lengths are, but the direction is always the same. Orca reported a 36.9x throughput gain over NVIDIA FasterTransformer on GPT-3 175B "at the same level of latency."
Continuous batching and paging need each other. Requests that join and leave at every step would shred a memory made of slabs. Every arrival needs a new slab, and every departure leaves a gap of a different size. With blocks, a finished request returns its blocks to the pool, and the next one takes whichever blocks it needs.
In vLLM's current engine, called V1, the batch for each step is one long row of tokens. Every scheduled token from every request sits end to end, and position numbers and the block tables keep each request's tokens apart. There is no padding and no fixed slot count. V1 became the default engine in v0.8 (2025), and since v0.11 it is the only one.
Chunked prefill
Continuous batching lets a new request join at the next step. But what if the new request's prefill is huge?
Prefill and decode behave in opposite ways (Part 1). A decode step is short and comes again and again. The prefill of a long prompt is one large step. Say 16 users are reading streamed answers, and a new user sends an 8,192-token document. Without care, that prefill runs as one step, and every streaming user freezes until it is done. From their side, the text stops.
The analogy is a supermarket with one till. A customer with a full trolley arrives behind ten people with one item each. A fair till serves a few of the trolley's items, then one of the quick customers, then a few more items. The trolley takes a little longer, but nobody else waits a long time.
Chunked prefill cuts the long prompt into slices and adds one slice to each step, next to the normal decodes. We can estimate the effect from step costs measured in Part 1. Sixteen decodes took 12.8 ms, a 512-token prefill 27.4 ms, and an 8,192-token prefill 450.5 ms.
| Longest pause for streaming users | Long prompt's first token | |
|---|---|---|
| no chunking | 463 ms | 463 ms |
| 512-token chunks | 40 ms | 643 ms |
These are estimates built by adding measured step costs, not a measured mixed batch. Later slices cost a little more than the first, because they attend to the slices before them. So the real chunked numbers would be a little higher. The trade they show is the real one.
The streaming users' worst pause drops from almost half a second to 40 ms. The price is that the long prompt's first token arrives later, because its prefill is now spread over 16 steps.
In vLLM V1, chunked prefill is on by default. Each step has a token budget, set by --max-num-batched-tokens. The scheduler first puts in every running decode, then fills the rest of the budget with prefill chunks. The docs describe the trade of this knob. Smaller values give better inter-token latency, because fewer prefill tokens slow down each decode step. Larger values give a better time to first token.
When memory runs out anyway
Paging makes memory go much further, but a GPU can still fill up. Many long answers can grow at once, and each one needs a new block every 16 tokens. When a running request needs a block and the pool is empty, something must give way. The scheduler preempts a request. It takes all of that request's blocks back and puts the request back in the queue.
Which request gives way? The one that arrived last. What happens to its blocks? There are two options.
- Swap: copy the blocks to CPU memory, and copy them back when the request runs again. This is the operating system's classic method, which moves pages to disk.
- Recompute: drop the blocks. When the request runs again, join its prompt and the tokens it already wrote into one new prompt. Then run one prefill over all of it.
The paper compared the two.
In vLLM V1 the default is recompute. The docs say it "has lower overhead in the V1 architecture". Preemption is a warning sign in production. If logs show it often, the server is short of KV memory, and the knobs in the next sections help.
The whole engine, in one loop
Every idea so far fits into one short loop that runs every few milliseconds.
- Schedule. Decide what runs in this step. First come the running requests that need their next token, then chunks of waiting prompts, within the token budget. Each new token needs a slot in a block. If the pool is empty, preempt the newest request. Look up each new prompt's full blocks by name, to skip work that the prefix cache already holds.
- Forward pass. Lay every scheduled token end to end in one batch, with each request's block table, and run the model once. The attention kernel reads and writes blocks through the tables.
- Post-process. Append each sampled token to its request. Return the blocks of finished requests to the free pool, with their names kept for the prefix cache. Then loop.
Each line uses an idea that this part built: blocks and tables (paging), continuous batching, chunked prefill, preemption and prefix caching.
The knobs that matter
Most of these ideas are on by default. These are the settings people tune most often. Defaults are from vLLM v0.32.0.
| Flag | What it controls |
|---|---|
--gpu-memory-utilization | Fraction of GPU memory vLLM may use (default 0.92). What the weights and working memory do not need becomes KV blocks. |
--max-model-len | Longest prompt + answer allowed. A lower value means less memory per request in the worst case. |
--max-num-seqs | Most sequences in one step. |
--max-num-batched-tokens | Token budget per step. Smaller: smoother streaming. Larger: faster first token. |
--block-size | Tokens per KV block (16 by default). |
--kv-cache-dtype | Store the cache in fp8 to fit about twice as many tokens (Part 2). |
--enable-prefix-caching / --no-enable-prefix-caching | Reuse cached blocks across requests (on by default). |
Engine arguments change between releases, so check them against the version you run (vllm serve --help).
Running vLLM for the first time
Everything on this page happens inside one command. On a machine with an NVIDIA GPU and vLLM installed, this starts a server for Llama 3.1 8B:
vllm serve meta-llama/Llama-3.1-8B-InstructThe server downloads the weights and loads them onto the GPU. Then it answers requests on port 8000, in the same format as the OpenAI API, so most client libraries can talk to it. The knobs from the table above go on the same line. For example, this asks for shorter requests and an FP8 cache:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--gpu-memory-utilization 0.92 \
--max-model-len 8192 \
--kv-cache-dtype fp8The two lines to look for at startup
Before the server accepts its first request, it prints a long log. Two lines in it tell you how big the KV cache is. In vLLM v0.32.0, the code that prints them is in vllm/v1/worker/gpu_worker.py and vllm/v1/core/kv_cache_utils.py, and their format is:
Available KV cache memory: %s GiB
GPU KV cache size: %s tokens, Maximum concurrency for %s tokens per request: %.2fxThe first line is the memory that became KV blocks. The second line turns it into tokens. "Maximum concurrency" is how many requests of the full --max-model-len could have their whole cache in memory at the same time. Real requests are usually shorter, so more of them fit.
Computing those lines yourself
You can work out both lines before you start the server, with the formula from Part 2 and the blocks from this page. We have no NVIDIA GPU for this series, so the numbers below are computed, not measured. They follow vLLM v0.32.0's own formulas, for Llama 3.1 8B on one 80 GB H100.
- What vLLM may use. The total memory times
--gpu-memory-utilization: 80 GB × 0.92 = 73.60 GB. - What is left for the cache. Take away the weights, 16.06 GB, and the working memory. vLLM measures the working memory in a test run at startup. We assume 4 GB, as in the simulation above. That leaves 53.54 GB, which the log shows as 49.86 GiB.
- Blocks. One block holds 16 tokens at 128 KiB each: 2 MiB. So 53.54 GB holds 25,529 blocks, which is 408,464 tokens.
- Concurrency. Llama 3.1 8B allows 131,072 tokens by default. One request that long needs 8,192 blocks, so 25,529 ÷ 8,192 = 3.12 such requests fit.
With these assumptions, the two lines would read:
Available KV cache memory: 49.86 GiB
GPU KV cache size: 408,464 tokens, Maximum concurrency for 131,072 tokens per request: 3.12xWith --max-model-len 8192, the cache is the same size, but each full request needs only 512 blocks, so the concurrency becomes 49.86x. With --kv-cache-dtype fp8, each block is 1 MiB, and the cache doubles to 816,944 tokens. The simulation earlier on this page kept 90% of memory instead of 92%, which is why its budget was a little smaller.
The real numbers will differ a little. Your driver reports its own total memory, and the test run measures the working memory instead of assuming it. But if your log shows a much smaller cache than this, look for the cause. Another process may be using the GPU, or --gpu-memory-utilization may be set low.
Watching the ideas at work: the metrics
A running server shows the ideas of this page as numbers. It publishes them at http://localhost:8000/metrics, in the text format of Prometheus, a common monitoring tool. These are the names in vLLM v0.32.0. Prometheus adds _total to the names of counters.
| Metric | What it shows | Healthy | A warning sign |
|---|---|---|---|
vllm:kv_cache_usage_perc | share of KV blocks in use (1 means 100%) | high under load, with room left | stuck near 1, together with waiting requests |
vllm:num_requests_running | requests in the current batch | close to what memory allows | far below it while requests wait |
vllm:num_requests_waiting | requests in the queue | near 0 most of the time | growing: the server cannot keep up |
vllm:num_preemptions | requests stopped to free blocks (counter) | flat, or rising slowly | rising fast: the cache is too small for the load |
vllm:prefix_cache_hits ÷ vllm:prefix_cache_queries | share of prompt tokens found in the prefix cache | high when prompts share a system prompt | near 0 when you expected reuse |
vllm:time_to_first_token_seconds | TTFT, as a histogram | within your target | a long tail: queueing or long prefills |
vllm:inter_token_latency_seconds | the time between tokens (TPOT), as a histogram | steady | spikes: long prefills in the same steps, or a batch that is too large |
The server also prints a short summary to its log every 10 seconds by default. It shows the prompt and generation throughput, the running and waiting requests and the KV cache usage. When there are preemptions or prefix cache hits, it shows those too. A later page in this series covers monitoring in depth.
Beyond one GPU
A model too large for one GPU is split across several. The most common split is tensor parallelism: every layer's matrices are cut into parts, and each GPU computes its part. For attention, each GPU takes some of the heads.
How do blocks work then? Every GPU processes the same tokens, so every GPU needs the same block positions. The scheduler keeps one block table per request and sends it to all GPUs with each step. Block 7 means the same thing on every GPU. Each GPU stores only the keys and values of its own heads in its copy of block 7.
Two more ideas build on what this part showed. Part 7 covers both in depth.
Separate GPUs for prefill and decode. Chunked prefill lets the two phases share a GPU politely. The stronger option is to put them on different GPUs. Some GPUs only prefill, others only decode, and the KV cache moves from one to the other. DistServe (Zhong et al., OSDI 2024) showed that this can serve up to 7.4x more requests. Or it can meet 12.6x tighter latency targets, because each phase gets GPUs sized for its own limit.
Speculative decoding. Part 1 measured that processing 16 tokens costs about the same as processing one (10.8 ms against 10.0 ms). Speculative decoding uses that. A cheap helper guesses the next few tokens, and the big model checks them all in one pass, up to the first wrong guess. The output is the same as if the big model wrote every token itself. Part 4 builds it from scratch and measures it.
The series so far, on one page
| Idea | The problem | The fix | What we measured or simulated |
|---|---|---|---|
| prefill vs decode | writing is 190x slower than reading | understand it: decode is memory-bound | 27 ms to read 512 tokens, 5.2 s to write 512 |
| batching | one sequence wastes each weight read | decode many sequences per step | 44x throughput at 128 sequences |
| KV cache | recomputing the past every step | keep keys and values | next token at 8,192: 449 ms to 11.5 ms |
| PagedAttention | reserved memory sits empty | 16-token blocks, allocated on demand | 48 to 415 requests at once |
| sharing blocks | parallel samples store one prompt many times | point at the same blocks, copy on write | (paper) 6.1% to 55.2% less KV memory |
| prefix caching | the same prompt prefilled again and again | reuse blocks by name (hash) | first token 98 ms to 14 ms |
| continuous batching | slots idle behind the slowest request | refill slots every step | 4.6x fewer steps |
| chunked prefill | long prompts freeze streaming users | slice prompts into each step | worst pause 463 ms to 40 ms |
Every one of these ideas goes back to two facts from Part 1. Decode is limited by memory, not by arithmetic. And a GPU is productive only when it decodes many sequences at once. The whole design of vLLM fits as many sequences as possible into memory, and keeps every step full.
The simulation and the small checks: p3_paging.py
The script code/writings/llm-inference/p3_paging.py reproduces the two simulations of this page and the small examples. Here is the simulation part.
"""KV allocation policies and static vs continuous batching (pure Python + numpy)."""
import heapq, math
import numpy as np
rng = np.random.default_rng(42)
# ---- how much KV memory each policy wastes (Llama 3.1 8B on one 80 GB GPU)
PER_TOKEN = 2 * 32 * 8 * 128 * 2 # 128 KiB, from the config
BUDGET = 0.9 * 80e9 - 8.03e9 * 2 - 4e9 # 90% of memory, minus weights and ~4 GB working memory
N, MAX_LEN, MAX_TOKENS, BLOCK = 200_000, 8192, 2048, 16
prompt = np.clip(rng.lognormal(math.log(512), 0.9, N), 16, 6000).astype(int)
output = np.clip(rng.lognormal(math.log(256), 0.9, N), 8, MAX_TOKENS).astype(int)
now = prompt + (rng.uniform(0, 1, N) * output).astype(int) # a random moment in each request's life
policies = {
"reserve max length": np.full(N, MAX_LEN),
"reserve prompt + max_tokens": prompt + MAX_TOKENS,
"reserve exact final length": prompt + output,
"paged, 16-token blocks": np.ceil(now / BLOCK).astype(int) * BLOCK,
}
for name, reserved in policies.items():
print(f"{name:28s} useful {now.sum() / reserved.sum():6.1%} fits {BUDGET / (reserved.mean() * PER_TOKEN):5.0f} requests")
# ---- static vs continuous batching: 4,096 requests, 64 slots
SLOTS = 64
lens = np.clip(rng.lognormal(math.log(256), 0.9, 4096), 8, 2048).astype(int)
static = sum(lens[i:i + SLOTS].max() for i in range(0, len(lens), SLOTS))
free = [0] * SLOTS
for L in lens:
heapq.heappush(free, heapq.heappop(free) + L)
continuous = max(free)
print(f"static {static:,} steps, continuous {continuous:,} steps, {static / continuous:.1f}x")== 1. How much reserved KV memory holds real tokens (simulation, Llama 3.1 8B, one 80 GB GPU)
reserve max length useful 11.6% fits 48 requests
reserve prompt + max_tokens useful 33.7% fits 141 requests
reserve exact final length useful 83.5% fits 349 requests
paged, 16-token blocks useful 99.2% fits 415 requests
paged: mean empty slots in the last block 7.5 (always 0 to 15)
== 2. Static vs continuous batching (simulation, 4,096 requests, 64 slots)
static 114,566 steps, useful 21%
continuous 25,127 steps, useful 96%
ratio 4.6xThe startup arithmetic: p3_vllm_startup.py
The script code/writings/llm-inference/p3_vllm_startup.py works out the two startup lines with vLLM v0.32.0's formulas. It runs no model and needs no GPU. Its output:
1. vLLM may use 0.92 x 80 GB = 73.60 GB
2. minus weights 16.06 GB and working memory 4 GB = 53.54 GB for the KV cache
log line: Available KV cache memory: 49.86 GiB
--kv-cache-dtype auto (BF16): one block = 65,536 bytes per layer x 32 layers = 2 MiB
3. blocks = 53.54 GB / 2 MiB = 25,529 blocks = 408,464 tokens
4. --max-model-len 131,072: 8,192 blocks per request -> log line: GPU KV cache size: 408,464 tokens, Maximum concurrency for 131,072 tokens per request: 3.12x
4. --max-model-len 8,192: 512 blocks per request -> log line: GPU KV cache size: 408,464 tokens, Maximum concurrency for 8,192 tokens per request: 49.86x
requests of 1,024 tokens (prompt + answer): 398 fit at once
requests of 4,096 tokens (prompt + answer): 99 fit at once
--kv-cache-dtype fp8: one block = 32,768 bytes per layer x 32 layers = 1 MiB
3. blocks = 53.54 GB / 1 MiB = 51,059 blocks = 816,944 tokens
4. --max-model-len 131,072: 8,192 blocks per request -> log line: GPU KV cache size: 816,944 tokens, Maximum concurrency for 131,072 tokens per request: 6.23x
4. --max-model-len 8,192: 512 blocks per request -> log line: GPU KV cache size: 816,944 tokens, Maximum concurrency for 8,192 tokens per request: 99.72x
requests of 1,024 tokens (prompt + answer): 797 fit at once
requests of 4,096 tokens (prompt + answer): 199 fit at onceResearch notes: what changed in vLLM since the paper
The paper describes vLLM as of 2023. Several parts changed in the engine that people run today, V1, which became the default in v0.8 (2025) and the only engine in v0.11.
- Prefix caching went from provider-declared prefixes to automatic matching by chained block hashes, on by default. The hash also includes extra values, such as a LoRA adapter ID or a cache salt. Two requests then never share blocks that only look the same.
- Chunked prefill is on by default, and the scheduler always puts decodes first.
- Preemption recomputes by default. The paper's swap path to CPU memory is no longer the default.
- Attention kernels come from several libraries, which all read the cache through block tables. The paper's own kernel was the first of them.
The paper's footnote 1 adds a detail. A block can hold the keys and values of all layers together, or each layer can have its own blocks and block tables. The authors report no performance difference and chose separate blocks per layer. That is why this page talks about "a block" as 16 tokens across layers: it is the same total memory either way.
References
- W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
- G.-I. Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.
- A. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
- Y. Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
- vLLM documentation: Optimization and tuning, Engine arguments, Automatic prefix caching design.
- vLLM source, v0.32.0: cache configuration (block size, prefix caching and memory defaults), kv_cache_utils.py (the KV cache size and concurrency log line), gpu_worker.py (the available KV memory) and metrics/loggers.py (the metric names).
- vLLM documentation: Production metrics.
Next: Part 4, speculative decoding: a cheap helper guesses several tokens, and the big model checks them all in one step.