The KV Cache: An LLM's Short-Term Memory, and Its Bill
Why an LLM keeps notes on every token, how big they get (16 GiB for one 128K-token Llama 3.1 8B chat), and why the old way of storing them wasted most of the memory.
Part 1 ended with a rule. A decode step reads all the model's weights from memory, whether it writes one token or a hundred. So a GPU is only productive when it decodes many conversations at once. The catch was in the last line: every conversation in that batch needs its own memory for its notes. Those notes are called the KV cache.
This part answers four questions about the notes. What are they? Why does the model keep them? How big do they get? And why did the first serving systems waste most of the memory they set aside for them? The last answer is the problem that Part 3 solves.
What the model needs from the past
Start with the loop from Part 1. To write the next token, the model takes everything so far, runs it through its stack of layers and gets one new token. Then it appends that token and runs again.
Here is the question that Part 1 skipped. When the model writes token 500, how does it use tokens 1 to 499? A word like "it" means nothing on its own. To write the next word well, the model must find out what "it" refers to, and that answer sits somewhere earlier in the text.
The part of each layer that looks back at earlier tokens is called attention. To understand the KV cache, you need to understand attention first. Attention is a lookup, and a small example shows all of it.
Attention: a lookup with a question, labels and contents
Think of a library with a card for every book. Each card has a label on the front (what the book is about) and the contents inside. You walk in with a question. You compare your question with every label, pick the cards that match best, and read mostly their contents.
Attention does the same thing with vectors. At each layer, every token makes three vectors from its own vector:
- a query: the question this token asks ("which earlier word do I refer to?");
- a key: the label it shows to later tokens ("I am a noun, an animal");
- a value: the content it hands over when a later token picks it.
Each of the three comes from multiplying the token's vector by a fixed matrix of weights. The model learned these three matrices during training, and they stay the same at inference.
A lookup with tiny numbers
Take the text "The cat sat" and a new token, "it". To keep the arithmetic small, every key, value and query here has only 2 numbers. Real models use 64 or 128 numbers per head, but the steps are the same.
The three earlier tokens have these keys and values:
| Token | Key | Value |
|---|---|---|
| The | [0, 1] | [1, 0] |
| cat | [3, 0] | [0, 4] |
| sat | [1, 1] | [2, 2] |
The new token "it" asks with the query [1, 0]. Attention now takes three steps.
Step 1: compare the query with every key. The comparison is a dot product: multiply the numbers in matching positions and add the results. For "cat", that is 1 × 3 + 0 × 0 = 3. For "The" it is 0, and for "sat" it is 1. A higher score means a better match.
Step 2: turn the scores into weights. First divide each score by the square root of the vector length, here √2. This keeps scores from growing too large in long vectors. Then apply softmax: raise e to the power of each score and divide by the total. The weights come out as 0.088, 0.734 and 0.178, and they add up to 1.
Step 3: mix the values. Multiply each value by its weight and add them up. The answer is [0.44, 3.29]. It is mostly the value of "cat", because the key of "cat" matched the query best.
The three steps fit in one line. Here is the query, holds the keys of all earlier tokens, holds their values and is the length of one key:
Read it from the inside out. is step 1, one dot product for every key. Dividing by and the softmax are step 2. Multiplying by is step 3. Our script checked the hand result against PyTorch's own attention function, and both give [0.44, 3.29].
Now look at what the lookup needed. It needed one query, from the new token only. It needed the key and the value of every earlier token. The next section is about where those keys and values come from on each step.
The waste hiding in the loop
A model like Llama writes from left to right. Each token may look only at the tokens before it, never at the tokens after it. So adding a new token at the end changes nothing about the tokens already there.
That has a strong consequence. The key and value of "cat" at layer 5 depend only on "The cat" and on the fixed weights. When "sat", "on" and "the" arrive later, the key of "cat" stays exactly the same. It was the same on step 3 and it will be the same on step 300.
Now picture a model without notes. On every step it would run the whole text through every layer again. It would compute every key and value again, only to get the same numbers as last time. It is like a writer who rereads the whole book from page one before each new sentence.
The fix is to keep notes. After a token's key and value are computed at a layer, store them. On the next step, compute the key and value of the new token only, add them to the store, and read the old ones back. That store is the KV cache.
What the cache saves, measured
How much time do the notes save? We measured the time to produce one more token on Qwen2.5-0.5B, the small model from Part 1, on the GPU of an Apple M5 Pro. Without a cache, the model runs the whole text again. With a cache, it runs only the new token.
| Tokens so far | No cache | KV cache | Slower without |
|---|---|---|---|
| 128 | 13.6 ms | 9.9 ms | 1.4x |
| 512 | 27.3 ms | 10.3 ms | 2.7x |
| 2,048 | 94.9 ms | 10.9 ms | 8.7x |
| 4,096 | 200.7 ms | 11.0 ms | 18x |
| 8,192 | 449.1 ms | 11.5 ms | 39x |
Without a cache, every new token costs a full prefill of the whole text. Compare the no-cache column with Part 1's prefill table: the numbers match. So a long answer gets slower with every token it writes, and at 8,192 tokens each step is 39 times slower. With a cache, each step costs about the same, whatever the length. Every serious inference engine uses one.
The price is in the word "store". The cache swaps compute for memory, and the next sections show how fast that memory adds up.
What exactly is stored
A real model does attention many times for each token, not once. It has a stack of layers, and each layer has its own attention. Inside each layer, attention is split into several heads: smaller lookups that run side by side, each with its own keys and values.
So for every token, the cache keeps one key and one value per KV head, in every layer. Each key and each value is a vector of head_dim numbers, and each number takes a fixed number of bytes.
Is that the real layout? We can look inside a running model. We ran a 100-token prompt through Qwen2.5-0.5B and opened its cache. Layer 0 holds a key tensor of shape (1, 2, 100, 64): 1 sequence, 2 KV heads, 100 tokens and 64 numbers per key. There are 24 layers, each with keys and values.
== 2. The KV cache of a real model: Qwen2.5-0.5B, bfloat16
config: layers 24, query heads 14, KV heads 2, head_dim 64, device mps
100 tokens: 24 layers, key tensor of layer 0 (1, 2, 100, 64) torch.bfloat16, cache 1,228,800 bytes
200 tokens: 24 layers, key tensor of layer 0 (1, 2, 200, 64) torch.bfloat16, cache 2,457,600 bytes
measured: 12,288 bytes per token; formula 2 x 24 x 2 x 64 x 2 = 12,288 bytesThe cache for 200 tokens is exactly twice the cache for 100 tokens. Every token adds the same 12,288 bytes. That gives a formula you can apply to any model.
The size formula
Every number in the formula comes from a model's config.json file. Here is the file for Llama 3.1 8B, the main example of this part:
Read it from left to right. The 2 counts the key and the value. Layers and KV heads count how many keys and values each token leaves behind. Head dim is the length of each one. Bytes per number is 2 for the 16-bit formats that models usually run in.
Check it on Qwen2.5-0.5B first: bytes, which is exactly what we measured. Now Llama 3.1 8B: bytes. That is 128 KiB for every token of every conversation the GPU serves.
How big it gets
The cache grows by one entry per token, so its size is the formula times the number of tokens. Llama 3.1 8B accepts up to 131,072 tokens (128K). One conversation at that length needs KiB, which is 16 GiB.
| Model | Layers | KV heads | Head dim | Per token | One 128K-token conversation |
|---|---|---|---|---|---|
| Qwen2.5-0.5B | 24 | 2 | 64 | 12 KiB | 1.5 GiB |
| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB | 16 GiB |
| Llama 3.1 70B | 80 | 8 | 128 | 320 KiB | 40 GiB |
| Llama 2 7B | 32 | 32 | 128 | 512 KiB | 64 GiB |
Look at the Llama 3.1 8B row again. The weights of this model take about 16 GB. One conversation at full length needs a cache of the same size. One long chat costs as much memory as the whole model.
Now put that on a real GPU. An NVIDIA H100 has 80 GB of memory. The weights of Llama 3.1 8B take 16.1 GB (8.03 billion numbers at 2 bytes each). One 128K-token cache is 17.2 GB. After the weights, the card has room for only three such conversations.
Part 1 showed that throughput grows with the batch. Here the batch is three. Most conversations are much shorter than 128K tokens, so a real server fits more. But the point stands: the cache, not compute, sets how many conversations a GPU can hold.
Shrinking the notes
The table above holds a surprise. Llama 2 7B and Llama 3.1 8B are about the same size, but the older model needs four times more cache per token. The difference is one number in the config: 32 KV heads against 8.
Fewer key and value heads: MHA, GQA and MQA
In the original transformer, every query head has its own key head and value head. That is multi-head attention (MHA). Llama 2 7B works this way: 32 query heads and 32 KV heads.
The idea behind the newer designs is that query heads can share keys and values. Each query head still asks its own question. But a group of query heads compares against the same keys and mixes the same values. Only the shared keys and values go into the cache.
- Multi-query attention (MQA) shares one key head and one value head across all query heads.
- Grouped-query attention (GQA) sits between the two: each group of query heads shares one KV head. Llama 3.1 8B has 32 query heads in 8 groups of 4.
What would Llama 3.1 8B cost under each design? Keep 32 layers, a head dim of 128 and 2 bytes, and change only the number of KV heads:
| Design | KV heads | Per token | 128K-token conversation |
|---|---|---|---|
| MHA | 32 | 512 KiB | 64 GiB |
| GQA (the real model) | 8 | 128 KiB | 16 GiB |
| MQA | 1 | 16 KiB | 2 GiB |
Compressed keys and values: MLA
A third design stores something smaller than the keys and values. Multi-head latent attention (MLA), from DeepSeek, squeezes each token's keys and values into one short vector, called a latent vector. At attention time, the model expands it back. The cache holds only the short vector.
Fewer bytes per number: a quantised cache
The other lever is precision. The cache normally uses 2 bytes per number. An 8-bit format, such as FP8, stores each number in 1 byte. The numbers keep fewer digits, and the cache takes half the memory.
For Llama 3.1 8B, an FP8 cache needs 64 KiB per token, so 8 GiB for a 128K-token conversation instead of 16 GiB. The vLLM engine offers this with the flag --kv-cache-dtype fp8. The cost is a small risk to accuracy, so measure it on your own tasks before you use it.
All of these levers make each token cheaper. None of them changes the shape of the problem. The cache still grows by one entry per token, for every conversation, as long as it runs.
How many conversations fit on a real GPU
Now we can answer the question an inference engineer asks first about any deployment: how many conversations can this GPU hold at once? The size formula is all you need. Take the GPU's memory, take away the weights and some working memory, and divide what is left by the cache of one conversation.
Think of a car park. The building has a fixed number of spaces. Some are taken by staff cars all day, which are the weights. A few are kept free for deliveries, which is the working memory. The rest is for visitors. How many visitors fit depends on the size of each car, and a long conversation is a long car.
A small example first
Take the H100 from above, with Llama 3.1 8B and conversations of 4,096 tokens.
- Memory for the cache. Keep 10% of the 80 GB as working memory, which leaves 72 GB. The weights take 16.1 GB, so 55.9 GB is left for the cache.
- Cache per conversation. At 128 KiB per token, 4,096 tokens need 512 MiB, or 0.54 GB.
- Divide. 55.9 GB ÷ 0.54 GB = 104 conversations.
With an FP8 cache, each token needs 64 KiB, so the same memory holds twice as many: 208. Conversations of 32,768 tokens are 8 times larger, so only 13 fit, or 26 with FP8.
Three GPUs, two models
A newer GPU has more memory, and the weights stay the same size. So most of the extra memory goes to the cache. These numbers come from the script p2_gpu_capacity.py, with the same 10% kept for working memory.
| Model and GPUs | Memory for the cache | 4,096 tokens, BF16 cache | 4,096 tokens, FP8 cache | 32,768 tokens, BF16 | 32,768 tokens, FP8 |
|---|---|---|---|---|---|
| 8B on 1 H100 (80 GB) | 55.9 GB | 104 | 208 | 13 | 26 |
| 8B on 1 H200 (141 GB) | 110.8 GB | 206 | 412 | 25 | 51 |
| 8B on 1 B200 (180 GB) | 145.9 GB | 271 | 543 | 33 | 67 |
| 70B on 4 H100 | 146.9 GB | 109 | 218 | 13 | 27 |
| 70B on 2 H200 | 112.7 GB | 83 | 167 | 10 | 20 |
| 70B on 2 B200 | 182.9 GB | 136 | 272 | 17 | 34 |
Why 70B needs several GPUs
Llama 3.1 70B has 70.6 billion weights, which is 141.1 GB in BF16. With 10% kept for working memory, an H100 offers 72 GB and an H200 offers 127 GB. Neither can hold the weights. A B200 offers 162 GB, so the weights fit, but only 20.9 GB is left for the cache: 15 conversations of 4,096 tokens.
So the model is split across several GPUs, and each GPU holds a share of every layer. On H100s, two GPUs would hold the weights with only 2.9 GB to spare, which is about 2 conversations. Four H100s leave 146.9 GB for the cache. The split must divide the model's 8 KV heads evenly, so the usual counts are 1, 2, 4 or 8 GPUs. The weights and the cache are then shared out in the same way. Part 3 shows how the cache works on split GPUs, and Part 7 covers the split itself.
Note one more effect in the table. Per token, the 70B model's cache is 320 KiB, 2.5 times the 8B model's. So four H100s with 146.9 GB of cache hold only about as many 70B conversations as one H100 holds for the 8B model.
What this number means for the batch
This number caps the batch from Part 1. A decode step can only include conversations whose notes are in GPU memory. On one H100 with 4,096-token conversations, the batch can never pass 104. But Part 1 showed that the ridge of an H100 is near a batch of 295. So memory, not compute, stops the batch first, as Part 1 said.
The table is also the best case. It assumes every byte of cache memory holds real tokens. The next sections show that the first serving systems got nowhere near that.
The cache is read on every step
Size is the first cost of the cache. The second cost takes us back to Part 1.
Part 1 showed that a decode step is slow because it must read all the weights from GPU memory. Memory delivers only so many bytes per second. That rate is the memory bandwidth. Now look again at the attention lookup. To write one token, the new query is compared with the key of every earlier token, and every value is mixed in. So every decode step must also read the whole cache of every sequence in the batch.
On an H100, with a bandwidth of 3.35 TB per second, reading the 16.1 GB of Llama 3.1 8B weights takes at least 4.8 ms. Reading the 17.2 GB cache of one 128K-token conversation takes at least 5.1 ms more. For a long conversation, the cache costs as much reading time as the whole model, on every step.
Our small model shows the same effect in a real run. Here is the decode step time as the context grows, for 1 sequence and for 16 sequences at once:
| Context per sequence | 1 sequence | 16 sequences |
|---|---|---|
| 128 tokens | 10.2 ms | 11.3 ms |
| 1,024 | 11.5 ms | 15.4 ms |
| 4,096 | 10.9 ms | 23.4 ms |
| 8,192 | 11.5 ms | 35.9 ms |
For one sequence the line is flat. This model's cache is small, 12 KiB per token, so even 8,192 tokens is only 96 MiB. That is tiny next to the 0.99 GB of weights. For 16 sequences at 8,192 tokens each, the cache is 1.5 GiB, bigger than the weights. Every step must read all of it, and the step time more than triples.
So decode is limited by memory reads twice over. Each step reads the weights once, and it reads the cache of every sequence once. Long contexts and big batches make the second read the larger one. This is also why the shrinking methods above make decode faster, not only cheaper: fewer bytes in the cache means fewer bytes to read.
Nobody knows how long the answer will be
So far the cache sounds like a sizing problem: work out the bytes and buy enough memory. The real difficulty is that a request's cache grows while the request runs, and nobody knows in advance how far it will grow.
When a request arrives, the server knows the length of its prompt. It does not know the length of the answer. The model decides that on its own, token by token, when it writes its end token. The answer could be 3 tokens or 3,000.
How much does the length vary? The vLLM series on this site asked a small model, OPT-125M, the question from Part 1 twenty times. It used random sampling and a limit of 200 tokens:
The same question gave answers from 2 to 200 tokens. Across 2,000 real answers in the Alpaca dataset, the median answer has 43 tokens. But one answer in ten has 127 tokens or more, and the longest has 499. (The vLLM Part 2 page has the script and its output.)
Every server sets an upper limit. The model has a maximum length, such as 2,048 tokens for OPT or 131,072 for Llama 3.1 8B. A request can also set its own limit on new tokens, often called max_tokens. These limits say how big the cache could get. They say nothing about how big it will get.
How older systems stored the cache: one slab per request
To see why the size matters so much, we need one fact about how GPU programs keep data in memory.
Deep learning libraries such as PyTorch expect each tensor to be contiguous. That makes reading fast and simple. To find token 5 of a cache that starts at address 1,000, the code computes 1,000 + 5 × (bytes per token) and jumps there. No lookup table is needed.
The catch comes when the cache grows. Picture a group of friends at a theatre who want seats side by side. If a stranger already sits in the next seat, a late friend cannot join the row. In memory, if another request's data sits right after request A's cache, A has no room to add its next token.
The systems before vLLM, such as FasterTransformer and Orca, chose the safe answer. When a request arrives, they book one contiguous slab of memory big enough for the longest output the request could ever produce. Then the cache can grow into its own slab, and it never runs into a neighbour.
That last line is the whole problem. A slab sized for the maximum is safe, but most of it sits empty. The vLLM paper splits that empty space into three kinds of waste. Each has a different cause, so we take them one at a time, each on a small memory of slots.
In the pictures below, GPU memory is a row of slots. One slot holds the keys and values of one token, for all layers. For Llama 3.1 8B, one slot would be 128 KiB.
Waste 1: reserved slots
Picture a friend who saves a row of seats at a cinema for friends who will arrive during the trailers. The seats will be used. But until each friend arrives, the seat is empty, and nobody else may sit in it.
Request A shows the same thing. Its slab has 8 slots. Its prompt has 3 tokens, and over its life it will write 5 more, so in the end all 8 slots will hold a token.
Follow it step by step. After the prompt, 3 slots hold tokens and 5 are reserved: booked for tokens that do not exist yet. After 2 new tokens, 5 slots hold tokens and 3 are still reserved. Only when the answer ends are all 8 slots full.
Now scale it up. Suppose a request will write a 500-token answer, and its slab already covers that length. On its first decode step, 499 of its answer slots are reserved and empty. One slot fills per step. So over the life of the answer, about half of its answer slots are empty at any moment.
The cost is not lost data. The cost is timing. The memory sits idle exactly while other requests in the queue could have used it.
Waste 2: internal fragmentation
Now picture a booking at a restaurant: a table for eight. Only four guests come. The other four chairs stay empty all evening, and the restaurant cannot give them to anyone else, because they belong to your table.
Request B shows the same thing. Its slab has 8 slots, because 8 is its maximum length. Its prompt has 2 tokens. Its answer ends after 2 tokens, when the model writes its end token.
Request B used 4 of its 8 slots. The other 4 were empty from start to finish. This is internal fragmentation: memory inside a booking that the owner never uses. "Internal" means the waste sits inside a slab that someone owns.
Note the difference from reserved slots. A reserved slot will be used later. An internally fragmented slot will never be used. But at the moment of booking, the server cannot tell the two apart. Both are just empty slots in a slab, and only the end of the answer shows which was which.
At real sizes this waste is large. Take a request with a 100-token prompt, a slab for OPT's maximum of 2,048 tokens, and an answer of 43 tokens, the Alpaca median. It fills 143 of its 2,048 slots, about 7%. The other 1,905 slots are internal fragmentation.
Waste 3: external fragmentation
The third waste sits between slabs, not inside them. Picture a street with no marked parking bays. Cars of different lengths park along it, and some of them leave. Now the free spaces add up to enough room for a bus, but no single gap is long enough, so the bus cannot park.
Here is the same thing in a memory of 24 slots. Four requests have booked slabs of different sizes: A has 8, B has 4, C has 6 and D has 6. Then B and D finish and give their slabs back.
Now 10 slots are free. Request E arrives and needs a slab of 7. Ten is more than seven, but the free slots sit in two gaps, of 4 and 6. Neither gap holds 7 in one piece, so E must wait. This is external fragmentation: free memory cut into gaps that are each too small to use.
Slabs of different sizes cause these gaps. They come in different sizes because each request has its own prompt length and its own limit. Many memory allocators also round each booking up, for example to a power of two, which makes the sizes even more uneven. The vLLM paper names one such allocator, the buddy allocator.
Why not slide the slabs together to close the gaps? That is called compaction, and it means copying the cache of every request that moves. A cache can be many gigabytes, and no request can write a token while its cache is in motion. The paper calls compaction "impractical" for a fast serving system, and it does nothing for the first two wastes anyway.
All three wastes at once
In a real server, all three wastes happen at the same time. Here is one moment in a 24-slot memory, with two requests and the gaps around them:
Count the slots. Request A holds 5 tokens, has 2 reserved slots and 1 slot it will never use. Request B holds 3 tokens, has 1 reserved slot and 4 it will never use. Between and after them, 8 slots are free, in gaps of 3 and 5, and the next request needs 6. So only 8 of the 24 slots, 33%, hold real keys and values.
The vLLM paper draws the same picture with real words and real slab sizes:
How much memory was really wasted
The authors measured the three wastes in real serving runs. Orca, the leading system they compared with, is not public, so they built their own version of it in three variants. The variants differ only in how much space each request books for its answer:
- Orca (Max) books the model's maximum length, 2,048 tokens, for every request.
- Orca (Pow2) books at most twice the real answer length, rounded up to a power of two. An answer of 25 tokens gets 32 slots.
- Orca (Oracle) books exactly the real answer length. No real system can do this, because it needs to know the future. It shows the best that slab booking could ever do.
What the waste costs: a smaller batch
Why does wasted memory matter so much? Go back to the rule from Part 1. Throughput grows with the number of sequences in each decode step. Each sequence needs its cache in memory. So memory that holds nothing is memory that could have held another sequence.
The question for Part 3
Put the three wastes side by side, and each one points to what a better design must do:
- Reserved slots come from booking memory before the tokens exist. A better design gives memory to a request only when a new token actually needs it.
- Internal fragmentation comes from booking for the maximum length. A better design never books more than a small amount beyond what the request has already written.
- External fragmentation comes from slabs of different sizes that must each be one unbroken piece. A better design lets a request use free memory wherever it is, in pieces that all fit.
But the old design had a reason for its slabs. Attention reads keys and values fast because they sit in one contiguous run. If a request's cache is spread across memory, how does attention find its tokens? And how can memory grow in small pieces without a slab?
Operating systems faced the same problem with programs in the 1960s, and they solved it with an idea called paging. Part 3 shows how vLLM applied that idea to the KV cache, and what happens to each of the three wastes.
The scripts behind the numbers
The cache and no-cache timings and the decode step times come from the measurement below (Qwen2.5-0.5B, bfloat16, Apple M5 Pro GPU, median of 5 runs):
import statistics, time
import torch
from transformers import AutoModelForCausalLM, DynamicCache
DEV, sync = "mps", torch.mps.synchronize # "cuda", torch.cuda.synchronize on NVIDIA
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B", dtype=torch.bfloat16).to(DEV).eval()
def median_ms(fn, reps=5):
fn(); sync()
ts = []
for _ in range(reps):
sync(); t = time.perf_counter(); fn(); sync(); ts.append(time.perf_counter() - t)
return statistics.median(ts) * 1000
with torch.inference_mode():
for n in (128, 512, 2048, 4096, 8192):
x = torch.randint(0, model.config.vocab_size, (1, n), device=DEV)
# no cache: producing the next token means running the whole sequence again
no_cache = median_ms(lambda: model(input_ids=x, use_cache=False, logits_to_keep=1))
# cache: run the prompt once, then each new token is a single-token step
cache = DynamicCache()
out = model(input_ids=x, past_key_values=cache, logits_to_keep=1)
tok = out.logits[:, -1:].argmax(-1)
def step():
global tok
o = model(input_ids=tok, past_key_values=cache, logits_to_keep=1)
tok = o.logits[:, -1:].argmax(-1)
print(f"{n:5d} tokens: no cache {no_cache:6.1f} ms, cache {median_ms(step):5.2f} ms")The other new numbers come from p2_kv_numbers.py: the attention example, the measured cache, the sizes, the H100 times and the slot counts. Its full output:
== 1. Attention as a lookup: a query, three keys, three values
The key [0.0, 1.0] q.k = 0 /sqrt(2) = 0.00 weight = 0.088
cat key [3.0, 0.0] q.k = 3 /sqrt(2) = 2.12 weight = 0.734
sat key [1.0, 1.0] q.k = 1 /sqrt(2) = 0.71 weight = 0.178
answer = 0.088*[1.0, 0.0] + 0.734*[0.0, 4.0] + 0.178*[2.0, 2.0] = [0.44, 3.29]
PyTorch scaled_dot_product_attention gives [0.44, 3.29]
== 2. The KV cache of a real model: Qwen2.5-0.5B, bfloat16
config: layers 24, query heads 14, KV heads 2, head_dim 64, device mps
100 tokens: 24 layers, key tensor of layer 0 (1, 2, 100, 64) torch.bfloat16, cache 1,228,800 bytes
200 tokens: 24 layers, key tensor of layer 0 (1, 2, 200, 64) torch.bfloat16, cache 2,457,600 bytes
measured: 12,288 bytes per token; formula 2 x 24 x 2 x 64 x 2 = 12,288 bytes
== 3. KV bytes per token = 2 x layers x KV heads x head_dim x bytes
Qwen2.5-0.5B 24 layers 2 KV heads 64 dims: 12 KiB/token, 1.5 GiB for 131,072 tokens
Llama 3.1 8B 32 layers 8 KV heads 128 dims: 128 KiB/token, 16.0 GiB for 131,072 tokens
Llama 3.1 70B 80 layers 8 KV heads 128 dims: 320 KiB/token, 40.0 GiB for 131,072 tokens
Llama 2 7B 32 layers 32 KV heads 128 dims: 512 KiB/token, 64.0 GiB for 131,072 tokens
Llama 3.1 8B with other attention layouts (32 layers, head_dim 128, 2 bytes):
MHA, 32 KV heads 512 KiB/token, 64.0 GiB for 128K tokens
GQA, 8 KV heads 128 KiB/token, 16.0 GiB for 128K tokens
MQA, 1 KV head 16 KiB/token, 2.0 GiB for 128K tokens
GQA + 8-bit cache 64 KiB/token, 8.0 GiB for 128K tokens
Qwen2.5-0.5B, 16 sequences x 8,192 tokens: 1.50 GiB of cache
== 4. One decode step of Llama 3.1 8B on an H100 must read (3.35 TB/s)
weights 16.1 GB -> at least 4.8 ms
one 128K-token cache 17.2 GB (16 GiB) -> at least 5.1 ms more
on an 80 GB H100, after the weights: room for 3 such conversations at full length (63.9 GB / 17.2 GB)
== 5. Toy GPU memory: a row of 24 slots, one slot = the K and V of one token (all layers)
reserved: A books 8, prompt 3, wrote 2 -> holds 5, 3 reserved for tokens still to come
internal: B books 8, uses 4, 4 never used
external: AAAAAAAABBBBCCCCCCDDDDDD -> B, D finish -> AAAAAAAA....CCCCCC......; free 10 in gaps [4, 6]; E needs 7 in one piece: refused
all three: PPPWWRRI___PPWRIIII_____ (24 slots): tokens 8, reserved 3, never used 5, free 8 in gaps of 3 and 5 (next request needs 6: refused); holding tokens 33%The GPU capacity table comes from p2_gpu_capacity.py, which only does arithmetic on the configs and the vendor specs. Its full output:
Llama 3.1 8B: 8,030,261,248 weights -> 16.1 GB in BF16; KV per token BF16 128 KiB, FP8 64 KiB
Llama 3.1 70B: 70,553,706,496 weights -> 141.1 GB in BF16; KV per token BF16 320 KiB, FP8 160 KiB
assumption: 10% of GPU memory kept for working memory
Llama 3.1 8B on 1 x H100 80GB : weights 16.1 GB per GPU, KV memory 55.9 GB in total = 426,784 tokens BF16 / 853,568 FP8 | conversations that fit: 4,096 tok BF16 104, FP8 208; 32,768 tok BF16 13, FP8 26
Llama 3.1 8B on 1 x H200 141GB: weights 16.1 GB per GPU, KV memory 110.8 GB in total = 845,638 tokens BF16 / 1,691,276 FP8 | conversations that fit: 4,096 tok BF16 206, FP8 412; 32,768 tok BF16 25, FP8 51
Llama 3.1 8B on 1 x B200 180GB: weights 16.1 GB per GPU, KV memory 145.9 GB in total = 1,113,429 tokens BF16 / 2,226,859 FP8 | conversations that fit: 4,096 tok BF16 271, FP8 543; 32,768 tok BF16 33, FP8 67
Llama 3.1 70B on 2 x H100 80GB : weights 70.6 GB per GPU, KV memory 2.9 GB in total = 8,827 tokens BF16 / 17,654 FP8 | conversations that fit: 4,096 tok BF16 2, FP8 4; 32,768 tok BF16 0, FP8 0
Llama 3.1 70B on 4 x H100 80GB : weights 35.3 GB per GPU, KV memory 146.9 GB in total = 448,280 tokens BF16 / 896,561 FP8 | conversations that fit: 4,096 tok BF16 109, FP8 218; 32,768 tok BF16 13, FP8 27
Llama 3.1 70B on 2 x H200 141GB: weights 70.6 GB per GPU, KV memory 112.7 GB in total = 343,910 tokens BF16 / 687,820 FP8 | conversations that fit: 4,096 tok BF16 83, FP8 167; 32,768 tok BF16 10, FP8 20
Llama 3.1 70B on 4 x H200 141GB: weights 35.3 GB per GPU, KV memory 366.5 GB in total = 1,118,446 tokens BF16 / 2,236,893 FP8 | conversations that fit: 4,096 tok BF16 273, FP8 546; 32,768 tok BF16 34, FP8 68
Llama 3.1 70B on 1 x B200 180GB: weights 141.1 GB per GPU, KV memory 20.9 GB in total = 63,759 tokens BF16 / 127,518 FP8 | conversations that fit: 4,096 tok BF16 15, FP8 31; 32,768 tok BF16 1, FP8 3
Llama 3.1 70B on 2 x B200 180GB: weights 70.6 GB per GPU, KV memory 182.9 GB in total = 558,143 tokens BF16 / 1,116,287 FP8 | conversations that fit: 4,096 tok BF16 136, FP8 272; 32,768 tok BF16 17, FP8 34
70B on one H100 80GB: usable 72 GB vs weights 141.1 GB -> does not fit
70B on one H200 141GB: usable 127 GB vs weights 141.1 GB -> does not fit
70B on one B200 180GB: usable 162 GB vs weights 141.1 GB -> fits, 20.9 GB left for KVReferences
- W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. (Figures 1, 2, 3 and 13; 800 KB per token for OPT-13B; 20.4% to 38.2% of KV memory used in existing systems.)
- A. Vaswani et al. Attention Is All You Need. NeurIPS 2017.
- N. Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. 2019.
- J. Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.
- DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.
- G.-I. Yu et al. Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.
- NVIDIA, H100 Tensor Core GPU (3.35 TB/s memory bandwidth, 80 GB), H200 (141 GB) and the Blackwell datasheet (HGX B200: 180 GB per GPU).
- vLLM documentation: Quantized KV cache.
- Model configs: Llama 3.1 8B, Llama 3.1 70B, Qwen2.5-0.5B.
Next: Part 3: How vLLM Serves Thousands, where paging removes the three wastes.