vLLM, explained
The 2023 paper behind vLLM: it stores the memory of each chat request in small blocks, the way an operating system stores memory in pages, and serves 2 to 4 times more requests on the same GPU. Read with the paper's own text and real code.
Efficient Memory Management for Large Language Model Serving with PagedAttention. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica. SOSP 2023, 2023. arXiv:2309.06180
What you will learn
- Why a language model on a GPU is limited by memory, not by arithmetic, when it writes text
- What the KV cache is, how large it gets, and why it grows and shrinks with every request
- Why storing each request's cache in one contiguous chunk wastes 60% to 80% of the memory
- PagedAttention: attention over keys and values stored in small blocks, as an operating system stores pages
- How vLLM shares blocks between samples, schedules and preempts requests, and runs on many GPUs
- What the experiments show: 2 to 4 times the throughput of earlier systems at the same latency
- How to build a small paged KV cache and serving loop yourself, in Python
Why this paper
A chatbot answers one word at a time. To write the next word, the model looks back at every earlier word of the conversation, so it keeps a record of them in GPU memory: the KV cache. Thousands of people send requests at the same time, and the GPU can serve many of them together only if their caches fit in memory.
This paper shows that the serving systems of 2023 wasted most of that memory. They reserved one large chunk per request, sized for the longest possible answer. The authors borrow an old idea from operating systems: cut memory into small pages and hand them out on demand. The result is PagedAttention and the serving engine vLLM, which serves 2 to 4 times more requests on the same hardware. vLLM became one of the most used open-source engines for serving language models.
How the six parts are organised
Before Part 1 (optional): The KV Cache and Paging. How attention uses keys and values, and how an operating system gives each program its own pages of memory.
- The Big Idea: the title, the abstract, Figures 1 and 2, and §1. Where the memory of a GPU goes, and why most of the KV cache memory was wasted.
- Why Memory Is the Bottleneck: §2 and §3, Figure 3. The two phases of generation, batching, and the three kinds of waste.
- PagedAttention: §4.1 to §4.3, Figures 4 to 7. Attention over blocks, the block table, and the KV cache manager.
- Sharing, Scheduling and Many GPUs: §4.4 to §4.6 and §5, Figures 8 to 10. Copy-on-write, beam search, preemption and swapping, and distributed execution.
- The Experiments: §6 to §10. Throughput against latency, sharing, the ablations, and the limits.
- Build It from Scratch: a small paged KV cache and serving loop in Python, checked against a library model.
How to read the boxes
- A teal "From the paper" box shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, What it says gives the main words of the paper. What it means then explains those lines, and what the authors mean but do not write.
- A yellow definition box explains a technical word the first time it occurs. If you know the word, skip the box.
- The small Paper §1 tag above a heading tells you which section of the paper you read.
- A green "Key takeaways" box ends each part with the facts to remember.
Code blocks show real code, and the output under a code block is what that code printed. The scripts are in code/papers/vllm. They run on a laptop with a small OPT model. The paper used NVIDIA A100 GPUs, and each part says what that changes.
What you need to know first
You need to know what a language model is and what a token is. Some Python helps. The Transformer series explains attention, keys and values in depth, but each part defines what it uses.
Parts
- Background: The KV Cache and Paging
The two ideas the paper joins: the cache of keys and values that a language model keeps while it writes, and the pages an operating system uses to hand out memory. With small runs you can check.
- Part 1: The Big Idea: Memory, Not Arithmetic, Limits LLM Serving
How a chat server answers many users at once, why its GPU runs out of memory before it runs out of speed, and the paper's fix: store each request's memory in small pages. We read the title, the abstract, Figures 1 and 2 and §1, and we measure the KV cache of a real model.
- Part 2: Why Memory Is the Bottleneck
How a language model writes text, why one GPU must serve many requests at once, and where the memory for those requests goes. We read §2, §3 and Figure 3, with runs on a real model.
- Part 3: PagedAttention
How vLLM stores each request's keys and values in small blocks that can sit anywhere in memory, and still computes the same attention. We read §4.1 to §4.3 and build a block table that drives a real model.
- Part 4: Sharing, Scheduling and Many GPUs
How vLLM lets several answers share one prompt's memory, what it does when memory runs out, and how it runs one model on many GPUs. We read §4.4 to §4.6 and §5, and build each piece in Python.
- Part 5: The Experiments
How the paper tests vLLM against FasterTransformer and three versions of Orca, how to read its latency plots, and what the ablations show. We rebuild the main experiments as a small serving simulation on real request lengths.
- Part 6: Build It from Scratch
We write a small vLLM in about 500 lines of Python: blocks, block tables, PagedAttention, a scheduler and parallel sampling. It runs the real OPT-125M, matches the transformers library and serves real requests.