# Efficient Memory Management for Large Language Model Serving with PagedAttention

Author: Ishwar Jangid
Canonical URL: https://ishwarj.com/papers/vllm/
Updated: 2026-10-24
Source: https://github.com/ishwar6/ishwar-books/blob/main/content/papers/vllm/index.md


## Why this paper

A chatbot answers one word at a time. To write the next word, the model looks back at every earlier word of the conversation, so it keeps a record of them in GPU memory: the **KV cache**. Thousands of people send requests at the same time, and the GPU can serve many of them together only if their caches fit in memory.

This paper shows that the serving systems of 2023 wasted most of that memory. They reserved one large chunk per request, sized for the longest possible answer. The authors borrow an old idea from operating systems: cut memory into small pages and hand them out on demand. The result is **PagedAttention** and the serving engine **vLLM**, which serves 2 to 4 times more requests on the same hardware. vLLM became one of the most used open-source engines for serving language models.

## How the six parts are organised

**Before Part 1 (optional): [The KV Cache and Paging](https://ishwarj.com/papers/vllm/background-the-kv-cache-and-paging/).** How attention uses keys and values, and how an operating system gives each program its own pages of memory.

1. **[The Big Idea](https://ishwarj.com/papers/vllm/part-1-the-big-idea/)**: the title, the abstract, Figures 1 and 2, and §1. Where the memory of a GPU goes, and why most of the KV cache memory was wasted.
2. **[Why Memory Is the Bottleneck](https://ishwarj.com/papers/vllm/part-2-why-memory-is-the-bottleneck/)**: §2 and §3, Figure 3. The two phases of generation, batching, and the three kinds of waste.
3. **[PagedAttention](https://ishwarj.com/papers/vllm/part-3-pagedattention/)**: §4.1 to §4.3, Figures 4 to 7. Attention over blocks, the block table, and the KV cache manager.
4. **[Sharing, Scheduling and Many GPUs](https://ishwarj.com/papers/vllm/part-4-sharing-scheduling-and-many-gpus/)**: §4.4 to §4.6 and §5, Figures 8 to 10. Copy-on-write, beam search, preemption and swapping, and distributed execution.
5. **[The Experiments](https://ishwarj.com/papers/vllm/part-5-the-experiments/)**: §6 to §10. Throughput against latency, sharing, the ablations, and the limits.
6. **[Build It from Scratch](https://ishwarj.com/papers/vllm/part-6-build-it-from-scratch/)**: a small paged KV cache and serving loop in Python, checked against a library model.

## How to read the boxes

- A **teal "From the paper" box** shows the exact lines of the paper as a highlighted screenshot. Click the screenshot to see it at full size. Under it, **What it says** gives the main words of the paper. **What it means** then explains those lines, and what the authors mean but do not write.
- A **yellow definition box** explains a technical word the first time it occurs. If you know the word, skip the box.
- The small **Paper §1** tag above a heading tells you which section of the paper you read.
- A **green "Key takeaways" box** ends each part with the facts to remember.

Code blocks show real code, and the output under a code block is what that code printed. The scripts are in [`code/papers/vllm`](https://github.com/ishwar6/ishwar-books/tree/main/code/papers/vllm). They run on a laptop with a small OPT model. The paper used NVIDIA A100 GPUs, and each part says what that changes.

## What you need to know first

You need to know what a language model is and what a token is. Some Python helps. The [Transformer series](https://ishwarj.com/papers/transformer/) explains attention, keys and values in depth, but each part defines what it uses.

