vLLM on a Free GPU: Run It, Load It, Break It

A real vLLM server on a free Colab T4, measured like a production service: KV cache, batching, the latency knee, prefix caching, preemption, failures.

Seven parts of this series took an inference engine apart. Most of their numbers came from small models on an Apple GPU and from simulators, because the subject was the ideas inside the engine. This part does the opposite. It runs the real thing: vLLM 0.33.0, serving a real model on a real NVIDIA GPU, the way a production team would run it.

The GPU is a free one. Google Colab gives anyone with a Google account an NVIDIA T4 for a few hours at a time. It is an old card, from 2018. That turns out to be useful: its limits make every effect in this part easy to see.

Here is the result that matters most. With one user, the server wrote about 61 tokens per second. With 8 users at the same time, it wrote about 307, about 5 times as much, and each user's stream slowed only a little. With 128 users it wrote about 668, about 11 times as much. But now each user got only about 6 tokens per second. Where between those two points you run your server is most of what this part is about.

Open the notebook in Colab   or download the notebook.

Every number on this page comes from one full run of that notebook, on 10 November 2026. The run served Qwen/Qwen2.5-1.5B-Instruct in float16 on a Colab Tesla T4 (15.0 GiB), with vLLM 0.33.0. Your numbers will differ by a few percent. The shapes will not.

The page follows the notebook's 11 steps. Each step asks one question, shows what we saw, and ends with what it means for a real server.

1. What Colab gives you, and what it does not

Colab is a notebook service. You get a Linux machine with a GPU for a few hours, and you run code in cells. It is meant for experiments, not for serving users. That is fine here. We want to learn how a serving engine behaves, and for that one GPU and one model are enough.

Three facts about the GPU decide most of what follows.

T4 (free)L4 (paid)A100 (paid)
Memory16 GB (15 GiB usable)24 GB40 or 80 GB
Memory bandwidth320 GB/s300 GB/s1,555 to 2,039 GB/s
Compute capability7.5 (Turing)8.9 (Ada)8.0 (Ampere)
bfloat16noyesyes
FP8noyesweights only, slower path
Attention kernel in vLLMTritonFlashAttentionFlashAttention

The T4 has no hardware for bfloat16, the 16-bit number format most models are trained in. So we run the model in float16, the older 16-bit format, and we tell the server so with --dtype half.

What Colab cannot give you matters as much. There is no Docker, so you cannot run vLLM's official container image. There is one GPU, so there is no tensor parallelism. The server is reachable only from inside the notebook machine. And the session ends after some hours, or sooner if you leave it idle. A later section lists what that leaves out.

The model

The notebook serves Qwen/Qwen2.5-1.5B-Instruct. It has 28 layers and 1.5 billion weights, about 3 GB in float16. It is small enough to leave most of the T4's memory for the KV cache, and good enough to give sensible answers. It needs no Hugging Face login.

A larger model would make each step slower and leave a smaller KV cache. The effects in this part would look the same. On an L4 or an A100, set MODEL = "Qwen/Qwen2.5-7B-Instruct" in the notebook and compare.

2. Start the server, and read what it tells you

The question: how long does a server take to start, and how much room does it have for conversations?

One command starts the server:

bash
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --dtype half --max-model-len 4096 --gpu-memory-utilization 0.85 \
  --api-key colab-demo-key --port 8000

Three settings on that line matter in production:

  • --max-model-len 4096 caps one request at 4,096 tokens, prompt and answer together.
  • --gpu-memory-utilization 0.85 lets vLLM take 85% of the GPU's memory. The default is 0.92. We leave a little more room for Colab.
  • --api-key makes every request to /v1 carry a key. Without it, anyone who can reach the port can use your GPU.

The notebook runs this command in the background, writes its log to a file and waits until GET /health answers.

Where the startup time goes

The first start took 295 s, about five minutes, from the command to the first healthy answer. That is a cold start.

Later in the notebook, the server restarts four more times on the same machine. Those warm starts took between 68 s and 108 s. The log shows where the difference comes from:

PhaseWhat happensCold startFirst warm start
Load the weightsDownload about 3 GB (cold start only), then copy them into GPU memory160 s14 s
torch.compileFuse the model's operations into faster kernels; the result is cached on disk20 s0.3 s
Engine startCompile, measure memory, make the KV cache, record CUDA graphs73 s22 s
From command to healthyAll of the above, plus starting Python and the HTTP server295 s79 s

Most of the cold start was the download. The same "Model loading" line said 160 s on the cold start and 14 s on the warm one. The compile step fell from 20 s to under a second, because its result came from the cache.

The lines that matter

Most of the startup log is noise. These lines are not:

plain text
Using TRITON_ATTN attention backend
Model loading took 2.98 GiB memory and 160.272162 seconds
[backbone] torch.compile took 20.50 s in total
Graph capturing finished in 11 secs, took 0.29 GiB
Available KV cache memory: 8.38 GiB
GPU KV cache size: 313,920 tokens, Maximum concurrency for 4,096 tokens per request: 76.64x
Graph capturing finished in 7 secs, took 0.18 GiB
init engine (profile, create kv cache, warmup model) took 73.35 s (compilation: 20.50 s)

Read them from the top.

The attention backend is TRITON_ATTN. vLLM picks the fastest attention kernel your GPU supports. FlashAttention and FlashInfer need compute capability 8.0. So on a T4, vLLM falls back to a kernel written in Triton, a language for writing GPU kernels in Python. The Triton kernel works. But on this older GPU it is slower at prefill than a native kernel. That is one reason the TTFT numbers below are higher than on an L4.

The weights took 2.98 GiB.

The KV cache got 8.38 GiB, which holds 313,920 tokens. The same line turns that into a capacity: 76.64 requests of the full 4,096 tokens could keep their caches in memory at once.

That last number is the one to remember. It is the server's memory budget for conversations. Sections 4 to 8 are about how it gets spent.

Tesla T4: 15.0 GiB of memory, 85% given to vLLMweights2.98 GiBworking memory: 1.39 GiBKV cache8.38 GiBoutside vLLM's share: 2.25 GiBKV cache: 313,920 tokens of 28 KiB each, enough for 76.6 requests of the full 4,096 tokens at once.Working memory: activations for the largest batch, measured by a profile run at startup, plus the CUDA graphs.
Where the GPU's memory went. vLLM takes the share set by --gpu-memory-utilization, loads the weights, keeps working memory for the largest batch, and gives everything left to the KV cache.

3. The first requests

The question: what does one user feel?

The server speaks the OpenAI API. Any program written for OpenAI's service works with it if you change two settings, the base URL and the key:

python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="colab-demo-key")
r = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct", temperature=0, max_tokens=120,
    messages=[{"role": "user", "content": "Explain in three sentences why an LLM server batches requests."}])
print(r.choices[0].message.content)

The model answered:

An LLM (Large Language Model) server batches requests to improve efficiency and scalability. By processing multiple requests simultaneously, the server can handle more queries per unit of time, reducing wait times for users and optimizing resource utilization. This approach also helps in managing memory usage effectively, as it avoids unnecessary storage overhead associated with individual request handling. Additionally, batching allows for better load balancing across servers, ensuring that no single machine becomes overwhelmed while others remain underutilized.

That compatibility is a big part of why vLLM is easy to put into production. Your application, your SDKs and your tools stay the same. Only the address changes.

One user, timed

With stream=True, the server sends each token as soon as it exists. The notebook times two things on the client side.

For a short prompt and a 256-token answer, three runs gave:

RunTTFTTPOTTokens per second
132 ms14.7 ms68
236 ms14.7 ms68
329 ms14.8 ms68

So one conversation gets its first token in about 32 ms and then streams at about 68 tokens per second. That is far faster than anyone reads. It is also a waste of the GPU, as Section 5 shows.

Why decode is slow: a quick check

Part 1 explained why the two numbers differ so much. Prefill handles all the prompt tokens in one pass. Decode writes one token per step, and every step must still read all 2.98 GiB of weights from memory.

The T4 reads memory at most at 320 GB/s. Reading the weights once at that speed takes:

tread=weight bytesbandwidth=3.20 GB320 GB/s≈10.0 mst_{\text{read}} = \frac{\text{weight bytes}}{\text{bandwidth}} = \frac{3.20 \text{ GB}}{320 \text{ GB/s}} \approx 10.0 \text{ ms}

Read it left to right: one decode step can never be shorter than the time to read every weight once. The measured TPOT was 14.7 ms. The rest goes to reading the KV cache, launching kernels and work on the CPU, and no kernel reaches the full bandwidth. So the weight read explains most of each step.

4. The KV cache budget, checked by hand

The question: can we predict the KV cache size from the model's shape?

The log said the KV cache holds 313,920 tokens. Predicting that number yourself is the best way to make sure you know what it means.

Think of the cache as a notebook that each layer keeps, with one line per token. The size of a line depends on how many key and value heads the layer has, and how long each head's vector is.

bytes per token=2×L×Hkv×dhead×b\text{bytes per token} = 2 \times L \times H_{kv} \times d_{\text{head}} \times b

The 2 counts one key and one value. LL is the number of layers. HkvH_{kv} is the number of key/value heads per layer. dheadd_{\text{head}} is the length of one head's vector. bb is the bytes per number, 2 for float16.

For Qwen/Qwen2.5-1.5B-Instruct, the model's config.json gives L=28L = 28, Hkv=2H_{kv} = 2 and dhead=128d_{\text{head}} = 128:

2×28×2×128×2=28,672 bytes=28 KiB per token2 \times 28 \times 2 \times 128 \times 2 = 28,672 \text{ bytes} = 28 \text{ KiB per token}

Divide the cache memory by that:

8.38 GiB×23028,672 bytes≈313,824 tokens\frac{8.38 \text{ GiB} \times 2^{30}}{28,672 \text{ bytes}} \approx 313,824 \text{ tokens}

The log said 313,920. The two differ by only 96 tokens. The small gap is rounding: the log prints the memory with only two decimals. Turn it around and 313,920 tokens at 28,672 bytes each is 8.3826 GiB, which the log rounds to 8.38. The engine stores the cache in fixed-size blocks of 16 tokens, the pages of Part 3. So the token count is always a whole number of blocks.

Sizing the KV cache for your traffic

The same arithmetic tells you whether a server has enough room. Multiply the requests you expect to run at once by the tokens each one holds at its longest, prompt plus answer:

tokens needed=requests at once×(prompt tokens+output tokens)\text{tokens needed} = \text{requests at once} \times (\text{prompt tokens} + \text{output tokens})

The requests in Sections 5 and 6 have 512 prompt tokens and 128 output tokens, 640 in all. The cache holds about 490 of them at once. That is more than vLLM will ever run together. The scheduler also caps the number of requests in one batch, and Section 6 shows that cap at work. So on this GPU, with this model, memory is not the first limit. Section 8 makes it the limit on purpose.

5. Batching: the free lunch, and its bill

The question: what happens when many users share the GPU?

This is the most important experiment in the notebook. We send identical requests: 512 random prompt tokens and exactly 128 output tokens. We keep a fixed number of them in flight at the same time: 1, then 2, 4, 8 and so on up to 128. As soon as one finishes, the next one starts. The tool is vllm bench serve, the load generator that ships with vLLM:

bash
vllm bench serve --backend openai --base-url http://localhost:8000 --model Qwen/Qwen2.5-1.5B-Instruct \
  --dataset-name random --random-input-len 512 --random-output-len 128 --ignore-eos \
  --max-concurrency 64 --num-prompts 256 \
  --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,90,99 --save-result

--ignore-eos makes every answer exactly 128 tokens long, so every run does the same work. The random prompts are random token IDs. Their text is noise, but their cost to the GPU is real.

Whole server: output tokens per second (higher is better)02004006008001248163264128requests in flight1 in flight: 61 tok/s2 in flight: 111 tok/s4 in flight: 195 tok/s8 in flight: 307 tok/s16 in flight: 441 tok/s32 in flight: 534 tok/s64 in flight: 625 tok/s128 in flight: 668 tok/s61668Each user: milliseconds per token (lower is better)0501001502001248163264128requests in flight1 in flight: 15.6 ms per token2 in flight: 16.6 ms per token4 in flight: 17.2 ms per token8 in flight: 20.9 ms per token16 in flight: 28.6 ms per token32 in flight: 49.3 ms per token64 in flight: 91.2 ms per token128 in flight: 178.7 ms per token15.6 ms178.7 ms
Top: output tokens per second for the whole server. Bottom: milliseconds per token for each user. Both against the number of requests in flight, on a log scale.
In flightServer output, tok/sSpeed-up vs 1Per-user TPOT, p50Per-user tok/sTTFT p50TTFT p99
1611.0x15.6 ms64118 ms211 ms
21111.8x16.6 ms60219 ms336 ms
41953.2x17.2 ms58427 ms571 ms
83075.1x20.9 ms48667 ms1.00 s
164417.3x28.6 ms35966 ms1.80 s
325348.8x49.3 ms201.27 s3.51 s
6462510.3x91.2 ms111.34 s7.08 s
12866811.0x178.7 ms61.88 s14.40 s

Read the table in two halves.

The top half is the free lunch. From 1 to 8 requests in flight, the server's output grew about 5 times. Each user's time per token grew only from 15.6 to 20.9 ms.

Part 1 promised this. A decode step reads all the weights once, whether it writes a token for one request or for eight. The extra arithmetic for eight tokens is small next to that read. So eight requests per step cost little more than one.

The bottom half is the bill. From 8 to 128 in flight, throughput grew only about 2.2 times. Each user's time per token grew from 20.9 to 178.7 ms, so each stream fell to about 6 tokens per second. That is slower than many people read. The TTFT p99 grew from 1.00 s to 14.40 s.

Two things end the lunch. As the batch grows, the arithmetic and the KV cache reads grow with it, until they, not the weight read, set the step time. And every new request must be prefilled: 512 prompt tokens, in the same steps as everyone else's decodes. With more requests in flight, more prefills crowd into those steps. New requests wait longer for their first token, and every running stream stutters while a prefill runs.

6. Past capacity: where latency breaks

The question: how much traffic can the server take before users notice?

Real users do not wait for each other. They arrive at random times, whether the server keeps up or not.

The notebook takes the best request rate from Section 5, 5.22 requests per second, as the server's capacity. Then it sends about a minute of traffic at 25%, 50%, 75%, 90%, 100% and 120% of that rate.

It also measures goodput against a latency target. We chose one that suits a chat product: TTFT under 1 second and TPOT under 100 ms, for each request.

Offered loadArrival rate, req/sServed, req/sGoodput, req/sTTFT p50TTFT p99TPOT p50TPOT p99
25%1.301.251.25164 ms337 ms20.4 ms27.3 ms
50%2.612.482.48182 ms434 ms28.7 ms46.6 ms
75%3.913.623.62268 ms586 ms56.1 ms72.0 ms
90%4.704.233.02383 ms1.03 s83.4 ms114.5 ms
100%5.224.640.90505 ms1.38 s145.7 ms175.2 ms
120%6.265.010.001.41 s6.46 s277.3 ms357.3 ms
arrivedservedmet the SLO (goodput)25% loadgoodput 1.25 req/s50% loadgoodput 2.48 req/s75% loadgoodput 3.62 req/s90% loadgoodput 3.02 req/s100% loadgoodput 0.90 req/s120% loadgoodput 0.00 req/s02468requests per second
For each load: how many requests per second arrived, how many the server finished, and how many met the SLO (TTFT under 1 s and TPOT under 100 ms).

Read the goodput column from the top. Up to 75% load, every request met the SLO. At 90%, about 29% of them missed it. At 100%, goodput fell to 0.90 requests per second. At 120%, it was 0.00: the server still finished 5.01 requests per second, and not one of them was fast enough.

So the useful capacity of this server is not 5.22 requests per second. It is about 3.62, the goodput at 75% load.

Now look at which target broke. At 100% load, the TTFT p50 was still 505 ms, well under 1 second. The TPOT p50 was 146 ms, far over 100. The streams broke before the first tokens did.

100 ms300 ms1 s3 s10 s25%50%75%90%100%120%offered load, as a share of measured capacityTTFT (log scale)capacity25% load: TTFT p50 164 ms50% load: TTFT p50 182 ms75% load: TTFT p50 268 ms90% load: TTFT p50 383 ms100% load: TTFT p50 505 ms120% load: TTFT p50 1.41 s25% load: TTFT p99 337 ms50% load: TTFT p99 434 ms75% load: TTFT p99 586 ms90% load: TTFT p99 1.03 s100% load: TTFT p99 1.38 s120% load: TTFT p99 6.46 sTTFT p50TTFT p99
TTFT p50 and p99 against the offered load, as a share of the capacity measured in Section 5. The vertical axis is logarithmic.

TTFT did break too, one step later. At 120%, the TTFT p99 was 6.46 s. Above capacity, more requests arrive each second than leave, so the backlog grows for as long as the overload lasts.

Inside the server during the overload

During the 120% run, the notebook read /metrics every half second. It recorded three numbers that a production dashboard shows: requests running, requests waiting, and the share of the KV cache in use.

0751502253000163147627893seconds since the run startedrequestsmost running at once: 2560%50%100%KV usedrunningwaitingKV cache used (right scale)
Requests running and waiting inside the server, and the share of the KV cache in use, during the run at 120% of capacity. The first 19 seconds are the load tool starting up.

The picture is not the one most people expect. The running line climbed for about 45 seconds, then flattened at 256. Until then, the waiting line stayed short: about 2 on average. Its peak, 25, came only after the batch was full. The KV cache never passed 48.0% full.

Here is why. vLLM admits a new request whenever there is memory for it. So in an overload, the backlog does not sit in the queue at first. It sits inside the batch.

The batch grows, every step gets longer, and every user's stream slows down. That is the TPOT column above. Only when the batch hits the scheduler's cap does the queue really grow. In this run that cap was 256 requests, which matches vLLM's --max-num-seqs limit.

That also checks the sizing sum from Section 4. 256 requests of at most 640 tokens need at most 163,840 tokens, about half of the 313,920 in the cache. The gauge showed 48.0%.

7. Prefix caching: pay for the shared prompt once

The question: how much does reusing a shared start of the prompt save?

Many products send the same long start with every request: a system prompt, tool definitions, a document. If the server keeps the KV cache of that start, later requests can skip its prefill and compute only what is new.

The workload: every request starts with the same 2,048 random tokens and adds 128 tokens of its own. It asks for 64 output tokens, with 8 requests in flight. We run it on a normal server, where prefix caching is on by default. Then we restart with --no-enable-prefix-caching and run it again.

cache offp50 3.71 sp99 6.81 scache onp50 735 msp99 1.85 sTime to first token, shared 2,048-token start + 128 own tokens
TTFT p50 and p99 with prefix caching off and on, for prompts with a shared 2,048-token start.
Prefix cachingCache hit rateTTFT p50TTFT p99TPOT p50Requests per second
off(off)3.71 s6.81 s99.0 ms0.81
on93.5%735 ms1.85 s29.2 ms3.05

The hit rate comes from the server's own counters: vllm:prefix_cache_hits_total divided by vllm:prefix_cache_queries_total, read before and after the run. With the cache on, the server found 93.5% of the prompt tokens it looked up already in the cache.

The median TTFT fell from about 3.7 s to 735 ms, about 5 times faster. The server finished about 4 times as many requests per second. Even the streams got faster, from 99 to 29 ms per token, because fewer long prefills crowded into the decode steps.

Why so large? Without the cache, every request prefills about 2,176 tokens. With it, the first request pays for the shared 2,048, and every later one prefills only its own 128. On a T4, where prefill runs on the slower Triton attention kernel, every prefill token saved counts even more.

8. A KV cache too small: preemption

The question: what happens when the KV cache runs out?

Section 4 showed that this server has room for hundreds of requests. Production servers often do not. Bigger models, longer prompts and longer answers fill the cache quickly.

vLLM starts a request when there is room for its prompt. As the answer grows, the request needs new blocks. When a running request needs a block and none is free, the scheduler must take memory from someone.

The notebook restarts the server with --kv-cache-memory-bytes set to fit only 6,144 tokens. Then it sends 32 requests at a time, each with 512 prompt tokens and 512 output tokens. Together they would need 32,768 tokens of cache, more than five times what there is. The same job also runs on a normal server, for comparison.

ServerKV cache, tokensPreemptionsOutput tok/sTTFT p50TTFT p99TPOT p50End-to-end p99
normal320,76807141.27 s4.43 s41.5 ms25.44 s
small KV6,1447032035.82 s43.33 s22.6 ms57.58 s

The small server ran out of room 70 times. Each time, a request lost its cache and had to rebuild it, so part of the GPU's work was done twice. Throughput fell to 44.8% of the normal server's, a bit less than half.

The bigger effect is on waiting. The cache has room for only about 6 requests of 1,024 tokens at once. So most of the 32 requests in flight sit in the queue. The median TTFT went from about 1.3 s to about 36 s.

Now look at the TPOT column. The small server's streams were faster: 22.6 ms per token against 41.5. Only a few requests ran at a time, so each step was short. A dashboard that showed only TPOT would say this server was healthier. Its users waited more than half a minute for a first word.

9. Break it on purpose

The question: what does failure look like from the client's side?

Production skill is knowing what a failure looks like before it happens at 3 a.m. The notebook tries four.

Clean failures: bad requests

What we sentStatusWhat the server said
The word hello 4,596 times, over the 4,096-token limit400This model's maximum context length is 4096 tokens. However, your prompt contains at least 4097 input tokens, which exceeds this limit even without any output t
The wrong API key401Unauthorized
max_tokens of 4,106400max_tokens=4106 cannot be greater than max_model_len=4096. Please request fewer output tokens. (parameter=max_tokens, value=4106)

These are good failures. The server rejects the bad request at once, says why, and keeps serving everyone else. A client should treat any 4xx status as "my request is wrong": fix it, do not retry it.

Two of these protect the server. The 4,096-token limit from --max-model-len stops one huge prompt from taking the KV cache from everyone. The API key stops strangers from using your GPU. Note that /health and /metrics stay open without the key, so keep the port itself private.

The ugly failure: the server is gone

The notebook started a load test with 32 requests in flight in a background thread. After 15 seconds it killed the server with SIGKILL, the way an out-of-memory killer or a failed machine would.

Result
Requests completed0
Requests failed200
Time until all of them had failed0.7 s
/health after the killConnectionError (no answer at all)

One honest note about this run. The load tool spent more than 15 seconds starting up, so the kill came before its first request. The server log confirms that no request from the test ever reached it. So this drill measured a server that was already gone, not one that died halfway through an answer. Every one of the 200 requests failed within 0.7 s, with a connection error.

That fast failure is the better case. When a server dies in the middle of an answer, the requests in flight are lost too. Their KV caches lived in the GPU memory of a process that no longer exists, so nothing inside vLLM can save them. The client sees a stream that stops partway.

That is why production setups never rely on one server. They run several replicas behind a load balancer that calls /health and stops routing to a dead one. Clients retry failed requests with exponential backoff and jitter: they wait a little, then longer, with a random part added. Then a thousand clients do not all retry at the same moment.

10. FP8 weights: skipped on the T4

The question: what do 8-bit weights buy?

FP8 stores each weight in 8 bits instead of 16. vLLM can convert the weights while it loads them (--quantization fp8). The weights then take half the memory, so the KV cache grows, and each decode step reads half the bytes.

The notebook skips this step on the T4. FP8 needs a GPU with compute capability 8.0 or newer, and the T4 is 7.5. On an L4 the step runs, and on an A100 it runs on a slower path that saves memory but may not save time. We have no measured FP8 numbers for this page, so it shows none. Part 6 measures quantization on a real model.

11. Save everything, and run it yourself

The last step writes vllm_colab_results.json, with every number above plus the GPU and the versions. It also writes a zip with every server log and raw benchmark file. This page was built from exactly those two files.

Open the notebook in Colab

  1. Open the notebook in Colab and choose Runtime > Change runtime type > T4 GPU.
  2. Run every cell from the top with Shift + Enter. The whole run takes about an hour on a T4. Set QUICK = True in the settings cell to cut it to about 25 minutes, with noisier numbers.
  3. The last cell downloads vllm_colab_results.json and vllm_colab_run.zip. Run it after any experiment you care about: if Colab disconnects, everything in memory is gone.

Then change one thing and run it again. Try Qwen/Qwen2.5-0.5B-Instruct and Qwen/Qwen2.5-3B-Instruct, and see how the KV cache budget and the knee move. Try an L4: the attention backend changes, and step 10 runs.

Running vLLM in production: what this run teaches

Every lesson above, in one place.

The settings that matter.

SettingWhat it controlsWhat we saw
--gpu-memory-utilizationThe share of GPU memory vLLM may take; all of it beyond weights and working memory becomes KV cache0.85 gave 8.38 GiB of KV cache
--max-model-lenThe longest prompt plus answer for one requestLonger requests were rejected with a 400
--max-num-seqsThe most requests in one batchThe batch stopped at 256, and only then did a queue form
--dtypeThe number format of weights and cachehalf on the T4, which has no bfloat16
--api-keyA key on every /v1 requestA wrong key got a 401
--no-enable-prefix-cachingTurns off prefix caching, which is on by defaultTTFT p50 about 5 times slower with a shared prompt
--kv-cache-memory-bytesA fixed KV cache size, in place of the share aboveA small cache gave 70 preemptions

What to watch on /metrics.

MetricWhat it tells youThe warning sign in this run
vllm:num_requests_runningRequests in the batch nowStuck at the --max-num-seqs cap (256)
vllm:num_requests_waitingRequests in the queueAbove zero once the batch was full
vllm:kv_cache_usage_percShare of KV blocks in useNear 100% together with preemptions
vllm:num_preemptions_totalRequests stopped to free memoryAny steady rise (70 in one small-cache run)
vllm:prefix_cache_hits_total ÷ vllm:prefix_cache_queries_totalShare of prompt tokens reusedNear zero when you expected reuse
vllm:time_to_first_token_secondsTTFT, as a histogramp99 over your SLO
vllm:inter_token_latency_secondsThe gaps between tokensp99 over your SLO: on vLLM this breaks first

How to size the KV cache. Multiply the requests you expect at once by their longest prompt plus answer, and compare with the log's "GPU KV cache size" line. Check the result on the running server: the KV gauge should match the sum.

Where latency breaks. Measure capacity with a closed loop. Then find the knee with an open loop and an SLO, and plan with goodput, not throughput. Here the closed loop said 5.22 requests per second, and the SLO said about 3.62.

When prefix caching pays. When many requests share a long start. Measure the hit rate, and put the fixed parts of the prompt first.

What preemption looks like. A rising preemption counter, a full KV cache, long waits for the first token, and streams that look fine.

How it fails. Bad requests fail cleanly with a 4xx. A dead server fails everything, fast. Run replicas behind health checks, and retry with backoff.

All the results in one table

WhatMeasured value
SetupTesla T4 (15.0 GiB), vLLM 0.33.0, Qwen/Qwen2.5-1.5B-Instruct, float16, max_model_len 4,096, gpu_memory_utilization 0.85
Attention backendTRITON_ATTN
Cold start / warm start294.7 s / 79.2 s
Weights in GPU memory2.98 GiB
KV cache (log)8.38 GiB = 313,920 tokens = 76.64 requests of 4,096 tokens
KV cache (by hand)28,672 bytes per token, predicts 313,824 tokens
One user, streamingTTFT about 32 ms, 14.7 ms per token
1 to 128 requests in flight61 to 668 output tok/s (11.0 times); TPOT p50 15.6 to 178.7 ms
Capacity used for the load test5.22 req/s (best closed-loop rate)
Best goodput (TTFT under 1 s, TPOT under 100 ms)3.62 req/s, at 75% load
Goodput at 100% and 120% load0.90 and 0.00 req/s
Prefix caching off / on (2,048 shared tokens)TTFT p50 3.71 s / 735 ms (5.0 times); hit rate 93.5%
Small KV cache vs normal6,144 tokens: 70 preemptions, TTFT p50 35.82 s vs 1.27 s
Bad requeststoo long: 400; wrong key: 401; max_tokens too big: 400
Server killed0 completed, 200 failed; /health then gave ConnectionError
FP8 weightsskipped: needs compute capability 8.0 or newer, the T4 is 7.5

What Colab cannot teach

Everything above ran on one GPU, in one notebook, for free. That covered the engine: its memory budget, batching, the latency knee, caching, preemption and failure. It did not cover the system around the engine. Those parts need a rented GPU machine or a cluster:

TopicWhy Colab cannot do itWhat to try
ContainersNo Docker on ColabRun the official vllm/vllm-openai image on a GPU machine
Dashboards and alertsNo long-running servicesScrape /metrics with Prometheus, chart it in Grafana, alert on TPOT p99 and TTFT p99
Several GPUsOne GPU per session--tensor-parallel-size 2 with a model that does not fit on one GPU (Part 7)
Several replicasOne server at a timeTwo servers behind a router: the vLLM production stack or llm-d on Kubernetes
Autoscaling and cold startsNo orchestrationScale replicas on load, and watch the cold start from Section 2 decide how fast it reacts
Real trafficRandom tokens with fixed lengthsReplay real prompts with their real lengths and arrival times

The workflow from this part carries over unchanged:

  1. Start the server and read the startup log.
  2. Check the KV cache size against your traffic.
  3. Measure capacity with a closed loop.
  4. Find the knee with an open loop, and measure goodput against an SLO.
  5. Watch the batch, the queue, the KV cache and the preemption counter.
  6. Break it before your users do.
Research notes: how far to trust these numbers
  • One run, one machine. Each number is one benchmark run on one Colab session. Colab machines are shared, and GPU clocks, CPU speed and disk speed differ between sessions. Expect a few percent of difference between runs, more on TTFT p99, which depends on a handful of requests.
  • The client shares the machine. vllm bench serve runs on the same 2-CPU machine as the server and takes some CPU time. At high request rates, the client can add a little latency of its own.
  • Random prompts. Random token IDs have the right lengths but not the right content. Content does not change the cost of a step, but real lengths vary, and varied lengths make batching less even than here.
  • Fixed output lengths. --ignore-eos makes every answer the same length. Real answers stop at different points, which frees KV cache earlier and changes the batch over time.
  • The capacity estimate. Section 6 takes the best closed-loop request rate from Section 5 (at 128 in flight) as capacity. That row already had a TTFT p99 of 14.40 s, so it is an upper bound, not a safe operating point.
  • One failed request. The 50% load run finished 155 of 156 requests; one failed. The saved files do not record why.
  • The killed-server drill. As Section 9 says, the kill landed before the load tool sent its first request. A run that waits until requests are running before the kill would show a server dying mid-answer.
  • Different servers, slightly different caches. The first server got 8.38 GiB of KV cache; the restarted ones got 8.57 GiB. The prefix caching "off" run and the preemption baseline ran on restarted servers. The difference is about 2% and does not change any comparison.
  • The --max-num-seqs cap. The startup log does not print it. The value 256 is what the running gauge never went above.
  • Versions. vLLM 0.33.0. Defaults change between versions: in this version prefix caching and chunked prefill are on by default, and --gpu-memory-utilization defaults to 0.92. Flag names were checked against the source of this version.

Exercises

  1. Your prompts average 1,500 tokens and answers 500, and you want 100 requests running at once. With this model's 28 KiB per token, how much KV cache memory do you need? Does the T4 have it?
  2. Your SLO is TTFT under 2 seconds and 15 tokens per second per user. Using the table in Section 5, which concurrency would you run at, and what throughput would you get?
  3. In Section 6, the waiting queue stayed small while TPOT broke the SLO. Which setting would you change to trade slow streams for a longer wait before the first token, and what would you alert on afterwards?
  4. A product sends a 3,000-token system prompt, then the date, then the user's question. Why might its prefix cache hit rate be low, and how would you fix it?
  5. Design a version of the killed-server drill that is sure to kill the server while requests are running. Which metric would you watch before you send the kill?

Summary

The series so far

PartQuestionMain idea
1Why is generating slow?Prefill is compute-bound, decode is memory-bound; batching shares each weight read
2What does the model remember?The KV cache, its size per token, and why it limits the batch
3How does an engine fit many users?Paged KV memory, continuous batching, chunked prefill, preemption
4Can we write several tokens per step?A cheap draft, checked in one pass of the big model
5Can requests share work?Prefix caching with a radix tree or hashed blocks
6How do you run it for real?Latency percentiles, goodput, queueing, capacity, cost, failures
7What if one GPU is not enough?Tensor, pipeline and expert parallelism, split prefill and decode
8 (this part)What does it look like on a real GPU?A real vLLM server, measured, loaded and broken, in one free notebook

Next. Run the notebook yourself, then read Part 6 again. Its queueing maths, capacity planning and failure stories will now have numbers from your own server behind them.

References

Papers

  1. W. Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
  2. Y. Zhong et al. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024.
  3. A. Agrawal et al. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024.
  4. J. Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.

Documentation and code

  1. vLLM. Source code at v0.33.0, vllm serve, vllm bench serve, Metrics, Automatic prefix caching, Optimization and tuning.
  2. vLLM issue #60821: Triton attention on Turing GPUs (T4) and its prefill speed.
  3. Qwen. Qwen2.5-1.5B-Instruct model card.
  4. NVIDIA. T4, L4 and A100 specifications; CUDA GPU compute capability.
  5. Google. Colab FAQ.

Companion files

  1. The notebook, its generator, the saved results of the run on this page and its server logs and raw benchmark files.