01 · The KV Cache

Everyone budgets GPU memory for model weights. Almost nobody budgets for the KV cache, which is routinely larger than the weights and is the thing that actually determines how many users you can serve.

This page makes it computable. By the end you can open any model's config.json, work out what one token of one conversation costs in bytes, and turn that into a concurrency estimate — before renting a GPU rather than after.

---

The Problem

The 0.5B model from Stage 0 has about 1 GB of weights and your card has 16 GB. That's a 15 GB margin. So why does any of this need thinking about?

Then you hit one of these:

• You serve a 7B model on a 24 GB card. Weights are ~15 GB. It starts. It handles four users. The fifth gets queued and you cannot work out where the other 9 GB went.
• ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache — on a model whose weights fit with room to spare.
• Throughput collapses when one user pastes a long document, and everyone else's requests slow down or get preempted, despite the server being nowhere near a request-count limit.
• Two 7B models behave completely differently on identical hardware. One serves 60 concurrent users, the other 15. Same parameter count, same precision, same card.

Each of these is the same missing number. You budgeted the fixed cost and ignored the variable one:

	Model weights	KV cache
Size	Fixed, known from the model card	Grows with tokens and with users
Scales with concurrency	No — one copy serves everyone	Yes — every request has its own
Scales with sequence length	No	Linearly
In anyone's capacity plan	Always	Almost never

Weights are the entry fee. The KV cache is the per-seat cost, and it is the one that decides how many seats you have.

---

The Idea

You're asked to continue writing a long story, one word at a time, and someone has taken away your short-term memory. To write word 5,000 you must re-read all 4,999 previous words. Then, to write word 5,001, you re-read all 5,000. Each word costs more than the last, and by the end you are spending nearly all your effort re-reading and almost none writing.

The fix is a notebook. As you read each word once, you jot down what you'll need to know about it later. Now writing a new word means glancing at your notes rather than re-reading the story. Reading happens once per word, ever.

The KV cache is that notebook, and three properties of a physical notebook carry over exactly:

• It's per reader. Ten people continuing ten different stories need ten notebooks. There is no sharing — unless they happen to be reading the same opening pages, which is Prefix Caching.
• It grows as you read. One more page read, one more page of notes. Never smaller.
• Desk space is finite. The limit on how many people can work in the room isn't how many chairs there are — it's how many open notebooks fit on the desk.

That last one is the whole page. Your concurrency limit is desk space, not chairs. A server that "supports 256 concurrent requests" has that number as a ceiling it will rarely reach, because it will run out of notebook space first.

---

Under the Hood

Why the notes work, and why they're specifically K and V

A transformer generating text runs attention at every layer. For the token it's currently producing, it computes three vectors — a query (Q), a key (K) and a value (V) — and the token attends over the keys and values of every token before it, including the whole prompt.

The critical property is causality: token 12 attends to tokens 1–11, and can never attend forwards. Which means the K and V vectors computed for token 5 are identical at every subsequent step. They were computed from token 5's embedding and position, neither of which changes. Nothing about generating token 900 alters what token 5's key and value are.

That's what makes caching valid rather than merely convenient:

Vector	Cached?	Why
K (key)	✅ Yes	Fixed once computed; every future token needs it
V (value)	✅ Yes	Same
Q (query)	❌ No	Only the current token's query is ever used. Storing old ones would be dead weight

Hence "KV cache" and not "QKV cache" — a detail worth knowing because it halves the memory you might otherwise expect.

The saving is asymptotic, not marginal. Generating n tokens without a cache means recomputing K and V for every prior token at every step: O(n²) work. With the cache, each token's K and V are computed exactly once: O(n). For a 2,000-token response that is roughly a thousandfold difference in redundant attention work. Nobody serves LLMs without a KV cache. It isn't an optimisation vLLM adds; it's the baseline that creates the memory problem vLLM exists to manage.

The formula, derived

Per token, per layer, you store one K vector and one V vector. Each has as many elements as num_key_value_heads × head_dim. So:

KV bytes per token = 2  ×  num_hidden_layers  ×  num_key_value_heads  ×  head_dim  ×  dtype_bytes
                     │           │                       │                 │            │
                  K and V     every layer          not attention      usually 64      2 for fp16
                              caches its own          heads —          or 128         or bf16
                              separately             see GQA below

Every one of those five numbers is in the model's config.json. Two notes:

• head_dim is often not listed explicitly; it's hidden_size ÷ num_attention_heads.
• The term people get wrong is num_key_value_heads. Using num_attention_heads instead can overstate the answer by 8× on a modern model.

Worked, on two real models

Both figures below come from the actual published configs:

Qwen2.5-0.5B-Instruct — 24 layers, 14 attention heads, 2 KV heads, hidden_size 896, so head_dim = 896 ÷ 14 = 64:

2 × 24 × 2 × 64 × 2 bytes = 12,288 bytes = 12 KB per token

Qwen2.5-7B-Instruct — 28 layers, 28 attention heads, 4 KV heads, hidden_size 3584, so head_dim = 3584 ÷ 28 = 128:

2 × 28 × 4 × 128 × 2 bytes = 57,344 bytes = 56 KB per token

Now turn that into something operational:

	Qwen2.5-0.5B	Qwen2.5-7B
Weights (bf16)	~1 GB	~15 GB
KV per token	12 KB	56 KB
One 4,096-token conversation	48 MB	224 MB
50 such conversations	2.4 GB	11.2 GB
Cache exceeds weights at	~85 concurrent	~67 concurrent

The last row is the point. Past a fairly ordinary level of concurrency, the KV cache is the larger consumer of your GPU — and unlike the weights, it keeps growing.

Why two models of the same size differ several-fold

Look at what the formula does not contain: parameter count. It contains num_key_value_heads, and that number is an architectural choice that varies enormously.

Originally every attention head had its own K and V — multi-head attention (MHA). Then people noticed the KV cache was the bottleneck and started sharing:

Scheme	KV heads	Cache cost	Quality
MHA — multi-head	One set per attention head	Highest	Baseline
GQA — grouped-query	One set per group of heads	Several-fold lower	Very close to MHA
MQA — multi-query	One set, shared by all heads	Lowest	Some degradation

Nearly every modern model uses GQA. The effect is not subtle — the counterfactual for Qwen2.5-7B if it used MHA (28 KV heads instead of 4):

2 × 28 × 28 × 128 × 2 bytes = 401,408 bytes = 392 KB per token     ← 7× the real 56 KB

At 4,096 tokens that's 1.6 GB per conversation instead of 224 MB. Same weights, same quality class, seven times fewer users per GPU.

This is why parameter count is a poor predictor of serving capacity, and why "we're switching from a 13B to a 7B model to serve more users" sometimes produces a much smaller improvement than expected — or occasionally none, if the 7B is an older MHA architecture.

Where the memory actually goes

Reading the chart is the whole capacity model:

• The flat band never moves. Doubling your users doesn't cost a second copy of the weights.
• The wedge is the entire variable cost, and its slope is KV bytes per token × tokens per request.
• Where the wedge hits the ceiling is your concurrency limit. Everything in Stage 4 is either lowering the slope, raising the ceiling, or refusing to let the wedge cross it.

---

Try It

Two experiments. The first needs no GPU and takes a minute. The second confirms the formula against reality, which is the part that makes you trust the arithmetic.

Experiment 1 — cost out any model from its config (no GPU)

# kv_cost.py — KV cache cost per token, from published configs alone.
# pip install transformers
from transformers import AutoConfig

MODELS = [
    "Qwen/Qwen2.5-0.5B-Instruct",
    "Qwen/Qwen2.5-7B-Instruct",
    "mistralai/Mistral-7B-Instruct-v0.3",
    "microsoft/Phi-3-mini-4k-instruct",
]
DTYPE_BYTES = 2      # fp16 / bf16

print(f"{'model':<40} {'L':>3} {'KVh':>4} {'Ah':>4} {'hd':>4} {'KB/tok':>7} {'GB @ 50x4k':>11}")
for name in MODELS:
    cfg = AutoConfig.from_pretrained(name, trust_remote_code=True)
    layers   = cfg.num_hidden_layers
    att_h    = cfg.num_attention_heads
    kv_h     = getattr(cfg, "num_key_value_heads", att_h)          # falls back to MHA
    head_dim = getattr(cfg, "head_dim", None) or cfg.hidden_size // att_h

    per_tok = 2 * layers * kv_h * head_dim * DTYPE_BYTES
    at_scale = per_tok * 4096 * 50 / 1e9                            # 50 users, 4k tokens each
    print(f"{name:<40} {layers:>3} {kv_h:>4} {att_h:>4} {head_dim:>4} "
          f"{per_tok/1024:>7.1f} {at_scale:>11.2f}")

# VERIFIED for the two Qwen rows against the published config.json; others UNVERIFIED
model                                      L  KVh   Ah   hd  KB/tok  GB @ 50x4k
Qwen/Qwen2.5-0.5B-Instruct                24    2   14   64    12.0        2.46
Qwen/Qwen2.5-7B-Instruct                  28    4   28  128    56.0       11.47

Now change one thing

Set kv_h = att_h — the MHA counterfactual — and re-run.

What you should observe: KB/tok for Qwen2.5-7B jumps from 56 to 392, a 7× increase, and the 50-user figure goes from 11.5 GB to over 80 GB. Nothing about the model's size or quality changed; you only removed grouped-query attention.

That single edit explains more about why some models serve cheaply than any benchmark will. The ratio num_attention_heads ÷ num_key_value_heads is a model's serving-efficiency multiplier, and it takes ten seconds to look up.

Two more edits worth making:

• Set DTYPE_BYTES = 1 — the FP8 KV cache case. Everything halves, which is the entire pitch of --kv-cache-dtype fp8 in Dial It In.
• Change 4096 to your actual p99 conversation length from the workload script in Stage 0. This is the number that should drive your capacity plan, and it's usually far below the model's maximum.

Experiment 2 — verify the formula against real memory (needs a GPU)

Hardware: any CUDA GPU with ≥4 GB; a free Colab T4 is plenty.

The formula is only useful if it's true. This measures the actual cache tensors.

# kv_measure.py — measure the real KV cache and compare against the formula.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, device_map="cuda", dtype=torch.float16)

prompt = "The history of paged memory begins " * 40          # a few hundred tokens
ids = tok(prompt, return_tensors="pt").to("cuda")
n_tokens = ids["input_ids"].shape[1]

with torch.no_grad():
    out = model(**ids, use_cache=True)

# Sum the real bytes held in the returned cache
cache = out.past_key_values
measured = sum(t.numel() * t.element_size()
               for layer in cache for t in layer)              # K and V, every layer

cfg = model.config
predicted_per_token = (2 * cfg.num_hidden_layers
                         * cfg.num_key_value_heads
                         * (cfg.hidden_size // cfg.num_attention_heads)
                         * 2)                                  # fp16
predicted = predicted_per_token * n_tokens

print(f"tokens in prompt : {n_tokens}")
print(f"predicted        : {predicted:,} bytes  ({predicted_per_token/1024:.1f} KB/token)")
print(f"measured         : {measured:,} bytes  ({measured/n_tokens/1024:.1f} KB/token)")
print(f"ratio            : {measured/predicted:.3f}")

What you should observe: a ratio of essentially 1.000. Not approximately — the formula isn't a heuristic, it's a description of exactly what gets stored.

Then change one thing: double the prompt length (* 80 instead of * 40) and re-run. Measured bytes double, and KB/token is unchanged. That's the linear growth from The Idea, confirmed on your own hardware.

⚠️ past_key_values has been migrating to a Cache object across transformers versions, and newer versions may not be directly iterable as tuples. If the summation fails, the fix is a version-appropriate accessor rather than a change to the arithmetic. ⚠️ Verify against your installed transformers.

---

Dial It In

The knobs that change KV cache cost. All of them trade something real.

Knob	What it does	Sane start	Move it when
--max-model-len	Caps the per-sequence maximum, so it caps the worst case each request can claim	Your measured p99 conversation length, not the model's maximum	Almost always worth lowering. This is the highest-leverage flag on the page
--kv-cache-dtype fp8	Stores the cache in 8-bit instead of 16-bit: halves it	Leave off until you need it	You need roughly 2× concurrency and can validate quality. Needs Ada/Hopper-class hardware — not a T4
--gpu-memory-utilization	Raises the total budget the cache is carved from	0.9 (default)	Cautiously toward 0.95 on a dedicated card; lower only when sharing the GPU
--enable-prefix-caching	Shares identical prompt prefixes across requests instead of duplicating them	On, if your workload shares prefixes	Covered in Prefix Caching
Model choice (GQA)	num_attention_heads ÷ num_key_value_heads is a several-fold multiplier	Prefer GQA models	This is a selection criterion, not a runtime flag — which is why it gets missed
--max-num-seqs	Caps concurrent sequences	Leave default until measured	It bounds requests, not memory — it can't rescue you from an oversized max-model-len

If you change one thing, change --max-model-len. Serving a 32k context when your p99 request is 3k tokens costs you roughly 90% of your concurrency for traffic that never arrives — the exact inverse-proportionality you derived in Stage 0.

---

Where It Bites You

Budgeting weights and calling it capacity planning. "The model is 15 GB and the card is 24 GB, so we're fine" is the single most common sizing error in LLM serving. It's fine for one user. The question is how many 56-KB-per-token notebooks fit in the remaining 9 GB, and the answer is far smaller than people expect.

Using num_attention_heads in the formula. An easy slip that overstates cache cost by the GQA ratio — 7× on Qwen2.5-7B, 8× on several Llama-family models. If your prediction is wildly pessimistic against measurement, this is almost always why.

Comparing models by parameter count. Parameter count predicts weight memory and says nothing about cache memory. An older MHA 7B can cost several times more per token than a modern GQA 7B, so "same size" models can differ several-fold in users-per-GPU.

Believing a context-length number on a model card. "Supports 128k context" means the architecture handles it. Whether you can serve it is your KV cache arithmetic: at 56 KB/token, one 128k-token conversation is over 7 GB of cache — for a single user. Long-context support and long-context capacity are unrelated claims.

Assuming the cache is shared across a batch. It isn't. Every sequence has its own, and total cost is the sum. Batching amortises weight reads, not cache memory — which is precisely why throughput scales beautifully with batch size right up until you run out of cache and it stops dead.

Enabling FP8 KV cache without checking your hardware or your outputs. It halves memory, which is genuinely large. But it needs recent hardware (a T4 cannot do it), and it's a quantisation of your attention state with a real, model-dependent quality cost. Measure the quality, don't assume it.

Forgetting the prompt is in the cache too. People budget for generated tokens and forget that a 4,000-token RAG prompt occupies cache from the moment prefill completes. For retrieval-heavy workloads the prompt usually dominates the cache, not the output.

Treating a preemption warning as a bug. When the cache is exhausted, vLLM preempts running sequences — evicting them and recomputing their cache on resume. It's the system working as designed under memory pressure, and it's a capacity signal, not an error. See The Scheduler & Block Manager.

---

In Production

KV cache utilisation is your capacity metric. Not GPU utilisation, not request rate. vLLM exposes the fraction of KV cache in use, and it is the number that tells you how close you are to the wall. Rough operating guidance: sustained above ~80–90% means you are at capacity and preemption is imminent; consistently below ~30% means you over-provisioned and are paying for memory that never holds anything. Both are actionable, and neither is visible in latency until it's too late. Observability.

Capacity is measured in tokens, not requests. "We support 200 concurrent requests" is not a capacity statement unless it names a token length. 200 × 500-token chats and 200 × 20,000-token document summaries differ fortyfold in memory on identical hardware. Every capacity number you publish internally should be a pair: concurrency and length.

Admission control belongs upstream. The cache is a hard physical limit, so the only graceful behaviours are queue or reject. Deciding that at your gateway — with request size limits and per-tenant quotas — is far better than discovering it as preemption thrashing inside the engine, where one user's 100k-token request degrades everyone. Security Posture.

What changes at 10× traffic. The weights don't. That's the good news and it's why replicas scale cleanly. What changes is that the cache wedge hits the ceiling, and it does so abruptly: throughput holds up nicely and then falls off as preemption starts, because preempted sequences have to be recomputed and that work is pure waste. The failure isn't gradual, so the alert has to fire on cache utilisation rather than on latency.

The long-context tenant is your noisy neighbour. One user submitting 100k-token prompts can consume the cache of dozens of ordinary users, and nothing about request-count-based limits will stop them. This is a capacity-isolation problem, and it's why Long Context & Chunked Prefill is a separate page rather than a footnote.

---

Check Yourself

Recall the idea

What is the KV cache, and what problem does it solve?

The stored key and value vectors for every token processed so far, per request, per layer. Without it, generating each new token means recomputing K and V for every previous token — O(n²) work over a response. With it, each token's K and V are computed once, making generation O(n). It isn't an optimisation vLLM adds; it's universal, and it's what creates the memory problem vLLM manages.

Why is the query vector not cached?

Only the current token's query is ever used. K and V for past tokens are re-read at every subsequent step, so they're worth storing; a past query is never needed again. This is why it's "KV cache", and it's the reason the memory cost is two vectors per token rather than three.

Why is the cache not shared across a batch?

Because it's the state of one specific sequence — its own tokens, its own positions. Ten concurrent requests hold ten separate caches and total memory is their sum. The exception is identical prefixes, which prefix caching can share.

Which scales with concurrency: weights or cache?

The cache, entirely. One copy of the weights serves every concurrent user; every user brings their own cache. Weights are a fixed entry cost, cache is the per-seat cost, and the per-seat cost sets your capacity.

Explain the mechanics

Write the formula and say where each term comes from.

2 × num_hidden_layers × num_key_value_heads × head_dim × dtype_bytes. The 2 is K and V; layers because every layer caches separately; num_key_value_heads — not num_attention_heads, because GQA shares KV across query heads; head_dim is usually hidden_size ÷ num_attention_heads; and dtype_bytes is 2 for fp16/bf16, 1 for FP8. Every term is in config.json.

Compute it for a model with 32 layers, 32 attention heads, 8 KV heads, hidden_size 4096, bf16.

head_dim = 4096 ÷ 32 = 128. So 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KB per token. A 4,096-token conversation costs 512 MB; fifty of them cost 25 GB — more than the weights of the 8B model this describes.

Two 7B models, same precision, same GPU. One serves four times the users. Explain.

Almost certainly the KV-head ratio. Cache cost scales with num_key_value_heads, so a GQA model with 8 KV heads against 32 attention heads costs a quarter per token of an MHA model where they're equal. Parameter count is identical and irrelevant. Secondary possibilities: a smaller configured max_model_len, or FP8 KV cache on one of them.

Why does the cache make generation O(n) instead of O(n²)?

Without caching, producing token n requires computing K and V for all n−1 prior tokens, and doing that for every token sums to O(n²). Caching makes each token's K and V computed exactly once across the whole generation, so the total is O(n). The per-step attention still reads over all prior tokens, but reading a stored vector is far cheaper than recomputing it.

Reason about a trade-off

A 7B GQA model on a 24 GB card, target 40 concurrent users at ~4,000 tokens. Does it fit?

Weights ~15 GB, leaving ~6.6 GB after the default 0.9 utilisation and framework overhead. At 56 KB/token, 4,000 tokens is 224 MB per user, so 40 users need ~9 GB. It doesn't fit — roughly 28 users is the honest number. Options: cut max_model_len if the real p99 is below 4,000; quantise the weights to recover several GB for cache; FP8 KV cache to halve the per-user cost if the hardware allows; a smaller model; or a bigger card. What won't help is raising max_num_seqs, which permits more sequences without creating memory for them.

Your team wants to advertise 128k context on a 7B model. What do you tell them?

That the architecture supports it and your serving capacity probably doesn't. At 56 KB/token, one 128k conversation is ~7 GB of cache — a single user consuming what dozens of normal users would. So either you accept that a handful of long-context requests can exhaust the server, or you serve long-context traffic on a separate pool with its own limits and pricing. The concrete asks: what fraction of requests will actually use it, and are we willing to let one of them displace fifty others?

When is FP8 KV cache the right call, and when isn't it?

Right when you're cache-bound rather than weight-bound, on hardware that supports it, with a workload where you can measure quality and confirm the degradation is acceptable — it's close to a free doubling of concurrency in that situation. Wrong when you're weight-bound (it doesn't touch weights), on older hardware like a T4 that can't do it, or on quality-sensitive work where nobody has built an evaluation to detect the regression. The failure mode isn't a crash; it's slightly worse outputs that nobody attributes to a memory flag set six months earlier.

Why is KV cache utilisation a better alert signal than p99 latency?

Because it's a leading indicator of a cliff rather than a lagging indicator of one. Latency stays healthy while cache fills, then degrades sharply once preemption starts — by which point users have already felt it and the system is doing wasted recomputation. Cache utilisation crossing ~85% tells you the cliff is coming while there's still time to add a replica or shed load.

---

Cheat Sheet

The formula — memorise this one

KV bytes per token = 2 × num_hidden_layers × num_key_value_heads × head_dim × dtype_bytes

  head_dim = hidden_size ÷ num_attention_heads     (when not given explicitly)
  dtype_bytes = 2 for fp16/bf16,  1 for fp8

total cache for a request = KV bytes per token × (prompt tokens + generated tokens)

Reference values

Model	Layers	KV heads	head_dim	KB/token	4k conversation
Qwen2.5-0.5B-Instruct	24	2	64	12	48 MB
Qwen2.5-7B-Instruct	28	4	128	56	224 MB
8B-class, 32L / 8 KV heads / 128	32	8	128	128	512 MB
Same 7B without GQA (counterfactual)	28	28	128	392	1.6 GB

Facts worth committing to memory

Fact	Consequence
Cache scales with concurrency × length; weights don't scale at all	Weights are the entry fee, cache is the per-seat cost
Only K and V are cached, never Q	Two vectors per token, not three
num_key_value_heads, not num_attention_heads	Using the wrong one overstates cost by up to 8×
The cache is per sequence	Batching amortises weight reads, never cache memory
The prompt occupies cache too	RAG workloads are usually prompt-dominated
Cache exceeds weights at ordinary concurrency	~67 users for a 7B at 4k tokens

Flags

--max-model-len 4096          # cap the per-sequence worst case. Highest leverage flag here
--kv-cache-dtype fp8          # halve the cache. Needs Ada/Hopper — NOT a T4
--enable-prefix-caching       # share identical prefixes instead of duplicating
--gpu-memory-utilization 0.9  # the budget the cache is carved from (default)

The one-line capacity model

concurrent users ≈ (VRAM − weights − overhead) ÷ (KV bytes per token × tokens per request)

---

Sources

• Qwen2.5-0.5B-Instruct config — huggingface.co/Qwen/Qwen2.5-0.5B-Instruct
• Qwen2.5-7B-Instruct config — huggingface.co/Qwen/Qwen2.5-7B-Instruct
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — arxiv.org/abs/2305.13245
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (MQA) — arxiv.org/abs/1911.02150

---

← Back to Core Concepts · Next: Prefill vs Decode →

---

⚠️ Verification checklist (delete before publishing)

Numbers — the two Qwen rows are verified, the rest are not

• Qwen2.5-0.5B-Instruct: 24 layers, 14 attention heads, 2 KV heads, hidden_size 896 → head_dim 64, 12 KB/token. Confirmed against the published config.json.
• Qwen2.5-7B-Instruct: 28 layers, 28 attention heads, 4 KV heads, hidden_size 3584 → head_dim 128, 56 KB/token. Confirmed against the published config.json.
• The Mistral and Phi-3 rows in kv_cost.py are unverified — run the script and either fill in real values or drop those models from the list.
• The "8B-class, 32L / 8 KV heads / 128 head_dim → 128 KB/token" reference row is described generically rather than named. Confirm against a specific model before naming one.
• Weight-size estimates (~1 GB for 0.5B, ~15 GB for 7B at bf16) — confirm from the safetensors index rather than from 2 bytes × parameter count.
• The "cache exceeds weights at ~85 / ~67 concurrent" row — recheck the arithmetic and state the assumed sequence length inline (currently 4,096, implied by the row above it).
• The ~80–90% and ~30% KV utilisation operating bands are rules of thumb, not documented thresholds. Either source them or label them explicitly as judgement.

Code

• Run kv_cost.py. Confirm AutoConfig exposes num_key_value_heads for all four models and that the head_dim fallback is correct for each.
• Run kv_measure.py on a T4 and confirm the ratio really is ~1.000. If it isn't, the page's central claim needs softening — this is the highest-value check on the page.
• Confirm the past_key_values iteration works on the current transformers version; the Cache object migration is flagged inline but needs a concrete version-appropriate snippet.
• Resolved: dtype= is current, torch_dtype= deprecated. Stage 0 page 1 has been updated to match; the whole article and the notebook now use dtype=.

Claims to verify

• FP8 KV cache hardware requirement — confirm which architectures support --kv-cache-dtype fp8 and that T4 is genuinely excluded.
• Confirm the exact flag spellings: --kv-cache-dtype, --enable-prefix-caching, --max-model-len, --max-num-seqs.
• Confirm vLLM exposes a KV cache utilisation metric and capture its exact metric name for the Observability cross-reference.
• Resolved: preempted sequences are recomputed. GPU↔CPU swapping was removed in V1, so there is no alternative mode. Wording corrected.

Rendering

• Three images generated and placed; rows added to image-prompts.md with ✅ status.
• All relative links resolve once target files exist.