Back to Articles/vLLM & LLM Inference Serving

03 · Prefix Caching

23 min read

PagedAttention showed how one request's parallel samples can share prompt blocks by reference. Prefix caching is the same machinery pointed at a far bigger target: sharing across requests, and across time.

The distinction matters. Copy-on-write saved memory within one request. Prefix caching saves computation between requests that arrived minutes apart — it skips prefill for any prompt whose beginning the server has seen before. On the right workload it is close to free, and vLLM's own documentation describes it as "almost a free lunch" that "won't change model outputs."

On the wrong workload it does nothing at all, and this page is equally about telling those apart.

The Problem

Every RAG request re-processes the same document. You retrieve the same 4,000-token policy manual for a hundred queries an hour, and the GPU prefills it a hundred times.
Turn 10 of a conversation is much slower than turn 1. Chat is stateless over HTTP, so each turn re-sends the entire history — and the server re-prefills all of it to generate one more reply.
Your agent loop is expensive in a way that feels wrong. Each step replays the whole trajectory as context, so the prompt grows every iteration and every iteration pays for all of it again.
Your benchmark showed 4× the throughput you see in production, because you sent the same prompt a thousand times and something quietly stopped doing the work.
You enabled prefix caching and nothing improved. Or worse — your hit rate is near zero and you can't see why, because the prompts look identical.

The first three are the same waste: you are paying for prefill on tokens the server has already processed. The last two are the two ways people get prefix caching wrong — measuring it dishonestly, and expecting it to fire when it structurally can't.

The Idea

A bookmark that is only valid if the entire book up to that point is identical.

Imagine answering questions about a long report. The first reader asks about page 300, so you read pages 1–300 and take notes. The second reader asks about the same report — you keep your notes and skip straight to page 300. That's the win, and it's large.

But now the third reader hands you the same report with one word changed on page 2. Your notes are worthless from page 2 onward, because everything you understood downstream was built on the text before it. You re-read almost the whole thing.

That's prefix caching exactly:

Reuse is prefix-anchored. You can skip work at position N only if every token from 0 to N matches. Not "mostly matches" — matches.
A change early poisons everything after it. One differing token near the start invalidates the entire remainder, no matter how much comes after and how identical it is.
The saving is compute, not memory. You're not storing less; you're skipping prefill you'd otherwise redo.

That third point separates this page from PagedAttention. Copy-on-write answers "can two sequences share this memory?" Prefix caching answers "has anyone computed this before, and can I skip it?"

Under the Hood

Hashing a block by everything before it

vLLM takes a hash-based approach. Each KV block gets a hash computed from a tuple:

Component	Why it's in the hash
Parent hash — the hash of the preceding block	This is what makes it a prefix chain: a block's identity includes its entire history
Block tokens — the exact token IDs in this block	Reduces collision risk
Extra keys — LoRA ID, multi-modal input hashes, cache salt	Keeps blocks unique across adapters, images and tenants

The parent hash is the whole trick. Block 3's identity depends on block 2's hash, which depends on block 1's, and so on back to the start:

                 Block 1                  Block 2                  Block 3
      [A gentle breeze stirred] [the leaves as children] [laughed in the distance]
Block 1: |<--- block tokens -->|
Block 2: |<------ prefix ----->| |<--- block tokens --->|
Block 3: |<--------------- prefix ------------------->| |<--- block tokens --->|

Change one token in block 1 and blocks 2 and 3 get different hashes even if their own tokens are identical. That's the "one word on page 2" property, implemented.

Two rules that decide your hit rate

Only full blocks are cached. A partially-filled block has no stable identity yet, so it isn't cached until it fills.

Therefore cache hits are block-granular, not token-granular. This is the single most practical consequence in the page, and it's easy to miss. From vLLM's own worked example with block_size = 4: a request shares its first 10 tokens with a cached request, and only 8 tokens hit the cache — the first two blocks. The third block matches on 2 of its 4 tokens, which is not a match at all.

You lose the remainder of any shared prefix that doesn't land on a block boundary. With the default block_size = 16, a shared prefix of 100 tokens gives you 96 tokens of hit (6 blocks) and re-computes the last 4.

For long shared prefixes that rounding is noise. For short ones it can be most of the benefit.

The free queue, and why eviction order is backwards

Blocks live in a pre-allocated pool with a doubly-linked free queue, and eviction is LRU — the head of the queue is the least recently used block and is evicted first.

The detail worth knowing: when a request finishes, its blocks are pushed to the tail of the free queue in reverse order. The reasoning is elegant — a request's last block hashes the most tokens, so it is the most specific and therefore the least likely to be reused by anyone else. Reversing the order means the most specific blocks are evicted first and the most generic (the shared opening blocks) survive longest.

Cached blocks aren't immediately gone when freed, either. They stay in the queue, still cached, until something needs to evict them — so a hit can land on a block whose original request finished long ago. And when a new request does hit cached blocks, those blocks are "touched": their reference count rises and they're pulled out of the free queue so they can't be evicted while in use.

What it costs

Nearly nothing, which is why it's described as almost free:

Hashing each block, once, when it fills.
A lookup per block at admission.
No extra memory — cached blocks live in the same pool. They're simply not returned to circulation while they might still be useful.

The honest caveats are two. First, if your workload has no shared prefixes, you pay the hashing and lookup for zero benefit — small, but not zero. Second, and more subtly, retaining cached blocks means fewer free blocks for live requests, which under memory pressure can trade against concurrency. That's a real interaction with the scheduler, not a theoretical one.

Is it on by default?

Here the documentation is genuinely ambiguous, so this page won't pretend otherwise. vLLM's feature guide says to set enable_prefix_caching=True to enable it, which reads like opt-in. But V1's stated design goal is to "require zero configs by enabling features/optimizations by default," and prefix caching is listed as fully functional in V1.

Rather than trust either reading, measure it — it takes ten seconds, and the first experiment below does exactly that. Knowing how to determine this for your own build is more durable than any default I could quote.

⚠️ Confirm against vllm serve --help and your startup log for the version you pin.

Try It

Hardware: Colab T4 or any CUDA GPU.

Experiment 1 — find out whether it's on, and what it's worth

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096

# prefix_hit.py — same long prefix, different questions. Does TTFT collapse?
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL  = "Qwen/Qwen2.5-0.5B-Instruct"

# A long, stable prefix — the shape of a RAG document or a system prompt.
DOC = "The system allocates memory in fixed-size blocks. " * 120   # ~1000 tokens

def ttft(prompt):
    t0 = time.perf_counter()
    for chunk in client.completions.create(model=MODEL, prompt=prompt, max_tokens=16,
                                           temperature=0.0, stream=True):
        if chunk.choices[0].text:
            return (time.perf_counter() - t0) * 1000
    return float("nan")

print(f"{'request':<34} {'TTFT (ms)':>10}")
print(f"{'1st: cold, nothing cached':<34} {ttft(DOC + ' Question 1: summarise.'):>10.0f}")
print(f"{'2nd: same prefix, new question':<34} {ttft(DOC + ' Question 2: list risks.'):>10.0f}")
print(f"{'3rd: same prefix, new question':<34} {ttft(DOC + ' Question 3: who signed?'):>10.0f}")
print(f"{'4th: DIFFERENT prefix':<34} {ttft('Unrelated. ' + DOC + ' Question 4?'):>10.0f}")

How to read it:

Result	Conclusion
Requests 2 and 3 are dramatically faster than 1	Prefix caching is on, and it's working
All four are about the same	Either it's off, or the prefix is too short to matter
Request 4 is as slow as request 1	Correct behaviour — prepending text changed the prefix, so nothing downstream matches

Request 4 is the important control. It uses the same document, just with eleven characters in front — and gets no benefit. That's the "one word on page 2" property, on your own server.

If requests 2 and 3 aren't faster, restart with --enable-prefix-caching and re-run. Whichever way that goes, you now know your build's default rather than guessing.

Now change one thing — the block-granularity effect

# block_boundary.py — does a shared prefix have to land on a block boundary?
PREFIXES = {
    "exactly 16 tokens shared":  "word " * 16,
    "exactly 96 tokens shared":  "word " * 96,
    "97 tokens shared":          "word " * 97,
}
for label, pre in PREFIXES.items():
    ttft(pre + " first")          # warm the cache
    t = ttft(pre + " second")     # measure the hit
    print(f"{label:<28} {t:>8.1f} ms")

What you should observe: hits round down to whole blocks. With block_size = 16, a 97-token shared prefix delivers a 96-token hit and recomputes the last token's block. The effect is proportionally large for short prefixes and negligible for long ones — which is exactly the rule of thumb you want when deciding whether restructuring a prompt is worth it.

A third variation worth running if you have time: put a timestamp at the start of your prompt versus at the end. At the start, your hit rate goes to zero. At the end, everything before it still hits. It's the single highest-leverage prompt-layout decision on this page and it costs nothing to verify.

Dial It In

Knob	What it does	Guidance
`--enable-prefix-caching`	Turns APC on	On for shared-prefix workloads. Verify your build's default rather than assuming
`--no-enable-prefix-caching`	Turns it off	Useful to A/B the benefit honestly, and to get clean benchmark numbers
`--prefix-caching-hash-algo`	Hash function for block identity	`sha256` is the default. See below
`--block-size`	Granularity of cache hits as well as allocation	Leave at 16. Smaller would improve hit granularity but costs kernel efficiency (PagedAttention)
`cache_salt` (per request)	Isolates cache reuse to requests sharing the salt	Multi-tenant deployments — see In Production

Hash algorithm options, which matter more than they look:

Value	Trade-off
`sha256`	Default. Collision-resistant. Uses `pickle`, so hashes may not be reproducible across Python or vLLM versions
`sha256_cbor`	Reproducible and cross-language compatible. Recommended when you need deterministic caching across environments
`xxhash`	Faster, non-cryptographic. Requires the `xxhash` package
`xxhash_cbor`	Reproducible variant of the above

⚠️ vLLM's own documentation warns explicitly about the non-cryptographic options: a non-cryptographically-secure hash theoretically raises collision risk, which "can cause undefined behavior or even leak private information in multi-tenant environments." Collisions remain very unlikely — but weigh that against the performance gain deliberately rather than reaching for the faster option. Note also that sha256 became the default only as of v0.11; earlier versions did not guarantee collision-free hashing.

Where It Bites You

Putting anything variable at the start of your prompt. A timestamp, a request ID, a user name, a randomised greeting — placed before your system prompt, any of these drives your hit rate to zero, because the parent-hash chain invalidates everything downstream. Variable content goes last. This is the highest-value single thing on the page.

Expecting token-granular hits. Hits round down to whole blocks, so a shared prefix that doesn't land on a 16-token boundary loses the remainder. Rarely material for long prefixes; occasionally material for short ones.

Benchmarking with repeated prompts. Send the same prompt a thousand times with caching on and you've measured your cache, not your server. This is the trap flagged back in Why Inference Servers Exist — and now you know the mechanism. Vary prompts, or disable caching for the baseline run.

Assuming it helps decode. It only skips prefill. If your workload is short prompts and long answers, most of your time is decode and prefix caching has almost nothing to work with. vLLM's docs are explicit: no gain when most time is spent generating, or when prompts don't share prefixes.

Multi-tenant cache sharing without thinking about it. Cached blocks are shared across requests by content. In a shared deployment that creates a timing side channel: an adversary can infer whether particular content is cached by observing latency. That's what cache_salt exists for — and if you serve multiple tenants and haven't considered it, consider it now.

Believing the hit rate you see in staging. Hit rate is a property of your traffic, not your config. A staging environment replaying a fixed script will show a hit rate production never approaches.

Forgetting cached blocks occupy the pool. They're retained rather than freed, which is the point — but under memory pressure that's fewer blocks available for live requests. If you enable caching and see preemption rise, that interaction is the reason.

Being surprised by duplicate blocks. V1's block table is append-only, so if two requests generate identical content concurrently you can briefly hold two blocks with the same hash. It resolves when the requests free. It's a known consequence of the V1 design, not a leak.

In Production

Cache hit rate is a first-class metric, and it's about traffic. Watch it and you learn something config can't tell you: whether your prompt structure is actually shareable. A sudden drop usually means a client changed a template — often by adding something variable near the front.

Use cache_salt for tenant isolation. Include a per-tenant (or per-trust-group) salt in requests and cache reuse is confined to that group:

{
  "messages": [ ... ],
  "cache_salt": "tenant-42"
}

The salt is injected into the first block's hash, so only requests sharing it can reuse each other's blocks. It preserves the benefit within a trust boundary while closing the cross-tenant timing channel — and vLLM's docs note it does so without a performance penalty.

Design prompts for cacheability, deliberately. Order matters:

[ stable system prompt ] [ stable tool definitions ] [ retrieved documents ] [ user turn ] [ timestamp ]
└─────────────── most cacheable ────────────────────┘                        └─ least ─┘

This costs nothing at design time and is expensive to retrofit, because it means changing every prompt template you've shipped.

Use sha256_cbor when caching must be deterministic across environments. The default sha256 uses pickle, so hashes aren't guaranteed reproducible across Python or vLLM versions. That's irrelevant for a single server and relevant the moment you care about consistent behaviour across a fleet or a rolling upgrade.

What changes at 10× traffic. Hit rate generally improves, because more concurrent requests share more prefixes — one of the few things in this article that gets better under load. The counterweight is pool pressure: more retained cached blocks against more live requests. Watch hit rate and preemption together; if preemption rises as hit rate rises, the cache is winning the allocation fight against live traffic.

Check Yourself

Recall the idea

What does prefix caching save, and how is that different from copy-on-write?

It saves computation — prefill on tokens already processed — and it works across requests separated in time. Copy-on-write saves memory by letting sequences within one request share prompt blocks by reference. Same block-sharing machinery, different resource and different scope.

Why does a block's hash include its parent's hash?

So that a block's identity encodes its entire preceding context. Two blocks with identical tokens but different histories must not be interchangeable, because their KV state differs. It's also what makes reuse strictly prefix-anchored: change anything early and every downstream hash changes.

Why are cache hits block-granular?

Only full blocks are cached, so matching is per block. With block_size = 16, a 100-token shared prefix yields a 96-token hit and recomputes the remainder — a partial block match counts for nothing.

When does prefix caching do nothing?

When prompts don't share prefixes, or when the workload is dominated by decode rather than prefill. Short prompts with long answers gain almost nothing; independent one-shot prompts gain nothing at all.

Explain the mechanics

Why does a timestamp at the start of a prompt destroy your hit rate?

The first block's hash includes those tokens, every subsequent block's hash includes the first block's hash, so a unique value at position zero makes every block in every request unique. Nothing can ever match. Move it to the end and everything before it still hits.

Why are a finished request's blocks pushed to the free queue in reverse order?

Because the last block hashes the most tokens and is therefore the most specific — least likely to be reused by anyone else. Reversing the order puts the most specific blocks nearest the head, so LRU evicts them first and the generic opening blocks survive longest. It's an eviction policy that matches how prefixes are actually shared.

Two identical concurrent requests can produce duplicate cached blocks. Why?

V1's block table is append-only, so a request that has already been allocated block 3 can't be retroactively repointed at an equivalent cached block 1. Both exist until the requests free. It's a deliberate consequence of the append-only design, and it resolves on completion.

What does cache_salt do mechanically, and what attack does it stop?

It's mixed into the hash of the first block, so blocks are only reusable between requests carrying the same salt. It closes a timing side channel: without it, an attacker can infer whether specific content is already cached by measuring how fast a request completes — which leaks information about other tenants' prompts.

Reason about a trade-off

Your RAG service has 4,000-token documents and 50-token questions. Estimate the benefit.

Very large, and it's the ideal case. Roughly 98% of each prompt is a stable prefix, so a repeat query against the same document skips prefill on ~3,990 tokens (rounded down to the block boundary). Since this workload is prefill-dominated, that's most of its TTFT and most of its GPU time. The condition is that documents genuinely repeat across queries — verify with the shared-prefix ratio from Stage 0 rather than assuming.

A colleague wants xxhash for speed. Respond.

Ask what fraction of time is actually going into hashing — it's per block, once, and almost certainly not your bottleneck, so the gain is likely unmeasurable. Then raise the cost vLLM documents explicitly: a non-cryptographic hash raises collision risk, and in a multi-tenant deployment a collision can cause undefined behaviour or leak information across tenants. Small probability, severe consequence, negligible upside. If they still want it, that's a security-tolerance decision that should be made explicitly rather than as a performance tweak.

Should you enable prefix caching on a workload with no shared prefixes?

There's little reason to. You pay hashing and lookup for zero hits, and retained cached blocks compete with live requests for the pool. The costs are small, so it's not harmful — but "on by default everywhere" isn't automatically right, and if you're memory-constrained and seeing preemption, this is worth A/B testing with --no-enable-prefix-caching.

You enable caching and preemption increases. Explain and resolve.

Cached blocks stay in the pool rather than returning to circulation, so under memory pressure there are fewer free blocks for live requests — the cache is competing with running traffic for the same resource. Confirm by A/B testing with caching off. If it's real, the resolution is more KV cache (raise gpu_memory_utilization, or reduce max_model_len to your measured p99) rather than disabling caching, since the caching win is usually larger. But measure both, because on a low-hit-rate workload you're paying the cost for nothing.

Cheat Sheet

The mechanism

block_hash = hash( parent_block_hash, block_token_ids, extra_keys )
                          │                                 └─ LoRA ID, image hashes, cache_salt
                          └─ makes reuse strictly PREFIX-anchored

The rules

Rule	Consequence
Only full blocks are cached	Hits round down to `block_size` (16)
Hash chains through the parent	One early token change invalidates everything after it
Eviction is LRU	Freed blocks queue in reverse — most specific evicted first
Saves prefill, not decode	No gain on short-prompt / long-answer workloads
Cached blocks stay in the pool	Can compete with live requests under memory pressure

Prompt layout for cacheability

[ system prompt ] [ tool defs ] [ retrieved docs ] [ user turn ] [ anything variable ]
└──────────── stable, cacheable ─────────────────┘               └─ put it LAST ─────┘

Flags

--enable-prefix-caching                  # verify your build's default rather than assuming
--no-enable-prefix-caching               # for honest A/B benchmarking
--prefix-caching-hash-algo sha256        # default; sha256_cbor for cross-environment determinism
                                         # xxhash is faster but non-cryptographic — read the warning

{ "messages": [...], "cache_salt": "tenant-42" }   // multi-tenant isolation

The three things to remember

Variable content goes at the end of the prompt. Anything unique at the front zeroes your hit rate.
Hits are block-granular, so shared prefixes round down to multiples of 16 tokens.
Hit rate is a property of your traffic, not your configuration — measure it in production, not staging.

Sources

vLLM, Automatic Prefix Caching (design) — hashing scheme, block pool and free queue, LRU eviction, cache_salt, hash algorithm options — github.com/vllm-project/vllm/blob/main/docs/design/prefix_caching
vLLM, Automatic Prefix Caching (feature guide) — enabling APC, example workloads, stated limits — github.com/vllm-project/vllm/blob/main/docs/features/automatic_prefix_caching

← Previous: Lifecycle of a Request · Next: Quantisation →

⚠️ Verification checklist (delete before publishing)

Verified against vLLM's design and feature docs this session

Hash components: parent hash, block token IDs, extra keys (LoRA ID, multi-modal hashes, cache salt). Quoted structure and the three-block worked example.
"We only cache full blocks" — stated in the design doc.
Block-granular hits: the doc's own example has 10 shared tokens producing an 8-token hit at block_size = 4, because the third block matches only 2 of 4 tokens.
LRU eviction via a doubly-linked free queue; freed blocks are added in reverse order because the last block hashes the most tokens and is least likely to be reused.
"Touching" cached blocks on a hit — reference count increases and they leave the free queue.
Append-only block tables cause duplicate blocks in V1, resolved when the request frees.
cache_salt exists, is injected into the first block's hash, and its stated purpose is preventing timing-based inference of cached content in multi-tenant environments.
--prefix-caching-hash-algo options and their trade-offs, including the explicit warning about non-cryptographic hashes and the note that sha256 became the default as of v0.11.
The claim that APC "won't change model outputs" and gives no gain when generation dominates or prefixes aren't shared — both from vLLM's docs.

Still open — deliberately

Is prefix caching on by default? Genuinely unresolved: the feature doc says to set enable_prefix_caching=True, while V1's design goal is "zero configs by enabling features/optimizations by default". The page does not assert either way — it teaches the reader to measure it in ten seconds. Settle it before publishing if a definitive source turns up, but the page is safe as written. (This closes the long-standing Stage 0 / Stage 1 contradiction: Stage 0 has been made conditional, and no page now asserts a default.)
Confirm --no-enable-prefix-caching exists with that exact spelling.
Confirm the claim that retained cached blocks can increase preemption under memory pressure. It follows from the pool being shared, but is reasoned rather than sourced, and it appears in both Where It Bites You and a Check Yourself answer.
Confirm the metric name for prefix cache hit rate — In Production tells readers to watch it.

Code

Run prefix_hit.py and capture the four TTFT values. This experiment doubles as the default-detection method, so it must work reliably — check that ~1,000 tokens is long enough to make the difference unmistakable on a T4.
Run block_boundary.py and confirm the 96-vs-97-token rounding is actually observable. If the effect is below measurement noise at these sizes, either scale the prefixes up or reframe the claim as arithmetic rather than something the reader can see.
block_boundary.py calls ttft() from the previous snippet — note the dependency explicitly or merge them into one script.
Run the timestamp-at-front versus timestamp-at-end variation and capture numbers; it's the page's headline advice and is currently unmeasured.

Rendering

One image generated and placed; row added to image-prompts.md.
The ASCII prefix-chain block renders correctly on the published site.
All relative links resolve once target files exist.