Back to Articles/vLLM & LLM Inference Serving

01 · The Scheduler & Block Manager

25 min read

Two components, one decision, made about a hundred times a second: which requests run this step, and is there memory for them?

The scheduler answers the first question and the block manager answers the second, and they are joined at the hip — a scheduling decision that the block manager can't fund isn't a decision, it's a preemption. Everything you tune in Stage 4 is a constraint on this pair.

This page is the real behaviour, not the simplified model from Stage 1. In particular, the scheduler has a policy, and knowing what it is explains most of what you'll see in production.

The Problem

Your server is running. Then, in roughly this order of how alarming they look:

A warning you don't recognise, repeating:

WARNING ... Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not
enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization
or tensor_parallel_size to provide more KV cache memory. total_cumulative_preemption_cnt=1

Nothing has failed. No request errored. But that counter is climbing.

Throughput falls as load rises. Not plateaus — falls. More traffic, fewer tokens per second, which violates every intuition you have about a queue.
Requests are waiting while max_num_seqs is nowhere near reached. You raise it. Nothing improves.
A user's request is admitted, starts streaming, then stalls for a second and resumes. It wasn't queued — it was running, and then it wasn't.

All four are the same event. The block manager ran out of blocks, and the scheduler responded by taking memory back from a request that already had it. That's preemption, it's the designed behaviour rather than a bug, and understanding it is the difference between "add a replica" and "raise a flag that will not help."

The Idea

An air traffic controller with a fixed number of runways.

Aircraft arrive continuously. Some are inbound and need to land (they've been circling — they're the ones already in your system, waiting on their next instruction). Some are on the ground waiting for clearance to take off. The controller reassesses every few seconds, and works to one rule that isn't obvious until you see the consequence:

Aircraft already in the air get priority. Always. A plane circling with low fuel is a worse outcome than a plane waiting at the gate — the one on the ground is merely inconvenienced, the one in the air is degrading. So the controller clears every inbound aircraft first, and only then uses the remaining capacity to release departures.

That is exactly vLLM's policy:

Air traffic	vLLM
Aircraft already airborne	Decode — requests mid-generation, streaming to a user right now
Aircraft waiting to depart	Prefill — new requests whose prompt hasn't been processed
The rule: airborne first	Decode-first scheduling. All pending decodes are batched before any prefill
Remaining runway capacity	The leftover token budget (`max_num_batched_tokens`)
A departure split across slots	Chunked prefill — a long prompt processed a piece at a time
Ordering a plane back into a holding pattern	Preemption — a running request loses its memory and must redo work

The rule follows directly from what users perceive. A request mid-stream that stalls is a visible stutter — the text stops moving. A request not yet started is just a slightly slower start. Given a choice, degrade the second.

Under the Hood

The two queues and one budget

Every request is in one of two states, and the scheduler moves it between them:

     new request
         │
         ▼
   ┌───────────┐   admitted when there is token budget AND free blocks
   │  WAITING  │ ─────────────────────────────────────────────────────┐
   └───────────┘                                                      │
         ▲                                                            ▼
         │                                                      ┌───────────┐
         │  preempted: blocks reclaimed,                        │  RUNNING  │
         └──────────  goes back to the front of WAITING ────────│           │
                                                                └───────────┘
                                                                      │
                                                     EOS / max_tokens / stop
                                                                      ▼
                                                                  FINISHED
                                                            (blocks returned to pool)

Each step the scheduler spends a token budget — max_num_batched_tokens — across the work it admits. And here is the policy that Stage 1 glossed over:

Decode first, always. The scheduler batches every pending decode request. Only then, with whatever budget remains, does it schedule prefill. If a pending prefill won't fit in the remaining budget, it is chunked — a piece runs now, the rest later.

Two consequences worth internalising:

A decode step costs one token per sequence. 200 running requests consume 200 of your budget. Cheap, and always served.
A prefill costs its whole prompt length. A 4,000-token prompt wants 4,000 budget in one go, and will be chopped up rather than allowed to displace decodes.

This is why chunked prefill and decode-first are the same design decision, and why they arrived together. It's also, precisely, why your ITL stays smooth when someone pastes a novel.

The block manager, and the moment it says no

The block manager owns the pool of KV blocks from PagedAttention. Its job each step is small and unforgiving:

Can each running sequence get the block it needs? A sequence crossing a 16-token boundary needs one new block this step. Most steps, most sequences need nothing.
Can a waiting request be admitted? That needs enough blocks for its whole prompt.
If neither is true — who loses?

Step 3 is preemption. Note what it means: the failure isn't at admission time, where you'd hope to catch it. A request that was safely admitted can later be evicted, because its memory demand grows one block at a time as it generates, and nobody reserved the future.

That's the price of not over-reserving. PagedAttention eliminated reservation waste by allocating on demand; the flip side is that demand can outrun supply mid-flight.

Preemption: what actually happens

The evicted request's blocks are freed and returned to the pool. The request goes back to the waiting queue, at the front. Later, when memory is available, it resumes.

How it resumes is the interesting part, and the V1 answer is not what older material says.

Mode	What it does	Status
`RECOMPUTE`	Throws the KV cache away. On resume, re-runs prefill over the prompt and everything generated so far	The V1 behaviour
`SWAP`	Copied blocks out to CPU RAM and back on resume	Removed in V1

This is worth being precise about, because a great deal of still-circulating material describes swapping as vLLM's preemption mechanism. GPU↔CPU KV cache swapping was removed in V1 — the simplified architecture no longer needs it, and recomputation has lower overhead than a PCIe round trip for gigabytes. Prefill is fast and parallel; copying memory is not.

So there is no recompute-versus-swap decision to make. There is one behaviour, and its cost is GPU time.

The operational consequence is the important bit:

A preempted request does its work twice. That's why throughput falls rather than plateaus when you exceed capacity — the machine is now spending real GPU time on output it already produced and discarded. Push harder and more requests get preempted, so more work is discarded, so throughput drops further. It is a genuine negative feedback loop, and it's the mechanism behind the capacity cliff described in The KV Cache.

Where the engine actually runs

vLLM V1 splits the work across processes, and the shape matters for anyone deploying it:

  1 × API server process     HTTP, tokenisation, input processing
  1 × engine core process    THE SCHEDULER lives here — runs a busy loop
  N × GPU worker processes   one per GPU, executes forward passes
  ─────────────────────────
  2 + N processes minimum, all competing for CPU

The scheduler runs in the engine core process as a busy loop, which makes it unusually sensitive to CPU starvation. Under-provision your cores and the scheduler is slow to dispatch work — so your GPU sits idle waiting for a CPU-bound scheduler, and every metric you'd normally check looks fine.

The documented minimum is 2 + N physical cores, and if hyperthreading is on, that's 2 × (2 + N) vCPUs. This is one of the most common causes of "GPU utilisation is lower than expected" in virtualised environments, and it has nothing to do with any flag on this page.

Try It

Hardware: Colab T4 or any CUDA GPU. The point of this experiment is to cause a preemption on purpose, because a failure mode you've induced deliberately is one you'll recognise at 3am.

Step 1 — start a deliberately memory-starved server

# Small KV cache on purpose: we want to hit the wall quickly.
vllm serve Qwen/Qwen2.5-0.5B-Instruct \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.35 \
  --max-num-seqs 256 \
  2>&1 | tee scheduler.log

Note the reported GPU block count at startup — that's your entire budget.

Step 2 — overwhelm it with long generations

# force_preemption.py — many concurrent requests, each generating a lot.
import concurrent.futures as cf, time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL, N = "Qwen/Qwen2.5-0.5B-Instruct", 64

def run(i):
    t0 = time.perf_counter()
    r = client.completions.create(
        model=MODEL,
        prompt=f"Write an extremely detailed essay about the number {i}.",
        max_tokens=1500, temperature=0.8,        # long outputs = growing KV demand
    )
    return time.perf_counter() - t0

t0 = time.perf_counter()
with cf.ThreadPoolExecutor(max_workers=N) as ex:
    times = list(ex.map(run, range(N)))
total = time.perf_counter() - t0
print(f"{N} requests in {total:.1f}s   "
      f"aggregate {N*1500/total:,.0f} tok/s   slowest request {max(times):.1f}s")

Then look at what the server said:

grep -ci "preempt" scheduler.log
grep -i "preempt" scheduler.log | tail -3

What you should observe: preemption warnings, with total_cumulative_preemption_cnt climbing. You have reproduced the production failure on purpose.

Now change one thing

Re-run with a larger cache and nothing else changed:

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096 \
  --gpu-memory-utilization 0.85 --max-num-seqs 256

Observation	What it proves
Preemption count falls or hits zero	Preemption is a memory symptom, not a concurrency one
Aggregate throughput rises, despite identical load	The preempted work was being redone. You removed waste, not added capacity
Slowest-request time improves disproportionately	Preempted requests are the tail. They pay for the whole generation twice

Then try the flag that doesn't work. Go back to --gpu-memory-utilization 0.35 and raise --max-num-seqs to 512. Preemption gets worse, because you've permitted more sequences to compete for the same blocks. That's the single most valuable negative result on this page: the concurrency cap cannot solve a memory problem, and raising it under memory pressure actively harms you.

Third variation, if you have time: keep the starved cache but set --max-num-seqs 16. Preemption should stop, throughput should be modest but stable, and the waiting queue should grow. That's the correct response to insufficient memory — queue politely rather than thrash.

Dial It In

The four levers vLLM's own documentation names for reducing preemption, plus the ones that shape scheduling.

Knob	Effect on preemption	Cost
`--gpu-memory-utilization` ↑	Directly increases the block pool. First thing to try	Risk of OOM if pushed too close to 1.0; less room for other processes
`--max-num-seqs` ↓	Fewer concurrent sequences competing for blocks	Lower peak throughput; longer waiting queue
`--max-num-batched-tokens` ↓	Less work admitted per step	Worse TTFT on long prompts
`--tensor-parallel-size` ↑	Shards weights across GPUs, leaving more room per GPU for KV cache	More GPUs; synchronisation overhead
`--pipeline-parallel-size` ↑	Distributes layers, indirectly freeing weight memory	More GPUs; latency penalty

And for shaping the latency/throughput balance:

Knob	Guidance
`--max-num-batched-tokens` 2048	Better ITL — fewer prefills slowing decodes
`--max-num-batched-tokens` > 8192	Better throughput, recommended for smaller models on large GPUs
Set it equal to `max_model_len`	Approximately the old V0 scheduling behaviour, except decode is still prioritised

⚠️ If you disable chunked prefill, max_num_batched_tokens must exceed max_model_len — set it lower and the server can crash at startup.

The order to try things, when you see preemption: raise gpu_memory_utilization first (free), then lower max_num_seqs (costs throughput but stabilises), then reconsider max_model_len (reclaims per-sequence worst case), and only then reach for more GPUs.

Where It Bites You

Treating preemption warnings as errors to be silenced. They're a capacity signal, and the signal is specific: you are out of KV blocks. Filtering them out of your logs removes your only warning that throughput is about to invert.

Raising max_num_seqs to fix queueing when the real constraint is memory. You proved this in the experiment. The cap permits concurrency; it doesn't create memory for it. Under memory pressure it makes things worse by admitting more competitors for the same blocks. Check whether running is at the cap before touching it — if running is below the cap and requests are waiting, memory is your binding constraint.

Assuming admission means safety. A request is admitted on the memory it needs now, and its demand grows a block at a time as it generates. Nothing reserved its future. Long generations are therefore the most likely to be preempted, and they're also the ones that lose the most work when they are.

Forgetting that recompute redoes generated tokens too. On resume, a preempted request re-prefills its prompt and everything it had already generated. A request 1,200 tokens into a 1,500-token answer loses far more than a request that just started. This is why preemption hurts your tail latency specifically.

Under-provisioning CPU. The scheduler is a busy loop in the engine core process, and there are 2 + N processes minimum. Starve them and the GPU idles waiting for scheduling decisions, while every GPU-side metric looks healthy. In virtualised environments with hyperthreading, remember the minimum is 2 × (2 + N) vCPUs.

Expecting FCFS to be fair. The waiting queue is first-come-first-served by default. A burst of long requests admitted first will hold blocks for thousands of steps while short requests queue behind them. Decode-first prioritisation protects running requests from prefill; it does nothing for requests that haven't started.

Reading old documentation about SWAP. Plenty of material describes swapping to CPU RAM as vLLM's preemption behaviour. It was removed in V1 — not merely made non-default. If you're tuning based on assumptions about PCIe traffic or sizing host RAM for swap, you're tuning something that no longer exists.

In Production

Alert on preemption count, not on latency. It is the earliest honest signal you have. Latency degrades after preemption starts, by which point you're already throwing away work. vLLM exposes the cumulative preemption count via Prometheus, and a rising rate should page someone before p99 does. Observability.

Read running and waiting together — the diagnosis is a two-by-two.

Running	Waiting	Preemptions	Diagnosis
At `max_num_seqs`	> 0	0	Concurrency-capped. Raise the cap if memory allows
Below cap	> 0	0	Memory-capped at admission. More cache, or fewer/shorter requests
Below cap	> 0	rising	Memory-capped mid-flight. You are past capacity — shed load or scale out
Below cap	0	0	Healthy, with headroom

Row three is the dangerous one, because it's the only state where adding traffic makes total throughput go down.

Capacity planning has a hard edge now. Your safe operating point is where preemption stays at zero under peak load — not where the GPU is busiest. A server running at 95% KV utilisation with occasional preemption is past its efficient point, not at it, because some fraction of its work is being discarded and redone.

Admission control belongs upstream. The scheduler's only tools when overloaded are queue and preempt, and preempt is expensive. A gateway that rejects or queues before requests reach the engine gives you a cheaper, more controllable failure — and lets you make per-tenant decisions the engine can't. Security Posture.

What changes at 10× traffic. The scheduler behaves correctly throughout — that's the point of the design. What changes is that you cross from "queueing" into "preempting", and those are qualitatively different regimes: queueing degrades latency linearly and predictably, preemption degrades throughput non-linearly. Autoscaling must trigger in the first regime, because by the second you're losing ground. That's the argument Autoscaling is built on.

Check Yourself

Recall the idea

What are the scheduler's two queues, and what moves a request between them?

Waiting and running. A request moves waiting → running when there's token budget and enough free KV blocks for its prompt. It moves running → finished on EOS, max_tokens or a stop string. And it can move running → waiting via preemption, when the block manager can't fund its next block.

State vLLM's V1 scheduling policy in one sentence.

Decode first: every pending decode request is batched before any prefill is scheduled, and prefill gets only the leftover max_num_batched_tokens budget — chunked if it doesn't fit.

Why does that policy exist?

Because a stalled decode is a visible stutter for a user mid-stream, while a delayed prefill is merely a slightly slower start. It also mixes compute-bound prefill with memory-bound decode in the same batch, which uses the GPU better.

What is the V1 default preemption mode, and why?

RECOMPUTE — the KV cache is discarded and regenerated on resume. It's the only mode: GPU↔CPU swapping was removed in V1, because recomputation has lower overhead than moving gigabytes across PCIe, and prefill is fast and parallel.

Explain the mechanics

Why can throughput fall as load rises?

Because preempted requests redo work. Past the memory limit, the engine spends GPU time regenerating tokens it already produced and discarded, so effective capacity drops — which causes more preemption, which discards more work. It's a negative feedback loop, not a plateau.

A request was admitted successfully. Why can it still be preempted later?

Admission checks the memory it needs at that moment; its KV demand then grows one block per 16 tokens generated, and nothing reserved that future. This is the direct trade-off of PagedAttention's on-demand allocation: no reservation waste, but demand can outrun supply mid-flight.

Requests are waiting but running is below max_num_seqs. What's the constraint, and what won't help?

KV cache memory — there aren't enough free blocks to admit anyone, regardless of the sequence cap. Raising max_num_seqs won't help and can hurt, since it permits more sequences to compete for the same blocks. The useful levers are more cache (gpu_memory_utilization, tensor parallelism) or less demand per request (max_model_len, shorter prompts).

Why does a decode step cost so little budget compared to a prefill?

A decode produces one token per sequence, so 200 running sequences cost 200 tokens of budget. A prefill must process its entire prompt, so one 4,000-token prompt costs 4,000. That asymmetry is why prefill is the thing that gets chunked and deferred, and decode is the thing that always runs.

Why is the engine sensitive to CPU under-provisioning?

The scheduler runs as a busy loop in the engine core process, alongside an API server process and one worker process per GPU — 2 + N processes minimum. If they contend for too few physical cores, scheduling decisions are dispatched slowly and the GPU idles, while GPU-side metrics look fine.

Reason about a trade-off

Preemption warnings under peak load. Walk through your response.

Confirm the regime first: check running against the cap, the waiting queue, and the preemption rate — row three of the diagnosis table means you're past capacity, not merely busy. Then apply the levers in cost order: raise gpu_memory_utilization (free, first choice); lower max_num_seqs to stabilise by queueing instead of thrashing; reduce max_model_len if your real p99 is well below it. Tensor parallelism and more replicas come last because they cost hardware. Throughout, resist raising max_num_seqs — the instinct is wrong here.

Why alert on preemption count rather than p99 latency?

Preemption is a leading indicator of a cliff; latency is a lagging indicator of one. Once preemption starts, throughput is actively falling, so the system is moving away from recovery on its own — and by the time p99 reflects it, users have felt it and the queue has grown. Preemption also names the cause, where a latency alert only says "something is slow."

A colleague suggests switching preemption to SWAP to avoid redoing work. Evaluate.

The intuition is reasonable — swapping preserves the KV cache instead of discarding it — but the option doesn't exist: GPU↔CPU KV cache swapping was removed in V1. Worth explaining why, because it's the same reasoning that makes recompute the right default anyway: moving gigabytes over PCIe and back is slower than re-running a parallel, compute-efficient prefill, and it consumes host memory and bandwidth you may need elsewhere. The productive redirect is that preemption is expensive either way, so the conversation should be about how to stop preempting at all — which is the Dial It In list.

When is a non-zero waiting queue fine, and when is it a problem?

Fine when preemptions are zero and latency meets your SLO: it means the server is at a healthy operating point, admitting what it can fund and queueing the rest, which is exactly the behaviour you want. A problem when it coincides with rising preemption (you're past capacity and losing work), or when queue time is pushing TTFT past your target — at which point the answer is more replicas, since no flag creates capacity you don't have.

Cheat Sheet

The policy

Decode first, always. All pending decodes are batched, then leftover max_num_batched_tokens goes to prefill, chunked if it doesn't fit.

The states

WAITING ──admit (budget + free blocks)──► RUNNING ──EOS/max_tokens/stop──► FINISHED
   ▲                                          │
   └──────── preempt (out of blocks) ─────────┘

Preemption

Fact	Consequence
V1 default mode is `RECOMPUTE`	Prompt and generated tokens are redone on resume
Preempted work is done twice	Throughput falls past capacity — it doesn't plateau
It's a memory signal	`max_num_seqs` won't fix it; raising it makes it worse
It hits long generations hardest	They lose the most work, so it's a tail-latency problem

The four documented fixes, in cost order

--gpu-memory-utilization 0.95     # ↑ more block pool. Try this first
--max-num-seqs 128                # ↓ fewer competitors for blocks
--max-num-batched-tokens 2048     # ↓ less admitted per step (also: better ITL)
--tensor-parallel-size 2          # ↑ shard weights, more room for cache per GPU
--pipeline-parallel-size 2        # ↑ distribute layers, indirectly frees memory

Tuning max_num_batched_tokens

Value	Optimises
~2048	ITL — fewer prefills interrupting decodes
> 8192	Throughput — recommended for small models on large GPUs
= `max_model_len`	Roughly V0 behaviour, but decode still prioritised

Diagnosis at a glance

Running	Waiting	Preemptions	Meaning
At cap	> 0	0	Concurrency-capped → raise cap if memory allows
Below cap	> 0	0	Memory-capped at admission
Below cap	> 0	rising	Past capacity — throughput is falling. Shed load or scale out
Below cap	0	0	Healthy

Process shape

1 API server + 1 engine core (the scheduler, a busy loop) + N GPU workers = 2 + N processes
minimum 2 + N physical cores   →   2 × (2 + N) vCPUs if hyperthreaded

Sources

vLLM, Optimization and Tuning — preemption modes, the V1 RECOMPUTE default, chunked prefill and decode-first scheduling, max_num_batched_tokens guidance, and CPU provisioning — github.com/vllm-project/vllm/blob/main/docs/configuration/optimization
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022 — usenix.org/conference/osdi22/presentation/yu
Agrawal et al., Sarathi — chunked prefill — arxiv.org/pdf/2308.16369

← Back to The Engine · Next: Lifecycle of a Request →

⚠️ Verification checklist (delete before publishing)

Verified against vLLM's own documentation this session — docs/configuration/optimization.md

V1 default preemption mode is RECOMPUTE, not SWAP. Quoted directly: "In vLLM V1, the default preemption mode is RECOMPUTE rather than SWAP, as recomputation has lower overhead in the V1 architecture." This resolves the open contradiction carried from Stage 1 — the PagedAttention and Continuous Batching pages both said "recompute or swap". Both have now been corrected, along with The KV Cache — swapping is described as removed, not as an alternative mode.
Chunked prefill is enabled by default in V1 — "In V1, chunked prefill is enabled by default whenever possible." This resolves the contradiction flagged across three Stage 1 pages. Action: confirm all three now say the same thing.
Decode-first scheduling policy — "the scheduling policy prioritizes decode requests. It batches all pending decode requests before scheduling any prefill operations." This corrects the simplified step loop given in Continuous Batching; the correction is noted in the Stage 2 README.
The preemption warning text is quoted from the docs verbatim.
The four preemption remedies and their trade-offs are the documented list.
max_num_batched_tokens guidance (2048 for ITL, >8192 for throughput, = max_model_len approximates V0) is documented.
The 2 + N process architecture and physical-core requirement is documented, including the hyperthreading doubling.
The warning that max_num_batched_tokens must exceed max_model_len when chunked prefill is disabled, or startup may crash.

Still to verify

--max-num-seqs: stopped asserting a number. It is platform- and version-dependent and is not stated in the tuning docs, so all three pages now tell the reader to read their own value from vllm serve --help or the startup log rather than quoting a figure that may be wrong for their build.
Confirm the exact Prometheus metric names for preemption count, running and waiting counts — the In Production section and the diagnosis table both depend on them, and Observability will need them too.
Confirm preempted requests return to the front of the waiting queue rather than the back. Asserted in the state diagram and not sourced.
--preemption-mode — since the SWAP mode itself was removed in V1, there is no mode to select. The flag row has been deleted from Continuous Batching's Dial It In rather than corrected. (Worth a final confirm that the flag doesn't linger as a no-op, but no page now depends on it.)
Confirm the scheduler runs specifically in the engine core process (implied by the docs' "1 engine core process — runs the scheduler and coordinates GPU workers", so likely fine).
Prefix caching default: the APC doc says "Set enable_prefix_caching=True to enable APC", which suggests not on by default — but this contradicts other sources claiming V1 enables it. The Stage 0/Stage 1 contradiction is therefore still open. Resolve it definitively on the Prefix Caching page and correct whichever pages are wrong.

Code

Run the three-way Try It experiment on a real T4 and capture actual preemption counts and throughput for each variation.
Confirm the key negative result: that raising max_num_seqs under memory pressure makes preemption worse. This is the most valuable claim in the experiment and is currently reasoned, not measured.
Confirm --gpu-memory-utilization 0.35 is low enough to force preemption with a 0.5B model on a T4, and adjust the value if not.
Confirm the preemption warning still appears on stderr in the current version and that grep -i preempt finds it.

Rendering

Two images generated and placed; rows added to image-prompts.md.
The ASCII state diagram and process-shape blocks render correctly on the published site.
All relative links resolve once target files exist.