Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

Stage 2 — The Engine

3 min read

You can trace one request from HTTP POST to streamed token and name the component responsible at every step. · ~1 week

Stage 1 gave you the ideas. This stage gives you the machine that implements them, and the vocabulary you need to read its logs.

The distinction matters because Stage 1's mental models are deliberately simplified. Continuous batching was described as "re-decide the batch every step" — true, but it doesn't tell you how the scheduler decides, which is the thing that determines whether your long prompts destroy everyone else's latency. PagedAttention was described as an allocator — true, but it doesn't tell you what happens when the pool runs dry, which is the failure you'll actually be paged for.

Everything here is the real behaviour, verified against vLLM's own documentation rather than inferred from the papers. Where the engine's V1 architecture differs from the original design — and it differs in ways that matter, including which preemption mode is the default — the pages say so.

Pages

# Topic What it covers Status
1 The Scheduler & Block Manager Waiting and running queues; the decode-first policy; the token budget per step; preemption by recompute; what a preemption warning in your logs means 🚧
2 Lifecycle of a Request HTTP → tokenise → admit → prefill → N decode steps → detokenise → stream; where queuing time actually accumulates 🚧
3 Prefix Caching Hash-based block reuse; the workloads it transforms (shared system prompts, RAG, agents, multi-turn chat) and the ones where it's pure overhead 🚧
4 Quantisation AWQ, GPTQ, FP8 and INT8 explained without the linear algebra; what each needs from your hardware; the accuracy cost nobody benchmarks 🚧
5 Speculative Decoding Draft-then-verify; n-gram and draft-model variants; acceptance rate as the number that decides everything; when it makes your server slower 🚧

What you'll be able to do by the end

  • Read a vLLM startup log and a steady-state log line and say what the engine is doing.
  • Explain a preemption warning to a colleague, and name the four things that reduce it.
  • Predict whether prefix caching will help your workload before enabling it — and know when it's pure overhead.
  • Choose a quantisation scheme your hardware actually supports, and state what it costs in accuracy.
  • Decide whether speculative decoding is worth it from one number: the acceptance rate.

A correction to Stage 1

Continuous Batching presented the step loop as schedule → execute → retire → admit → preempt. That's the right shape for building intuition and it understates one thing: the scheduler is not neutral between prefill and decode. In the V1 engine it prioritises decode — every pending decode is batched first, and prefill work only gets whatever token budget is left over.

That single policy explains most of what you'll see in production, and it's where The Scheduler & Block Manager starts.

← Back to Core Concepts · Next: The Scheduler & Block Manager →