Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

Stage 1 — Core Concepts

3 min read

You can look at a model card and a GPU spec and estimate whether it will fit, and how many concurrent requests you'll get. · ~1 week

Stage 0 ended with an arithmetic identity you derived on a laptop:

max concurrent seqs = (KV cache bytes available) ÷ (KV bytes per token) ÷ max_model_len

Three quantities, and Stage 0 handed you the first one — it's the memory left after the weights. This stage explains the other two, and then explains what the engine does with the capacity that results.

The pages are in dependency order and it is a strict one:

  1. The KV cache is the scarce resource. Understand what it is and you can compute KV bytes per token for any model from its config file.
  2. Prefill and decode are the two phases that consume it, and they behave nothing alike — which is why a single "tokens per second" number is almost always misleading.
  3. PagedAttention is how vLLM allocates that resource without wasting most of it.
  4. Continuous batching is what efficient allocation makes possible: keeping the batch full.
  5. Sampling parameters are the per-request settings that change how much of all of the above a single request consumes.

The last one is here, rather than filed with the API documentation where most guides put it, for a specific reason: max_tokens and n are not formatting preferences. They are memory reservations, and you can't reason about a batch until you know that.

Pages

# Topic What it covers Status
1 The KV Cache Why generation re-reads the whole prompt without it; the memory formula and how to compute it for a real model; why it, not the weights, is the constraint 🚧
2 Prefill vs Decode The two phases and why they behave nothing alike; compute-bound versus memory-bandwidth-bound; TTFT, ITL and TPOT defined 🚧
3 PagedAttention The OS virtual-memory analogy, then the real thing: blocks, block tables, internal versus external fragmentation, copy-on-write for shared prefixes 🚧
4 Continuous Batching Static batching and its convoy problem; iteration-level scheduling; the chef who never lets the pass go empty; the workloads where it changes nothing 🚧
5 Sampling Parameters temperature, top_p, top_k, max_tokens, stop, seed, n — what each does to the distribution, and what each costs the scheduler 🚧

What you'll be able to do by the end

  • Compute KV cache cost per token for any model from its config.json, and explain why two models with identical parameter counts can differ several-fold.
  • Predict roughly how many concurrent users a given GPU will support, before renting one.
  • Explain why your server's throughput and its per-user speed are different numbers that move in opposite directions.
  • Read a SamplingParams object and say what it costs the scheduler, not just what it does to the text.

Nothing here requires a GPU except the optional measurement in The KV Cache, which confirms the formula empirically and is worth running on a Colab T4 if you have one.

← Back to Orientation · Next: The KV Cache →