Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

01 · RAG Chatbot at Scale

2 min read

Status: 📋 Stub. Title and brief only; content not yet written.

Shape: latency-bound · high concurrency · heavy prefix reuse

An internal RAG assistant for a 2,000-person company, sitting over the intranet, the handbook and a few thousand engineering docs. Roughly 200 concurrent users at the morning peak. Every request carries the same ~4k-token system prompt, followed by three to five retrieved passages and the user's question. The product requirement that drives every decision on this page is a single sentence from the design review: the first word has to appear in under 800 ms, or people stop using it.

This is the study where latency is the whole game and throughput is only interesting because it's what latency degrades into when you get it wrong. The first attempt runs at acceptable throughput and misses the TTFT target by a factor of four, and the reason is visible in one metric.

Exercises: prefix caching · chunked prefill · max_num_seqs and queue depth · TTFT as an SLO · autoscaling on the right signal · a gateway in front of the server.

Runnable artefact: scripts/01-rag-capacity-model.py — no GPU required. Given a model, a GPU, a token distribution and a concurrency target, derives KV cache capacity, expected queue depth and modelled TTFT, so you can substitute your own numbers.

The Brief

📋

Architecture

📋

First Attempt

📋

What Broke

📋

The Diagnosis

📋

The Fix

📋

The Numbers

📋 All figures on this page are modelled, not measured — see the stage note.

What Changes at 10×

📋

Lessons

📋

Cheat Sheet

📋

← Back to Case Studies · Next: Offline batch summarisation →

⚠️ Verification checklist (delete before publishing)

  • Page is a stub. Nothing below the brief has been written or verified.
  • The 800 ms TTFT target is a plausible product requirement, not a measured or sourced figure — keep it labelled as the brief's constraint, never as an industry standard.
  • Confirm prefix-caching default behaviour on the target vLLM version before asserting it here; Stage 0 and Stage 1 currently disagree on this.