Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

02 · Offline Batch Summarisation

2 min read

Status: 📋 Stub. Title and brief only; content not yet written.

Shape: throughput-only · latency irrelevant · cost-dominated

Two million historical support tickets, each 300–2,000 tokens, to be summarised into a structured row in the warehouse. The job runs overnight from Airflow and nothing downstream reads the table until the morning. There is no user waiting, no TTFT, no p99. There are exactly two numbers that matter: does it finish before the 07:00 SLA, and what did it cost.

This is the study that exists to break the instincts Study 1 builds. Almost every reflex that made the chatbot fast makes this job slower and more expensive — small batches, latency-protective scheduling, a running server at all. The interesting decisions here are about saturating the GPU and about how much accuracy you're willing to trade for a smaller model or a cheaper numeric format, which is a question the latency-bound study never has to ask.

Exercises: the offline LLM class rather than a server · max_num_batched_tokens · quantisation and its accuracy cost · sorting inputs by length · spot instances and checkpointing · Airflow orchestration and idempotent restarts.

Runnable artefact: scripts/02-batch-cost-model.py — no GPU required. Given a corpus size, token distribution, GPU hourly rate and modelled throughput, derives wall-clock time and total cost, and shows the crossover where a cheaper GPU beats a faster one.

The Brief

📋

Architecture

📋

First Attempt

📋

What Broke

📋

The Diagnosis

📋

The Fix

📋

The Numbers

📋 All figures on this page are modelled, not measured — see the stage note.

What Changes at 10×

📋

Lessons

📋

Cheat Sheet

📋

← Previous: RAG chatbot at scale · Back to Case Studies · Next: Coding agent on long context →

⚠️ Verification checklist (delete before publishing)

  • Page is a stub. Nothing below the brief has been written or verified.
  • Any accuracy-cost claim about quantisation needs a named benchmark and task, or it must be stated as directional only.
  • Cloud GPU hourly rates change constantly — the cost model must take them as an input, and any illustrative rate must carry the date it was checked.