Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

03 · Coding Agent on Long Context

3 min read

Status: 📋 Stub. Title and brief only; content not yet written.

Shape: long context · bursty · KV-cache-pressure-bound

An agentic coding assistant working over a real repository. Each turn carries 40k–60k tokens of file context, tool output and conversation history, and the overwhelming majority of it is byte-identical to the previous turn. Traffic is spiky in a specific way: a developer mid-task generates ten turns in four minutes, then goes to a meeting and generates nothing for half an hour, and thirty developers do this out of phase with each other.

This is the study where the KV cache is unambiguously the constraint and where the two headline metrics fight each other: a 60k-token prefill occupies the GPU long enough to stall every other user's decoding, and the fix for that trades a little of each user's steady-state speed for a lot of everyone else's fairness. It's also the study where one number — the prefix cache hit rate — is worth more than every other dashboard on the page combined.

Exercises: long context and max_model_len · chunked prefill and the head-of-line blocking it solves · preemption, and reading preemption warnings in the logs · prefix cache hit rate as the primary metric · speculative decoding and its acceptance rate · burst capacity versus idle cost.

Runnable artefact: scripts/03-context-pressure-model.py — no GPU required. Simulates a bursty long-context workload against a fixed block pool, and reports how many concurrent sessions fit before preemption starts and how hit rate moves the answer.

The Brief

📋

Architecture

📋

First Attempt

📋

What Broke

📋

The Diagnosis

📋

The Fix

📋

The Numbers

📋 All figures on this page are modelled, not measured — see the stage note.

What Changes at 10×

📋

Lessons

📋

Cheat Sheet

📋

← Previous: Offline batch summarisation · Back to Case Studies · Next: Multi-tenant LoRA on one GPU →

⚠️ Verification checklist (delete before publishing)

  • Page is a stub. Nothing below the brief has been written or verified.
  • Confirm whether chunked prefill is on by default on the target version before this page asserts it — three other pages currently assert it and the claim is unverified.
  • Prefix cache eviction policy must be verified against the source, not inferred, before any claim about what survives a thirty-minute idle gap.
  • Speculative decoding acceptance-rate figures must be labelled as illustrative unless sourced.