Stage 3 — Running It Locally

You have a server running, you've called it three different ways, and you can read its startup log. · ~3 days

Three stages of theory, and you haven't installed anything. That was deliberate: you now know what gpu_memory_utilization does before you're asked to choose one, and the startup log you're about to read is a list of things you can already name.

This stage is short, because the hard part is behind you. It is also the stage where the article's promise gets tested — everything here runs on a free Colab T4.

Pages

#	Topic	What it covers	Status
1	Installing vLLM	uv versus pip; matching CUDA and driver versions; the Docker route; the five install errors everyone hits; getting a Colab T4 ready	🚧
2	Offline Batch Inference	The LLM class end to end; SamplingParams; why batch mode is the honest way to measure throughput	🚧
3	The OpenAI-Compatible Server	vllm serve; completions versus chat completions; chat templates; curl and the openai client	🚧
4	Streaming & Client Patterns	Server-sent events; streaming with the openai client; timeouts, cancellation, and the abandoned request	🚧
5	The Getting-Started Notebook	One Colab-runnable artifact tying Stages 0–3 together: load, generate, serve, stream, measure	🚧

What you'll be able to do by the end

• Install vLLM without breaking an existing environment, and diagnose it when it breaks anyway.
• Read a startup log line by line and say what each number means.
• Generate offline with the LLM class, and serve online with vllm serve.
• Call the server with curl, the openai client, and a streaming client — and know what differs.
• Measure TTFT and ITL on your own hardware, closing the loop on every experiment in Stages 0–2.

Before you start

One number decides whether any of this works: your GPU's compute capability must be 7.5 or higher. A Colab T4 is exactly 7.5 — at the floor, not above it. Check before installing:

nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

---

← Back to The Engine · Next: Installing vLLM →