Back to Articles/vLLM & LLM Inference Serving

01 · Installing vLLM

19 min read

The install is one command and it works most of the time. This page is about the rest of the time — and about the fact that the command most tutorials give you is no longer the recommended one.

Anatomy of a vLLM Setup established the shape: four layers that have to agree, of which you control roughly one. This page is the operational version of that — what to type, what breaks, and how to tell which layer broke.

The Problem

pip install vllm replaced the torch in your project and now something unrelated is broken.
It installed and then failed at import with an undefined symbol, or a CUDA error naming a library you've never heard of.
You're on a machine with an older driver and everything you try says the CUDA version is wrong.
You tried to install a nightly build with pip and got the released version instead, silently.
It works on your laptop and not in CI, or vice versa, and the difference is invisible.
Your GPU is a V100 and nothing works, for a reason no error message states plainly.

Every one of these is a version-agreement failure between the four layers. The install command isn't the interesting part; knowing which layer disagreed is.

The Idea

Installing vLLM is closer to installing a printer driver than a Python library.

An ordinary Python package is portable code — it runs anywhere Python runs, and pip's job is to fetch compatible versions of some other Python packages. vLLM is a large body of pre-compiled CUDA kernels built against one specific PyTorch and one specific CUDA version. The wheel isn't "a package that depends on torch"; it's a binary that was compiled assuming an exact torch, and will crash if it meets a different one.

That reframes everything awkward about it:

Why it's huge and slow to install — you're downloading compiled binaries, not source.
Why it overwrites your torch — it isn't being rude; it needs the one it was built against.
Why "just use my existing PyTorch" requires building from source — you're asking for a recompile, because the shipped binary won't match.
Why a fresh virtual environment is the standard advice, not a nicety.

The practical rule: give vLLM its own environment and let it bring whatever it wants. Fighting the pinning is a category error.

Under the Hood

The one requirement that's absolute

GPU compute capability 7.5 or higher.

That's the documented floor — T4, RTX 20xx, A100, L4, H100, B200 and newer. It's worth being precise about because it excludes hardware people still have:

GPU	Compute capability	Runs vLLM?
V100 (Volta)	7.0	❌ Below the floor
T4 (Turing)	7.5	✅ Exactly at the floor
RTX 20xx (Turing)	7.5	✅
A100 (Ampere)	8.0	✅
L4 / L40S (Ada)	8.9	✅
H100 (Hopper)	9.0	✅

The article's baseline — a Colab T4 — sits exactly on the minimum. That's a deliberate choice: if it runs there, it runs anywhere supported.

⚠️ The quantisation compatibility matrix in Quantisation lists Volta rows for GPTQ, bitsandbytes and GGUF, which sits oddly with a 7.5 floor for the CUDA build. Treat 7.5 as the requirement and the Volta entries as historical or applying to a non-default build path.

What the wheel actually contains

vLLM's binaries are pre-compiled against CUDA 12.9 by default, with alternative builds published for CUDA 12.8 and 13.0. The compilation "introduces binary incompatibility with other CUDA versions and PyTorch versions" — vLLM's own words — "even for the same PyTorch version with different building configurations."

So there are exactly three positions you can be in:

Situation	What to do
Fresh environment, standard CUDA	Install the wheel. This is the happy path
Different CUDA version	Install a matching published variant (12.8, 13.0)
Existing PyTorch you must keep	Build from source. There is no wheel that will fit

`uv` is now the recommended installer

This is the part most tutorials haven't caught up with. The documented command is:

uv pip install vllm --torch-backend=auto

--torch-backend=auto inspects your installed CUDA driver version and selects the matching PyTorch index automatically — which is precisely the version-agreement problem, solved by tooling rather than by you reading tables.

The pip equivalent requires you to name the index yourself:

pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

And for nightly builds, pip is not supported at all. The reason is a genuine behavioural difference: pip merges --extra-index-url with the default index and picks the highest version, so a development build numbered below the release gets silently passed over. uv gives the extra index priority. If you use pip for a nightly you must specify the full wheel URL.

The driver, and the escape hatch

The CUDA runtime ships inside the wheel; the driver is host state and pip cannot touch it. On a machine whose driver is older than the image's CUDA toolkit, the Docker image offers a way out:

--env "VLLM_ENABLE_CUDA_COMPATIBILITY=1"

This uses CUDA compatibility libraries bundled in the image, configuring LD_LIBRARY_PATH before PyTorch loads. It only supports select professional and datacenter GPUs — but on a managed cluster where you can't update the driver, it's often the difference between running and not.

Three ways in, and when each is right

Route	Command	Use when
uv wheel	`uv pip install vllm --torch-backend=auto`	Default. Fastest path to working
Docker	`vllm/vllm-openai:latest`	Reproducibility, CI, production, or a host you can't modify
From source	`uv pip install -e . --torch-backend=auto`	You must keep an existing PyTorch, or you're changing kernels

The Docker invocation is worth reading closely, because two of its flags are load-bearing:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \   # ← persist weights across restarts
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8000:8000 \
    --ipc=host \                                          # ← or --shm-size; PyTorch needs shared memory
    vllm/vllm-openai:latest \
    --model Qwen/Qwen2.5-0.5B-Instruct

The volume mount is the fix for the disk-and-startup problem from Anatomy of a vLLM Setup — without it, every container restart re-downloads the weights.

--ipc=host exists because PyTorch uses shared memory for inter-process communication, particularly under tensor parallelism, and Docker's default shared-memory allocation is too small. Omit it and you get failures that look like the model's fault.

Try It

Hardware: Colab T4, or any machine with compute capability ≥ 7.5.

Step 1 — check before you install

nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv
df -h ~/.cache
python -c "import sys; print(sys.version)"

Three gates, in order of how expensive they are to discover late:

Check	Requirement	If it fails
`compute_cap`	≥ 7.5	Stop. No install will help
Free disk on the cache path	Tens of GB	Set `HF_HOME` to a bigger volume before installing
OS	Linux	Use WSL2, Docker, or a remote box

Step 2 — install into a fresh environment

# The documented path
uv venv && source .venv/bin/activate
uv pip install vllm --torch-backend=auto

# On Colab, where a torch already exists and uv may not:
!pip install vllm
# Expect a restart prompt. Take it — the runtime's torch has been replaced.

Step 3 — read the startup log, which is the actual lesson

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096 2>&1 | tee startup.log

You have spent three stages learning to read this. Find these lines:

Look for	What it tells you	Where it was taught
GPU blocks allocated	× 16 = your total cacheable tokens — your real capacity	PagedAttention
The resolved `max_model_len`	May be lower than you asked, if the cache can't hold it	The KV Cache
The attention backend chosen	Which kernel path you actually got	Anatomy
Memory profiling figures	Weights, activations, and what's left	Stage 0 capacity arithmetic

Now do the comparison that closes Stage 0. Take the GPU-blocks number, multiply by 16, divide by max_model_len, and compare against what capacity.py predicted:

predicted (capacity.py):  247 concurrent @ 4096
actual (startup log):     num_gpu_blocks × 16 ÷ 4096 = ?

The gap is the overhead term the script estimated at 1 GB. Whatever that gap is, it's the honest error bar on every capacity number in this article — and knowing it for your own hardware is worth more than any figure I could publish.

Now change one thing

Re-run with --gpu-memory-utilization 0.5 and watch the GPU-blocks number fall. The relationship should be close to linear in the memory left after weights — which is the memory-budget diagram from Stage 0, confirmed on your own card.

Dial It In

Knob	What it does	When
`--torch-backend=auto`	Selects the PyTorch index from your driver version	Always, with `uv`
`--torch-backend=cu130`	Pins a specific CUDA backend	Your driver needs a non-default variant
`HF_HOME`	Where weights are cached	Set it before the first run. Retrofitting means re-downloading
`VLLM_ENABLE_CUDA_COMPATIBILITY=1`	Uses bundled compatibility libraries for older drivers	Docker, on a host you can't update. Select GPUs only
`MAX_JOBS`	Caps parallel compile jobs when building from source	Source builds on small machines — WSL defaults to half your RAM
`VLLM_TARGET_DEVICE=empty`	Builds without compiling	macOS development only. Imports work; nothing runs
`--ipc=host` / `--shm-size`	Shared memory for the container	Always, with Docker

The two that prevent the most pain: set HF_HOME before your first download, and use a fresh environment. Both are trivial upfront and expensive to retrofit.

Where It Bites You

Installing into an environment that already matters. vLLM brings its own pinned PyTorch and will replace yours. On Colab, expect a restart prompt and take it — the runtime's torch has genuinely been swapped underneath you.

Using pip for nightly builds. It doesn't work, and it fails silently by installing the release version instead. If you need a nightly, use uv, or specify the full wheel URL.

PyTorch installed via conda. It statically links NCCL, which causes problems when vLLM tries to use NCCL. This is documented, non-obvious, and produces errors that look like networking faults.

Assuming a V100 will work. Compute capability 7.0 is below the 7.5 floor. This is the one hardware failure no flag fixes, and it strands people who have older institutional GPUs.

Forgetting --ipc=host in Docker. PyTorch needs shared memory for inter-process communication. Without it you get failures under tensor parallelism that look like model or driver problems.

Not mounting the HuggingFace cache in Docker. Every restart re-downloads gigabytes. In Kubernetes this becomes minutes of pod startup and occasional full node disks — and it's an autoscaling problem, because scale-up latency includes it.

Expecting to keep your existing PyTorch. You can, but only by building from source. There is no wheel that will accommodate it, and trying produces undefined-symbol errors at import.

Building from source on a small machine. The compile is heavy and parallel by default. Under WSL, which is allocated half your RAM by default, MAX_JOBS=1 may be the difference between a slow build and an out-of-memory failure.

In Production

Pin the vLLM version and the CUDA variant explicitly. vllm==0.26.0 with a known backend, not vllm. A minor version can change a scheduler default, which changes your throughput, which you'll discover as a regression with no code change to blame.

Prefer the Docker image over a pip install. It resolves the four-layer agreement problem once, at build time, and ships it as one artifact. That's worth more than the image size for anything you'll be paged about — and it's the subject of Containerising vLLM.

Treat the driver as a fleet property, not a machine property. It's the one layer pip can't fix, so it belongs in your node-image definition alongside the kernel. VLLM_ENABLE_CUDA_COMPATIBILITY is a workaround for a mismatch, not a substitute for a policy.

Cache weights on shared, persistent storage. From Stage 0: baked into the image, on a shared volume, or downloaded at start. In production the middle option is usually right, and the decision is mostly about how fast a new replica must become ready.

What changes at 10× traffic. Nothing about installation — but everything about startup, which you'll now do far more often. Cold-start time becomes a scaling constraint, and it's dominated by weight loading and CUDA graph capture rather than by anything on this page. Measure it before you need it.

Check Yourself

Recall the idea

Why is vLLM's install unlike an ordinary Python package's?

The wheel contains pre-compiled CUDA kernels built against one exact PyTorch and CUDA version. It's a binary artifact, not portable code — so it must bring its own PyTorch, and a mismatch is an undefined symbol at import rather than a graceful degradation.

What's the minimum compute capability, and which common GPU falls below it?

7.5. The V100 (Volta, 7.0) is below the floor. A T4 is exactly 7.5 — at the minimum, not above it.

Why is uv recommended over pip?

--torch-backend=auto inspects your CUDA driver and picks the matching PyTorch index automatically, solving the version-agreement problem for you. And pip cannot install nightly builds correctly, because it merges indexes and picks the highest version number, silently skipping development builds.

What does --ipc=host do, and why does Docker need it?

It gives the container access to the host's shared memory. PyTorch uses shared memory for inter-process communication — particularly under tensor parallelism — and Docker's default allocation is too small. --shm-size is the alternative.

Explain the mechanics

Which of the four layers can pip actually change?

Only the Python-side ones: the vLLM wheel and the PyTorch and CUDA runtime bundled inside it. The driver is host state, and the GPU is hardware. That's why a CUDA error at startup usually means "update the host driver" rather than "reinstall vLLM".

You must keep an existing PyTorch build. What are your options?

Build vLLM from source against it — python use_existing_torch.py, then install with build isolation disabled — because no published wheel will match. Alternatively, isolate them: give vLLM its own environment or container, and let your other project keep its PyTorch. Isolation is usually cheaper than a source build.

Why does installing a nightly with pip silently give you the release version?

pip combines packages from --extra-index-url with the default index and selects the highest version. A development build is numbered below the next release, so the release wins. uv gives the extra index higher priority, which is why it's the supported path.

What does VLLM_ENABLE_CUDA_COMPATIBILITY do, and what are its limits?

It configures LD_LIBRARY_PATH to use CUDA compatibility libraries bundled in the Docker image before PyTorch loads, letting the container run on hosts whose driver is older than the image's CUDA toolkit. It only supports select professional and datacenter GPUs, so it's a targeted escape hatch rather than a general fix.

Reason about a trade-off

Your institution has a cluster of V100s. What do you tell them?

That vLLM's documented requirement is compute capability 7.5 and the V100 is 7.0, so this is a hardware constraint rather than a configuration problem — no flag, driver or build resolves it. Options: newer GPUs for vLLM workloads; a different engine, since llama.cpp and some GPTQ paths historically supported Volta; or CPU serving for low-concurrency work, which vLLM does support. Worth checking the current requirement directly before writing off the hardware, since floors move.

Docker image or pip install for a production deployment?

Docker, in almost all cases. It resolves the CUDA/PyTorch/driver agreement once at build time and ships one artifact, which is exactly what you want when the alternative is diagnosing a version mismatch during an incident. The costs — a large image, a slower registry pull — are real but one-time and predictable. Keep pip for development, where iteration speed matters more than reproducibility.

A colleague wants to pin only vllm and let everything else float. Respond.

Point out that vLLM's wheel is compiled against an exact PyTorch and CUDA combination, so "everything else" isn't free to float — the wheel already pins it, and pinning only vLLM just hides the real constraint. The productive version is to pin vLLM and record the CUDA variant and the driver baseline your nodes provide, because those three together determine whether it runs. Then the failure is a version conflict you can read, not an undefined symbol.

When is building from source justified?

Two cases: you're modifying vLLM's C++ or CUDA kernels, or you have a PyTorch build you genuinely cannot replace — a custom or nightly torch another part of your stack depends on. Otherwise it costs you a long compile, a reproducibility burden and a class of build failures, for no benefit over the published wheel. If you only need to change Python code, the precompiled editable install (VLLM_USE_PRECOMPILED=1) gives you a working environment without the full compile.

Cheat Sheet

Install

# recommended
uv pip install vllm --torch-backend=auto

# pip equivalent (must name the index yourself)
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

# Docker
docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest --model Qwen/Qwen2.5-0.5B-Instruct

Pre-flight

nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv   # need cap >= 7.5
df -h ~/.cache                                                                     # tens of GB
export HF_HOME=/mnt/models                                                         # BEFORE first run

The numbers

Number	Meaning
7.5	Minimum compute capability. T4 is exactly this; V100 (7.0) is below it
CUDA 12.9	Default build. Variants published for 12.8 and 13.0
Linux	The supported OS. WSL2 or Docker otherwise

Failure → cause

Symptom	Layer	Fix
Undefined symbol at import	Wheel vs PyTorch mismatch	Fresh environment
CUDA error at startup	Host driver	Update the driver, or `VLLM_ENABLE_CUDA_COMPATIBILITY=1` in Docker
Nightly install gave the release	`pip` index merging	Use `uv`, or the full wheel URL
Shared-memory failure under TP	Docker defaults	`--ipc=host` or `--shm-size`
Disk full / slow restarts	Unmounted cache	Mount `~/.cache/huggingface`, set `HF_HOME`
Nothing works on a V100	Compute capability 7.0	Different hardware or a different engine

The habit: read the startup log every time. GPU blocks, resolved max_model_len and the attention backend are all there — and you now know what all three mean.

Sources

vLLM, GPU installation (CUDA) — the 7.5 compute-capability requirement, CUDA 12.9 default build, uv --torch-backend=auto, the pip nightly caveat, Docker flags and VLLM_ENABLE_CUDA_COMPATIBILITY — github.com/vllm-project/vllm/blob/main/docs/getting_started/installation/gpu.cuda.inc

← Back to Running It Locally · Next: Offline Batch Inference →

⚠️ Verification checklist (delete before publishing)

Verified against vLLM's installation docs this session

Compute capability requirement is 7.5, not 7.0 — "GPU: compute capability 7.5 or higher (e.g., T4, RTX20xx, A100, L4, H100, B200)". This corrects Anatomy of a vLLM Setup, which says ≥ 7.0 and lists V100 as supported. Propagated — see below.
Binaries are pre-compiled against CUDA 12.9 by default, with 12.8 and 13.0 variants published.
uv pip install vllm --torch-backend=auto is the documented recommended install.
pip is not supported for nightly indices, because it merges indexes and selects the highest version.
conda-installed PyTorch statically links NCCL and causes problems.
Docker flags: --ipc=host (or --shm-size) for PyTorch shared memory; HF cache volume mount.
VLLM_ENABLE_CUDA_COMPATIBILITY=1 exists, works via LD_LIBRARY_PATH, and supports only select professional/datacenter GPUs.
VLLM_TARGET_DEVICE=empty for non-Linux development builds; MAX_JOBS for constrained builds; WSL's 50% memory default.

Propagation required

Stage 0 page 3 corrected from ≥ 7.0 to ≥ 7.5, and the V100 row fixed.
Unresolved tension inside vLLM's own docs: the quantisation matrix lists Volta (7.0) rows for GPTQ, bitsandbytes, DeepSpeedFP and GGUF, while the CUDA install requirement is 7.5. Flagged inline on this page. Determine whether the Volta rows are stale, apply to a non-default build, or whether the floor is softer than stated.

Needs verifying

Confirm the current stable version to pin in the In Production example (0.26.0 used).
Confirm Colab still requires a runtime restart after pip install vllm.
Confirm the startup log emits all four lines the Try It table tells readers to find, and capture their exact wording.

Code

Run the full Try It sequence on a Colab T4. The key capture is the GPU-blocks number, so the predicted-vs-actual comparison against capacity.py's 247 can be filled in — that gap is the honest error bar on every capacity figure in the article and is currently unquantified.
Run the --gpu-memory-utilization 0.5 variation and confirm the blocks number falls roughly linearly in post-weights memory.

Rendering

No diagram on this page yet. Consider one for the three install routes, or reuse the Stage 0 layer-stack image rather than adding a near-duplicate.
All relative links resolve once target files exist.