02 · Where vLLM Sits

The previous page argued that you need an inference server. This one is about the fact that there are five credible ones, they are not interchangeable, and the differences between them are smaller than their marketing suggests and larger than their benchmarks admit.

The useful output of this page is not "vLLM is best." It's a decision you can defend in a design review, including the three cases where the defensible decision is something else.

---

The Problem

You've accepted that a naive loop won't do. So you search, and within twenty minutes you have:

• A benchmark showing SGLang beating vLLM by 29%.
• A benchmark showing TensorRT-LLM beating both.
• A blog post saying just use Ollama, it's one command.
• A HuggingFace page for TGI that doesn't mention it's in maintenance mode until you look closely.
• Four "vs" articles that each conclude in favour of whoever published them.

Every one of those benchmarks is real. They disagree because they measured different workloads, and none of them measured yours. The symptom this page treats is the specific, expensive mistake that follows: you pick an engine on a benchmark number, spend two weeks integrating it, and discover that the axis it optimised isn't the axis your workload sits on.

Two concrete versions of that mistake, both common:

• A team picks TensorRT-LLM for peak throughput, then discovers every model or config change requires an ahead-of-time compilation step measured in tens of minutes, and their weekly fine-tune cadence now has a build queue.
• A team picks Ollama because it was working in five minutes, ships it behind an HTTP endpoint, and finds at twenty concurrent users that they have re-created every failure from the previous page.

---

The Idea

Think of it as choosing a database, not choosing a compiler.

Nobody asks "which database is fastest?" without immediately being asked back: fastest at what — point lookups, analytical scans, writes, joins? Everyone accepts that Postgres, SQLite, ClickHouse and DynamoDB are all "the fastest" on some axis, that the axes are genuinely different, and that SQLite winning on your laptop says nothing about DynamoDB losing.

Inference engines are the same and get discussed as though they aren't. There are four axes, and almost every disagreement between two benchmarks is the two of them sitting at different points on one of these:

Axis	The question	Who wins the extreme
Concurrency	One user, or hundreds?	llama.cpp at 1; vLLM/SGLang at hundreds
Prefix sharing	Do your requests share long prefixes?	SGLang when they do; nothing to gain when they don't
Hardware breadth	One fixed NVIDIA SKU, or whatever you can get?	TensorRT-LLM if fixed; vLLM if not
Iteration speed	How often do the model and config change?	vLLM/SGLang if often; TensorRT-LLM tolerable if never

Say where you sit on those four and the choice usually makes itself. Skip that step and you are choosing on someone else's workload.

---

Under the Hood

Each engine made one architectural bet. The bet explains the benchmark, and it explains the cost.

vLLM — pay allocation cost at runtime, in exchange for flexibility

Paged KV cache plus an iteration-level scheduler, both computed live. Nothing is compiled ahead of time for your specific model and configuration, so you can change model, sequence length, batch limits and adapters without a rebuild — and you can run on NVIDIA, AMD ROCm, Intel and TPU backends from the same codebase. The cost is that you leave some peak performance on the table relative to an engine that specialises the kernels for your exact shapes.

This is why vLLM is the default. Not because it wins benchmarks, but because it is the engine that is good on the widest range of workloads and hardware while remaining a pip install.

SGLang — bet that your requests share prefixes

The differentiating idea is RadixAttention: rather than caching prefixes by exact block hash, maintain a radix tree (prefix tree) over the KV cache so that partially-overlapping requests share computation automatically, with tree-aware eviction. On workloads where requests genuinely share long prefixes — multi-turn chat, agent loops replaying a trajectory, RAG over a common document set — this is a structural advantage that vLLM's prefix caching only partly matches. Reported margins over vLLM on those workloads are around 29%, narrowing to roughly 3–5% on very large models where other costs dominate.

The honest reading: the gap is a function of your prefix-sharing ratio. If your requests are independent one-shot prompts, there is nothing for the radix tree to find, and the advantage is close to zero. This is the single most workload-dependent claim in the whole landscape, which is why Try It below makes you measure your own ratio before believing anyone's number.

TensorRT-LLM — compile ahead of time for one exact target

You build an engine artifact specialised to a model, a precision, a GPU architecture, and a set of shape constraints. Because the kernels are fused and specialised for exactly that, it produces the best peak throughput and latency available on NVIDIA hardware, particularly on newer architectures.

The bet's cost is everything downstream of "specialised to": compilation is a real build step (commonly tens of minutes), it must be redone when the model or key config changes, the artifact is tied to the GPU architecture it was built for, and you are NVIDIA-only by construction. Setup is commonly described as a one-to-two-week integration rather than an afternoon.

Choose it when the model is stable, the hardware is fixed, and the throughput difference is worth money. At sufficient scale it clearly is. At most scales the engineering time costs more than the GPUs it saves.

llama.cpp and Ollama — bet that there is one user

A C++ inference implementation built around quantised GGUF weights, memory-mapping, and running well on CPU, Apple Silicon and modest GPUs. Ollama is a friendly wrapper over that with model management and a nice CLI.

They are not competing with vLLM and are constantly benchmarked as though they were. Their bet is that concurrency is one, memory is scarce, and the machine is a laptop or a small box. On that workload they are excellent and vLLM is actively worse — vLLM pre-allocates most of your VRAM at startup and buys you scheduling machinery you have no use for. On the many-concurrent-users workload they fall over, because they are not doing the things the previous page said needed doing.

TGI — the one to know the status of

HuggingFace's Text Generation Inference was for a period the obvious production choice and appears in a great deal of still-circulating documentation. It is now in maintenance mode, with HuggingFace themselves pointing users at vLLM or SGLang. It still works; it is not where new work should start.

Knowing this is worth more than any benchmark on this page, because the failure it prevents — adopting a framework whose upstream has stopped investing — is the expensive one.

⚠️ Project status changes. Verify TGI's current status before repeating this in a design review; it is stated here as of writing.

The category error: engines versus orchestrators

Ray Serve, KServe, BentoML and Databricks Model Serving are not alternatives to vLLM. They are layers around an engine that handle replicas, routing, autoscaling, versioning and multi-model deployment — and most of them run vLLM underneath. "Should we use vLLM or KServe?" is not a question with an answer; you will likely use both. This distinction gets its full treatment in The Serving Landscape.

The decision table

	vLLM	SGLang	TensorRT-LLM	llama.cpp / Ollama	TGI
Best at	General-purpose GPU serving	Shared-prefix workloads	Peak NVIDIA performance	Single-user, small hardware	—
Setup cost	pip install	pip install	Days to weeks, plus a build step	Minutes	Moderate
Hardware	NVIDIA, AMD, Intel, TPU	Mainly NVIDIA	NVIDIA only	CPU, Apple Silicon, any GPU	NVIDIA
Model/config changes	Restart	Restart	Recompile	Restart	Restart
Concurrency sweet spot	High	High	High	~1	High
Status	Active, de facto default	Active	Active, NVIDIA-backed	Active	Maintenance mode
Pick it when	You want one answer that's right most of the time	Multi-turn chat, agents, RAG with shared context	Fixed model + fixed NVIDIA fleet, at scale	Local, desktop, embedded, one user	Existing deployment only

---

Try It

Hardware: none. This runs anywhere Python does, and it is the most important experiment in Stage 0.

You cannot choose an engine from someone else's benchmark. You can choose one from three numbers about your traffic, and you can compute all three from a log file without a GPU.

Measure your own workload shape

# workload_shape.py — point this at your request logs (or a sample of expected prompts).
# Produces the three numbers that decide the engine choice.
import json, statistics
from collections import Counter

# Expect a JSONL file, one record per request:
#   {"ts": 1723300000.0, "prompt": "...", "output_tokens": 214, "duration_s": 3.4}
RECORDS = [json.loads(l) for l in open("requests.jsonl")]

# --- 1. Peak concurrency -----------------------------------------------------
# The number that decides whether you need a server at all.
events = []
for r in RECORDS:
    events.append((r["ts"], +1))
    events.append((r["ts"] + r["duration_s"], -1))
events.sort()
cur = peak = 0
for _, delta in events:
    cur += delta
    peak = max(peak, cur)

# --- 2. Length distribution --------------------------------------------------
# Decides how much KV cache each request costs, and whether long-context
# scheduling (Stage 4) is going to be your problem.
prompt_chars = [len(r["prompt"]) for r in RECORDS]
out_tokens   = [r["output_tokens"] for r in RECORDS]

def pct(xs, p):
    return statistics.quantiles(xs, n=100)[p - 1] if len(xs) > 1 else xs[0]

# --- 3. Shared-prefix ratio --------------------------------------------------
# THE number for the vLLM-vs-SGLang question. Fraction of prompt characters
# covered by a prefix that appears in more than one request.
PREFIX_LEN = 256
prefixes = Counter(r["prompt"][:PREFIX_LEN] for r in RECORDS)
shared = sum(n for p, n in prefixes.items() if n > 1)
shared_ratio = shared / len(RECORDS)

print(f"requests               : {len(RECORDS)}")
print(f"peak concurrency       : {peak}")
print(f"prompt chars  p50/p99  : {pct(prompt_chars,50):.0f} / {pct(prompt_chars,99):.0f}")
print(f"output tokens p50/p99  : {pct(out_tokens,50):.0f} / {pct(out_tokens,99):.0f}")
print(f"shared-prefix ratio    : {shared_ratio:.1%}  (first {PREFIX_LEN} chars)")

# UNVERIFIED — needs your own request logs. See docs/VERIFICATION.md, optional section.
# Example output shape only.
requests               : 41892
peak concurrency       : 37
prompt chars  p50/p99  : 1840 / 22610
output tokens p50/p99  : 186 / 1204
shared-prefix ratio    : 71.4%

Read the result

If you measured	The engine question is	Because
Peak concurrency ≤ 2	Do you need vLLM at all? Look hard at llama.cpp/Ollama	There is no batching opportunity to exploit; you're paying for a scheduler that has nothing to schedule
Peak concurrency ≥ 10	vLLM or SGLang. Ollama is out	This is exactly the regime the previous page's five failures live in
Shared-prefix ratio > ~50%	Benchmark SGLang against vLLM on your traffic before committing	This is the regime where SGLang's reported advantage is real rather than theoretical
Shared-prefix ratio < ~20%	The SGLang-vs-vLLM benchmarks you read do not apply to you	RadixAttention has nothing to share; the margin collapses
p99 output tokens ≫ p50	Continuous batching is not optional for you	This is head-of-line blocking waiting to happen
p99 prompt chars ≫ p50	Long-context scheduling will be your bottleneck, not engine choice	See Long Context & Chunked Prefill

Now change one thing

Re-run with PREFIX_LEN set to 64, then 1024. What you should observe: the shared-prefix ratio falls as the prefix window grows — sharply if your sharing is just a short system prompt, gently if your requests genuinely share long context.

That shape is the answer to "would SGLang help us?", and it's a shape you can only get from your own data. A workload sharing a 200-token system prompt and nothing else has far less to gain than one where every agent step replays a 4,000-token trajectory, even though both report high sharing at PREFIX_LEN = 256.

If you have no logs yet: write the prompts you expect, twenty of them, and run this anyway. A guess you wrote down and can revisit beats a benchmark someone else ran.

---

Where It Bites You

Every published benchmark is an argument, not a measurement. Not because anyone is lying — because a benchmark requires choosing a model, hardware, concurrency, prompt distribution and target latency, and those five choices determine the winner before any code runs. The specific tell: a comparison that reports throughput without stating concurrency, or that uses identical repeated prompts (which turns the test into a prefix-cache benchmark). Before believing any number, ask what the load shape was.

"We benchmarked vLLM and it lost." Usually true and usually uninformative, because the vLLM instance was at default settings against a competitor tuned for the test. Defaults are chosen to be safe across workloads, not to win benchmarks. A fair comparison tunes both, which almost nobody does.

Choosing TensorRT-LLM without pricing the build step. The throughput is real. So is the recompilation on every model update, precision change or major config change, and so is the coupling to a GPU architecture. Teams that fine-tune weekly discover they've added a multi-tens-of-minutes build to a weekly cadence. Price the workflow, not just the tokens per second.

Shipping Ollama behind an HTTP endpoint. It works in testing, where concurrency is one. The failure arrives with the tenth simultaneous user and looks exactly like the previous page's list. Ollama is not badly built — it is built for a different problem.

Adopting TGI from a still-highly-ranked 2024 tutorial. Maintenance mode means security fixes, not new model architectures. If you need a model released after upstream slowed down, you may simply not be able to serve it.

Confusing the engine with the orchestrator. "vLLM vs KServe" and "vLLM vs Ray Serve" are category errors that produce genuinely bad architecture decisions — usually a team building their own replica management because they thought the engine should have provided it.

Assuming the choice is permanent. It mostly isn't, and this is the most useful thing on the page: vLLM, SGLang, TGI and TensorRT-LLM's server all expose an OpenAI-compatible API. If your application talks to /v1/chat/completions and nothing else, swapping engines is a deployment change rather than a rewrite. Which means the cost of choosing wrong is much lower than the amount of agonising the decision usually receives — provided you don't couple your application to engine-specific extensions.

---

In Production

Design for replaceability, and it stops being a one-way door. Concretely: keep your application talking to the OpenAI-compatible surface; keep engine-specific flags in deployment configuration, not in application code; keep a benchmark harness that can point at any endpoint. Do that and re-running the decision in six months costs a day. Skip it and the engine becomes load-bearing.

Most organisations end up running more than one, deliberately. vLLM or SGLang for the high-concurrency production API; llama.cpp or Ollama on developer laptops for offline work and prompt iteration; occasionally TensorRT-LLM for one high-volume model where the compile cost amortises. That isn't indecision — it's the four axes having different answers for different workloads.

The signal to watch after you've chosen. Not throughput. Watch whether your model roadmap and your engine's release cadence stay compatible: how quickly does your engine support new model architectures, and how quickly do you need them? A team on a fast-moving model roadmap is buying upstream velocity as much as tokens per second, and that's the criterion that most often invalidates a decision made purely on benchmarks.

What changes at 10× traffic. The engine choice mostly doesn't — you scale by adding replicas, and that's an orchestration problem (Kubernetes Deployment, Autoscaling), not an engine one. Where it does change is at the point a single replica can no longer hold the model, which is when tensor parallelism support and multi-node behaviour start to differentiate the engines (Tensor & Pipeline Parallelism).

---

Check Yourself

Recall the idea

Name the four axes that differentiate inference engines.

Concurrency (one user versus many), prefix sharing (do requests share long prefixes), hardware breadth (one fixed NVIDIA SKU versus anything), and iteration speed (how often model and config change). Almost every disagreement between two published benchmarks is the two of them sitting at different points on one of these.

Why is vLLM usually the default recommendation?

Not because it wins benchmarks. Because it's good across the widest range of workloads and hardware while installing with one command and exposing an OpenAI-compatible server immediately — the best expected outcome when you don't yet know your workload precisely.

What is TGI's current status and why does it matter?

Maintenance mode, with HuggingFace pointing users to vLLM or SGLang. It matters because a large volume of still-circulating tutorial content recommends it, and adopting a framework whose upstream has stopped investing is expensive in a way no benchmark shows.

Is Ray Serve an alternative to vLLM?

No — it's an orchestration layer that typically runs vLLM underneath. Engines execute the model; orchestrators handle replicas, routing, autoscaling and versioning. You will probably use one of each.

Explain the mechanics

What does SGLang's RadixAttention do that ordinary prefix caching doesn't, and when does it not matter?

It maintains a radix (prefix) tree over cached KV blocks, so partially-overlapping requests share computation automatically and eviction is tree-aware, rather than matching only on exact block hashes. It stops mattering when requests don't share prefixes — independent one-shot prompts give the tree nothing to find, and the reported advantage collapses toward zero.

Why does TensorRT-LLM get better peak performance, and what does that cost?

It compiles an engine artifact specialised to one model, precision, GPU architecture and shape range, allowing kernel fusion and specialisation that a runtime-flexible engine can't do. The cost is that everything "specialised to" becomes rigid: a real build step on every model or config change, coupling to the GPU architecture, and NVIDIA-only by construction.

Why is llama.cpp better than vLLM for a single local user?

Because vLLM's advantages are all contention-management, and there's no contention. Meanwhile llama.cpp's bets — aggressive GGUF quantisation, memory-mapped weights, CPU and Apple Silicon support — are exactly right for scarce memory and one request, while vLLM pre-allocates most of your VRAM at startup for a batch that will never be full.

Why is engine choice cheaper to reverse than teams assume?

Because the major engines all expose an OpenAI-compatible HTTP API. An application that talks only to /v1/chat/completions can swap engines as a deployment change. The reversibility is conditional: it holds only if you avoid coupling application code to engine-specific extensions.

Reason about a trade-off

Your team runs multi-turn chat with a 3,000-token system prompt at ~40 concurrent users on A100s. A benchmark shows SGLang 29% ahead. Do you switch?

You investigate seriously, because this is the workload where the number is most likely to be real — long shared prefix, high concurrency, exactly RadixAttention's target. But you don't switch on someone else's benchmark. Measure your own shared-prefix ratio, then benchmark both on your traffic shape with both tuned, and weigh the result against the migration and operational cost of a second framework your team doesn't know. A 29% throughput gain is roughly 29% of your GPU bill — whether that's worth it is an arithmetic question you can actually answer, not a taste question.

A director asks why you're not using TensorRT-LLM when NVIDIA benchmarks show it's fastest. Answer.

Agree with the premise — on fixed NVIDIA hardware with a stable model, it is genuinely the peak, and the benchmark isn't dishonest. Then make it a total-cost argument: the compilation step must be re-run on every model and config change, which for our current release cadence adds a build stage to each deployment; the artifact is tied to the GPU architecture, so a fleet change means a rebuild; and integration is measured in weeks rather than days. Quantify: at our current spend, an X% throughput gain saves £Y a month against Z engineer-weeks and an ongoing workflow tax. Then name the condition that flips the answer — if the model stabilises and volume grows past a stated threshold, it becomes the right call, and we should revisit then.

When should you deliberately run two engines?

When the axes genuinely differ between workloads: high-concurrency production on vLLM or SGLang, developer laptops and offline prompt iteration on Ollama, and — at sufficient volume — one stable high-traffic model on TensorRT-LLM where the compile cost amortises. The thing to avoid is running two engines for the same workload because nobody made a decision.

What would make you re-open the engine decision, absent any performance problem?

A model architecture you need that your engine doesn't support yet, or a shift in your workload's position on one of the four axes — concurrency collapsing to near one, prefix sharing appearing where there was none, hardware moving to a fixed fleet. Upstream velocity against your model roadmap is the criterion that invalidates benchmark-based decisions most often, and it's the one nobody monitors.

---

Cheat Sheet

The one-line answers

Situation	Engine
You don't yet know your workload	vLLM
Many concurrent users, GPUs, mixed hardware	vLLM
Multi-turn chat / agents / RAG with long shared prefixes	SGLang (benchmark it against vLLM on your own traffic)
Fixed NVIDIA fleet, stable model, large scale	TensorRT-LLM
One user, laptop, CPU or Apple Silicon	llama.cpp / Ollama
Existing TGI deployment	Keep it running, plan the exit
Replicas, routing, versioning	Not an engine question — Ray Serve / KServe / BentoML over an engine

Facts worth memorising

Fact	Consequence
TGI is in maintenance mode	Don't start new work there; HuggingFace points at vLLM/SGLang
TensorRT-LLM requires ahead-of-time compilation	Model/config changes have a build step; artifact is tied to GPU architecture
SGLang's advantage scales with prefix sharing	~29% reported on sharing workloads, ~3–5% on very large models, ≈0 without sharing
All major engines speak the OpenAI API	Engine choice is far more reversible than it feels — if you don't couple to extensions
Ray Serve / KServe / BentoML wrap engines	"vLLM vs KServe" is a category error

The three numbers to measure before choosing

1. Peak concurrency — decides whether you need a server at all.
2. Shared-prefix ratio — decides whether the SGLang benchmarks apply to you.
3. p50 vs p99 output length — decides how badly you need continuous batching.

The question to ask any benchmark

At what concurrency, on what hardware, with what prompt distribution, with both sides tuned — and were the prompts varied or repeated?

---

Sources

• HuggingFace TGI status and current recommendations — verify at github.com/huggingface/text-generation-inference
• SGLang and RadixAttention — github.com/sgl-project/sglang
• TensorRT-LLM — github.com/NVIDIA/TensorRT-LLM
• Comparative figures summarised from 2026 landscape write-ups including Yotta Labs and DeployBase — secondary sources, see the verification checklist

---

← Previous: Why Inference Servers Exist · Next: Anatomy of a vLLM Setup →

---

⚠️ Verification checklist (delete before publishing)

Project status — the highest-risk claims on this page

• TGI maintenance mode. Confirm from HuggingFace's own repository or docs, not from a third-party comparison article. This claim carries the most weight on the page and is the most damaging if stale.
• Confirm SGLang, TensorRT-LLM and llama.cpp are all still actively maintained.
• Non-NVIDIA backend support confirmed: vLLM's V1 guide lists NVIDIA, AMD, Intel GPU, TPU and CPU all as 🟢 Functional, with further platforms via plugins (vllm-ascend, vllm-spyre, vllm-gaudi, vllm-openvino). The breadth claim in the decision table is sound.

Performance figures — all currently from secondary sources

• The "SGLang ~29% over vLLM on shared-context workloads" figure. Find the primary benchmark, note its model/hardware/concurrency, or downgrade the claim to "reported margins vary".
• The "3–5% on 70B+ models" narrowing claim — same treatment.
• "Tens of minutes" TensorRT-LLM compilation and "one to two weeks" integration — both are secondary-source characterisations. Either source them properly or soften to "a real build step" and "days to weeks".
• Confirm TensorRT-LLM engine artifacts are in fact tied to GPU architecture (not merely recommended to be rebuilt).

Technical claims

• RadixAttention description — confirm the radix-tree and tree-aware-eviction characterisation against the SGLang paper/docs rather than summaries.
• Confirm TensorRT-LLM's server exposes an OpenAI-compatible API (asserted in "all major engines speak the OpenAI API").
• Confirm the claim that vLLM pre-allocates most of VRAM at startup is stated accurately enough here given gpu_memory_utilization defaults (covered properly in Stage 4).
• Confirm Ray Serve / KServe / BentoML commonly use vLLM as a backend.

Code

• Run workload_shape.py against a real JSONL sample; confirm the concurrency sweep and statistics.quantiles percentile indexing are correct (the p-1 indexing is easy to get wrong, and behaviour with small len(xs) needs checking).
• Replace the UNVERIFIED example output with a real run.
• Confirm the PREFIX_LEN sweep produces the described behaviour on real data.

Rendering

• Image placeholders replaced; paths match image-prompts.md. Two images on this page: the four-axes sliders in The Idea and the landscape quadrant below it.
• Confirm the quadrant image shows TGI greyed out — if it doesn't, the visual contradicts the maintenance-mode point the page rests on.
• Check both images render legibly at mobile width.
• All relative links resolve once target files exist.