Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

02 · Offline Batch Inference

19 min read

The LLM class is the shortest path from installed to generating. It's also the honest way to measure throughput, because there's no HTTP, no network, and no client library between you and the engine — which is why every benchmark in Stages 0–2 that didn't need a server used it.

It has two behaviours that differ from the server in ways that will silently degrade your output if you don't know about them. Both are in this page, and both are the reason it isn't simply "the same thing without the port number."

The Problem

  • The same prompt gives worse answers through LLM than through vllm serve, on the same model, with the same sampling parameters. Nothing in the API suggests why.
  • You set no sampling parameters and got outputs you didn't expect — not vLLM's documented defaults, and you can't find where the values came from.
  • Your instruct-tuned model rambles, ignores instructions, or answers a different question. It worked fine in the chat UI you tested it in.
  • Your evaluation results don't match production, despite using the same model and the same prompts.
  • You benchmarked with LLM and the server was slower, and you're not sure whether that's real.

The first four are the same two causes, and they're specific enough to name now: generate() does not apply the chat template, and your sampling defaults are probably the model author's, not vLLM's.

The Idea

The server is a restaurant. The LLM class is the kitchen.

Both cook the same food on the same equipment. But the restaurant also takes your order in a language you speak, translates it into the kitchen's shorthand, plates the result and carries it out. Walk into the kitchen and hand the chef a scribbled note, and you'll get exactly what you asked for — including the parts you didn't know you were supposed to ask for.

That's the trade, and it cuts both ways:

  • You get direct access. No HTTP, no serialisation, no client library. For measuring engine behaviour, that's exactly right — you're timing the engine, not the network.
  • You inherit the responsibilities. Formatting the prompt the way the model expects is now your job, because nothing is doing it on your behalf.

The mistake isn't using the kitchen. It's assuming the kitchen will plate for you.

Under the Hood

generate() does not apply the chat template

This is the single most consequential difference, and vLLM documents it plainly: llm.generate does not automatically apply the model's chat template.

From Lifecycle of a Request, the chat template is what turns a list of messages into the string the model was actually trained on — role markers, special tokens, and often a default system prompt. /v1/chat/completions applies it. generate() does not.

So this:

llm.generate("Explain paged memory.")        # ← the model sees exactly this, raw

sends a bare string to a model that was fine-tuned to expect:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Explain paged memory.<|im_end|>
<|im_start|>assistant

The model still produces text — it's a language model, it will continue anything. But it's operating outside the format it was tuned for, and the result is the "rambles, ignores instructions" symptom. Nothing errors. The output is just quietly worse.

Two correct approaches:

# Option A — llm.chat() applies the template for you
outputs = llm.chat(
    [[{"role": "user", "content": "Explain paged memory."}]],
    sampling_params,
)

# Option B — apply it yourself, then generate
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(MODEL)
text = tok.apply_chat_template(
    [{"role": "user", "content": "Explain paged memory."}],
    tokenize=False, add_generation_prompt=True,
)
outputs = llm.generate(text, sampling_params)

When is raw generate() right? When you genuinely want completion rather than instruction following — base models, perplexity evaluation, text continuation, or measuring engine throughput where the content is irrelevant. That last case covers most of this article's benchmarks, which is why they use it.

Your sampling defaults are not vLLM's defaults

The second surprise, and it's documented but easy to miss:

By default, vLLM applies generation_config.json from the Hugging Face model repository if it exists — using sampling parameters recommended by the model's creator rather than vLLM's own defaults.

So when you write:

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct")
outputs = llm.generate(prompts)               # no SamplingParams at all

the temperature, top_p and top_k in effect come from the model author, not from vLLM and not from any table in this article's Sampling Parameters page. Different models ship different recommendations, so "the defaults" are not a fixed thing across models.

This is usually good — the model's creator knows what it was tuned for — but it has consequences:

  • Your "default" behaviour changes when you change model, with no change to your code.
  • Reproducing someone's results requires knowing which config was in effect, not just which model.
  • An unpinned model revision can change your sampling parameters, because generation_config.json lives in the repo.

To opt out and get vLLM's own defaults:

llm = LLM(model=MODEL, generation_config="vllm")     # offline
vllm serve MODEL --generation-config vllm            # online

Set SamplingParams explicitly for anything you'll compare against anything else. Then the question doesn't arise.

What generate() returns

outputs = llm.generate(prompts, sampling_params)     # list[RequestOutput], one per prompt

for out in outputs:
    out.prompt                    # the prompt string as submitted
    out.outputs                   # list[CompletionOutput] — length n from SamplingParams
    out.outputs[0].text           # the generated text
    out.outputs[0].token_ids      # the generated token IDs

Two things worth noting. outputs is a list of RequestOutput, one per prompt — and out.outputs is a second list, one entry per sample when n > 1 (Sampling Parameters). The double nesting catches everyone once.

And you pass a list of prompts, not one. That's not a convenience wrapper around a loop: all of them go into the engine's waiting queue together, and the scheduler batches them exactly as it would batch concurrent HTTP requests. Everything from Continuous Batching applies unchanged — which is why llm.generate(prompts) is dramatically faster than looping over llm.generate(p).

Why offline is the honest way to benchmark the engine

For measuring engine behaviour specifically, offline removes confounders that online adds:

Removed Why it matters
HTTP, JSON serialisation, sockets You're timing the engine, not your network stack
Client-side concurrency limits The openai client and your thread pool have their own ceilings
API server CPU contention From Lifecycle, P0 can bottleneck before the GPU does

The trade-off is that it also removes things you do want to measure for a real deployment — queueing behaviour under arrival patterns, streaming latency, per-request TTFT under contention. So: offline for engine capacity, online for user-facing latency. Using the wrong one is how benchmarks end up unrepresentative.

Try It

Hardware: Colab T4 or any GPU with compute capability ≥ 7.5.

Experiment 1 — see the chat template difference

This is the whole page in twenty lines.

# template_matters.py — same question, two paths, different quality.
from vllm import LLM, SamplingParams

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
llm = LLM(model=MODEL, max_model_len=2048)
params = SamplingParams(temperature=0.0, max_tokens=120)   # greedy, so differences are structural

question = "In one sentence, what is a KV cache?"

# A: raw generate — NO chat template applied
raw = llm.generate([question], params)[0].outputs[0].text

# B: llm.chat — template applied for you
chat = llm.chat([[{"role": "user", "content": question}]], params)[0].outputs[0].text

print("=== generate() — no template ===")
print(raw)
print("\n=== chat() — template applied ===")
print(chat)

What you should observe: the chat() output answers the question in the register you expect. The generate() output is more likely to continue the text, restate it, produce a list of similar questions, or drift — because the model is being asked to complete a string rather than respond to a turn.

Now change one thing: print the rendered template alongside, using the tokeniser snippet from Lifecycle of a Request. The difference between what you sent and what the model saw is the entire explanation.

Observation What it proves
chat() output is better-formed at temperature=0 The difference is structural, not sampling noise
generate() output continues rather than answers The model is outside its tuned format
Neither errors This failure is silent — nothing will warn you

Experiment 2 — whose defaults are you using?

# whose_defaults.py — the model author's, or vLLM's?
from vllm import LLM, SamplingParams

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
prompt = "Write one sentence about memory."

a = LLM(model=MODEL, max_model_len=2048)                                # model's config
b = LLM(model=MODEL, max_model_len=2048, generation_config="vllm")      # vLLM's own defaults

for label, llm in (("model's generation_config", a), ("vLLM defaults", b)):
    outs = {llm.generate([prompt])[0].outputs[0].text for _ in range(5)}
    print(f"{label:<28} {len(outs)} distinct outputs from 5 runs")

What you should observe: the two configurations behave differently — most visibly in output variety, since they're very likely running at different temperatures. If they're identical, this model ships no generation_config.json, which is itself worth knowing.

Then look at the file directly:

curl -s https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct/raw/main/generation_config.json

Whatever sampling values are in there are what you get when you pass no SamplingParams. That is the actual default for your deployment — not anything published in a guide.

Experiment 3 — batching is automatic, and it's the whole point

# batch_vs_loop.py — one call with N prompts, versus N calls with one prompt.
import time
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", max_model_len=2048)
params = SamplingParams(temperature=0.0, max_tokens=64)
prompts = [f"Write one sentence about the number {i}." for i in range(32)]

t0 = time.perf_counter(); llm.generate(prompts, params); batched = time.perf_counter() - t0
t0 = time.perf_counter()
for p in prompts: llm.generate([p], params)
looped = time.perf_counter() - t0

print(f"one call, 32 prompts : {batched:6.2f}s")
print(f"32 calls, 1 prompt   : {looped:6.2f}s   ({looped/batched:.1f}x slower)")

What you should observe: the loop is several times slower, for identical work. Passing a list puts every prompt in the waiting queue at once so the scheduler can batch them; looping serialises them at concurrency 1. This is Stage 0's opening argument reproduced in four lines, and it's the most common way people accidentally benchmark batch size 1.

Dial It In

Knob What it does Guidance
generation_config="vllm" Ignores the model's generation_config.json Use for reproducibility and cross-model comparison
SamplingParams(...) explicitly Overrides both Always, for anything you'll compare
llm.chat() vs llm.generate() Template applied or not chat() for instruct models; generate() for base models and throughput measurement
max_model_len Per-sequence cap, same as the server Your measured p99 — it divides your concurrency (Stage 0)
gpu_memory_utilization Size of the block pool 0.9 default; lower only when sharing the GPU
VLLM_USE_MODELSCOPE=True Fetch models from ModelScope rather than the Hub Where Hub access is restricted
--attention-backend Force a specific attention kernel Diagnosing performance, or when the auto-selected backend misbehaves

The LLM constructor takes the same engine arguments as vllm serve. Everything you learned in Stages 0–2 about max_model_len, gpu_memory_utilization, enable_prefix_caching, quantisation and speculative decoding applies here identically — same engine, different entry point.

Where It Bites You

Using generate() with an instruct model and no template. The highest-frequency mistake on this page. Output quality drops, nothing errors, and the cause is invisible unless you know to look. Use llm.chat(), or apply the template yourself.

Assuming your sampling defaults are vLLM's. They're the model author's, from generation_config.json. Change model and your defaults change silently; leave the revision unpinned and they can change under you.

Looping over prompts instead of passing a list. Serialises everything to concurrency 1 and throws away the entire benefit of the engine. If your offline job feels slow, check this first.

Comparing offline LLM numbers with online server numbers. They measure different things. Offline excludes HTTP, serialisation and API-server CPU; online includes queueing and streaming. Both are valid; neither is a substitute.

Forgetting the double nesting. outputs[i].outputs[0].text — the outer list is prompts, the inner is samples. outputs[i].text doesn't exist and the error isn't especially helpful.

Creating a new LLM per batch. Every instantiation reloads weights, re-profiles memory and re-captures CUDA graphs — tens of seconds to minutes. Create one and reuse it. In a script that processes files in a loop, this is an easy accidental order of magnitude.

Expecting generate() to stream. It returns when everything is finished. For token-by-token output you want the server, or the async engine — Streaming & Client Patterns.

In Production

Offline is for jobs, not for users. Evaluation suites, bulk classification, synthetic data generation, nightly re-embedding — anything with a list of inputs and no one waiting. It's simpler than a server: no networking, no timeouts, no cancellation, no availability requirement.

Pin generation_config behaviour explicitly in evaluation harnesses. An eval that inherits the model's recommended sampling parameters is measuring the model and its author's recommendations together, which makes cross-model comparison unfair. Either set SamplingParams explicitly or use generation_config="vllm", and record which you chose alongside the results.

Use it to size your server before you build one. An offline run tells you the engine's ceiling on your hardware with your model — the GPU-blocks number, the achievable throughput, whether a quantised variant fits. That's the input to the capacity plan, and it's much cheaper to obtain than a load test.

What changes at 10× traffic. Nothing here — offline batch jobs scale by running more of them, or larger ones, and they have no latency SLO to violate. The relevant scaling question is whether your batch job and your online server are competing for the same GPUs, which is a scheduling problem rather than an engine one. If they are, run the batch work on separate capacity or off-peak.

Check Yourself

Recall the idea

What's the difference between LLM and vllm serve?

Same engine, different entry point. LLM is a Python object you pass a list of prompts to; vllm serve wraps the same engine in an OpenAI-compatible HTTP server with request lifecycle management. Every engine argument applies to both.

Does llm.generate() apply the chat template?

No. This is documented and it's the most common source of unexpectedly poor output from instruct models. Use llm.chat(), or apply the template yourself with the tokeniser before calling generate().

Where do your sampling defaults come from if you pass no SamplingParams?

The model's generation_config.json from its Hugging Face repository, which vLLM applies by default. Not vLLM's own defaults — those require generation_config="vllm".

Why does passing a list of prompts matter?

They all enter the engine's waiting queue together, so the scheduler batches them. Looping over single-prompt calls serialises the work at concurrency 1 and forfeits the engine's main advantage.

Explain the mechanics

Your instruct model gives poor answers offline but good answers through the server. Explain.

The server's /v1/chat/completions endpoint applies the chat template; llm.generate() doesn't. The model is receiving a bare string rather than the role-marked format it was fine-tuned on, so it continues text rather than responding to a turn. Same weights, same sampling parameters, different input format.

Why can two people get different outputs from the same model, prompt and vLLM version?

Several ways, but the one specific to this page is generation_config.json: unless both set SamplingParams explicitly or pass generation_config="vllm", they inherit the model author's recommended parameters — which can differ if their model revisions differ. Add the batch-composition non-determinism from Sampling Parameters and exact reproduction needs more pinning than people expect.

Explain the double nesting in outputs[0].outputs[0].text.

The outer list has one RequestOutput per prompt you submitted. The inner list has one CompletionOutput per sample, so its length is n from SamplingParams. With one prompt and n=1 you need both indices, which looks redundant until you use either feature.

When is offline benchmarking the right choice, and when is it misleading?

Right for engine capacity: it removes HTTP, serialisation and API-server CPU, so you measure the engine. Misleading for user-facing latency, because it has no arrival pattern, no queueing dynamics and no streaming — so TTFT and ITL under contention, the numbers users actually feel, aren't represented.

Reason about a trade-off

You're building an evaluation harness. Offline or online, and what do you pin?

Offline — you have a fixed list of inputs, no latency requirement, and you want the simplest thing that reproduces. Pin the model revision, the vLLM version, and the sampling parameters explicitly rather than inheriting generation_config.json, so results are comparable across models and over time. Decide deliberately whether to apply the chat template: with it, you're evaluating the model as deployed; without it, you're evaluating raw completion. Both are legitimate, but the choice must be recorded, because it changes the numbers substantially.

A colleague's offline benchmark shows 3× the server's throughput. Is the server broken?

Probably not — they're measuring different things. Offline excludes HTTP, JSON serialisation, the API server's tokenisation and detokenisation CPU work, and any client-side concurrency ceiling. From Lifecycle, P0 can bottleneck before the GPU does, so a 3× gap may be entirely CPU-side. Worth checking: core count against the 2 + N process requirement, and whether --api-server-count helps. If the gap persists with ample CPU, then it's worth investigating.

Should an offline job set generation_config="vllm"?

If you're comparing across models or over time, yes — otherwise each model's own recommendations become an uncontrolled variable. If you're generating output for production use, no: the author's recommended parameters are usually well-chosen for that model, and matching production behaviour matters more than cross-model comparability. The general rule is that measurement wants generation_config="vllm" and production wants the model's own — and either way, setting SamplingParams explicitly makes the question moot.

Your offline job takes six hours. Where do you look first?

Whether you're passing a list or looping — that alone can account for most of it. Then whether you're constructing LLM more than once, since each instantiation reloads weights and re-captures CUDA graphs. Then max_model_len, which caps concurrency for the whole run (Stage 0). Only after those three would I look at quantisation or a bigger GPU — the first two are free and are usually the answer.

Cheat Sheet

The minimum viable script

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", max_model_len=4096)
params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)

# instruct model → chat() applies the template
outs = llm.chat([[{"role": "user", "content": "Explain paged memory."}]], params)
print(outs[0].outputs[0].text)

# base model / throughput measurement → generate() sends the string raw
outs = llm.generate(["The capital of France is"], params)

The two surprises

Surprise Consequence Fix
generate() does not apply the chat template Instruct models silently produce worse output llm.chat(), or apply it yourself
Defaults come from the model's generation_config.json "Default" behaviour changes with the model SamplingParams explicitly, or generation_config="vllm"

Output shape

outputs[i]              # RequestOutput   — one per PROMPT
outputs[i].outputs[j]   # CompletionOutput — one per SAMPLE (n)
outputs[i].outputs[0].text

Offline vs online

Use offline for Use online for
Engine capacity and throughput User-facing TTFT and ITL
Evaluation suites, bulk jobs Anything with a client waiting
Learning engine behaviour without networking Queueing, streaming, cancellation

The performance rules

  1. Pass a list, never loop. Looping is concurrency 1.
  2. Create LLM once and reuse it. Construction reloads weights and re-captures CUDA graphs.
  3. Set SamplingParams explicitly for anything you'll compare.

Sources

← Previous: Installing vLLM · Next: The OpenAI-Compatible Server →

⚠️ Verification checklist (delete before publishing)

Verified against vLLM's quickstart this session

  • llm.generate does not apply the chat template — stated explicitly in the docs, with llm.chat and the manual apply_chat_template route given as the alternatives.
  • generation_config.json from the model repo is applied by default, overriding vLLM's own sampling defaults; generation_config="vllm" (offline) and --generation-config vllm (serve) opt out.
  • LLM / SamplingParams API shape, llm.chat(messages_list, sampling_params), and the RequestOutput → outputs[j].text nesting.
  • Python 3.10–3.13 supported. Stage 0 says "Python 3.10+" — now precise, worth aligning.
  • VLLM_USE_MODELSCOPE, --attention-backend (FLASH_ATTN / FLASHINFER on CUDA).

Propagation required

  • Sampling Parameters presents its temperature/top_p recommendations without mentioning that vLLM applies the model's generation_config.json by default. That page's "Dial It In" table implies you're starting from vLLM's defaults. Add the generation_config caveat there — it materially changes what "default" means.
  • The landing page and Stage 0 describe vLLM as Linux-first. The quickstart now documents vLLM-Metal for Apple Silicon (MLX backend, mlx-community models) as a supported path. Decide whether to mention it; at minimum the flat "Linux-first" claim needs softening.
  • Stage 0 page 3 says Python 3.10+; make it 3.10–3.13.

Needs verifying

  • Confirm llm.chat() accepts a list-of-conversations and returns outputs in the same order.
  • Confirm generation_config="vllm" is the exact spelling for the LLM constructor.
  • Confirm whether Qwen2.5-0.5B-Instruct actually ships a generation_config.json — if it doesn't, Experiment 2 shows no difference and needs a different model.

Code

  • Run template_matters.py and paste both outputs into the page. Currently the difference is described rather than shown, and it's the page's central claim.
  • Run whose_defaults.py; if both configurations give identical output, pick a model that does ship a generation_config.json.
  • Run batch_vs_loop.py and capture the real ratio.
  • whose_defaults.py constructs two LLM objects in one process — confirm that works on a T4 without exhausting memory, since each pre-allocates a KV cache pool. It may need gpu_memory_utilization=0.4 on each, or two separate runs.

Rendering

  • No diagram yet. Consider one contrasting the offline and online paths into the same engine — it would pair with the Lifecycle page's pipeline image.
  • All relative links resolve once target files exist.