Back to Articles/vLLM & LLM Inference Serving

05 · Sampling Parameters

28 min read

Most guides file these under "API reference" — a list of fields with one-line descriptions of what they do to your text. This page is in Core Concepts instead, because that framing hides the half that matters once you're operating a server:

Every sampling parameter has a text effect, which is documented everywhere, and a systems effect, which is documented almost nowhere.

max_tokens isn't a formatting preference; it's how long a request may hold a slot. n isn't "give me options"; it's a question about copy-on-write. temperature=0 isn't determinism, whatever you've been told. You can't reason about a batch until you know which requests are cheap and which are expensive, and these parameters are what decides that.

The Problem

Symptoms, all of which come back to sampling parameters set by someone who was only thinking about the text:

Your concurrency collapsed and nothing about the deployment changed. A client started sending max_tokens: 4096 "to be safe."
Output is repetitive, or it's incoherent, and you have five knobs that all plausibly relate. You change two at once, it gets worse, and you have no model of why.
You set temperature=0 for reproducibility and the outputs still vary run to run. Not wildly — but enough to break a test suite, and it looks like a bug in the server.
Requests that never end. No stop sequence configured, so every response runs to max_tokens regardless of finishing at token 40, holding a slot for the other 4,056.
A user requests n=8 and you brace for 8× the cost. It isn't 8×. It also isn't 1×, and you have no idea how to price it.
You copied a top_p/temperature pair from another model's documentation and the outputs are subtly worse in a way nobody can pin down.

The unifying observation: the API surface presents these as text-quality settings, so they get chosen by whoever cares about text quality — and their cost lands on whoever operates the server. On a single-user script that separation is harmless. On a shared server it is the difference between 200 concurrent users and 40.

The Idea

The model does not produce text. At every step it produces a probability distribution over the entire vocabulary — for Qwen2.5, that's roughly 152,000 numbers, one per possible next token.

Something has to collapse those 152,000 numbers into exactly one token. Sampling parameters are that policy, and nothing more.

Picture an advisor who, at every word, hands you a ranked list of every word in the language with a confidence score beside each:

the    0.41
a      0.22
this   0.11
that   0.06
       ... 151,996 more, most of them essentially zero

Your options:

Always take the top one. Predictable, and repetitive — greedy decoding, temperature=0.
Sample proportionally to confidence. Natural variety, but the long tail of near-zero options means occasionally you'll pick something absurd.
Change how strongly you trust the confidences before sampling — sharpen them so the leader dominates, or flatten them so outsiders get a chance. That's temperature.
Refuse to consider anything outside the top few — the top 40 suggestions (top_k), or as many as it takes to cover 90% of the confidence (top_p).
Decide when to stop asking — after N words (max_tokens), or when they say something specific (stop).

That's the whole conceptual surface. Two families: reshape the distribution (temperature, penalties) and truncate it (top_k, top_p, min_p). Then sample from what's left.

The systems half of the idea is equally simple and much less discussed:

The parameters that decide when a request ends are the ones that decide what it costs. Everything else changes which token you get. max_tokens, stop and n change how long a slot is held and how many KV blocks are consumed.

Under the Hood

The pipeline

Every decode step, for every sequence in the batch:

model forward pass
      │
      ▼
   LOGITS            one raw score per vocabulary token (~152,000 of them)
      │
      ▼
   PENALTIES         presence / frequency / repetition — adjust scores of tokens
      │              already seen in this sequence
      ▼
   TEMPERATURE       logits ÷ T.   T<1 sharpens, T>1 flattens, T=0 → take the argmax
      │
      ▼
   TRUNCATION        top_k: keep the k highest
      │              top_p: keep the smallest set whose probabilities sum to p
      │              min_p: drop anything below (min_p × the top token's probability)
      ▼
   SOFTMAX           what remains becomes a probability distribution
      │
      ▼
   SAMPLE            draw one token   →   appended to the sequence, its KV cached

Order matters and is a common source of confusion. Temperature is applied before truncation, so top_p=0.9 at temperature=2.0 selects from a very different candidate set than the same top_p at temperature=0.5 — the flattening happens first, and then you take the nucleus of the already-flat distribution. Tuning them independently, as though they compose cleanly, is why people end up with pairs of values nobody can justify. ⚠️ Verify the exact order for your vLLM version; implementations differ across engines.

The two families

Reshaping — temperature. Divide every logit by T.

`T`	Effect	Use
`0.0`	Argmax — always the top token. Sampling is bypassed entirely	Extraction, classification, code, anything graded against a reference
`0.1–0.5`	Sharpened; the leader usually wins	Factual answers, RAG, tool calls
`~0.7–1.0`	Roughly the model's own distribution	General chat
`> 1.2`	Flattened; unlikely tokens get real probability	Creative work, and the region where incoherence starts

Truncating — top_k, top_p, min_p. All three delete candidates before sampling; they differ in how they choose the cut.

Parameter	Rule	Behaviour
`top_k`	Keep the k highest-probability tokens	Fixed count. Same width whether the model is certain or clueless — which is the weakness
`top_p` (nucleus)	Keep the smallest set summing to ≥ `p`	Adaptive. Narrow when the model is confident, wide when it isn't. The usual choice
`min_p`	Keep tokens with probability ≥ `min_p` × (top token's probability)	Adaptive, relative to the leader. Newer; robust at high temperature

Set several and you get the intersection — every filter applies. top_k=50 with top_p=0.9 means "at most 50 candidates, and also only the 90% nucleus."

Penalties operate on tokens already generated in this sequence:

Parameter	Mechanism	Range
`presence_penalty`	Flat penalty if a token has appeared at all	Encourages new topics
`frequency_penalty`	Penalty proportional to how often it has appeared	Suppresses repetition of common tokens
`repetition_penalty`	Multiplicative on the logit; a different formulation of the same idea	> 1 discourages repetition

These are blunt instruments. They cannot distinguish "the model is stuck in a loop" from "this is JSON and " legitimately appears forty times" — which is the failure mode in Where It Bites You.

The systems effects — the part that isn't in the API docs

This is the table that makes the page worth its place in Core Concepts.

Parameter	Text effect	Systems effect
`max_tokens`	Caps output length	The worst case the scheduler must accommodate. Bounds KV blocks and slot-hold time. The single most expensive parameter a client can set carelessly
`stop` / `stop_token_ids`	Ends output at a string	Frees the slot and its blocks early. Often the biggest practical throughput lever available to a client
`ignore_eos`	Keeps generating past EOS	Guarantees every request runs to `max_tokens`. Useful for benchmarking, catastrophic in production
`min_tokens`	Forces a minimum length	Prevents early release of a slot — a floor on cost
`n`	n completions	Prompt blocks shared via copy-on-write (PagedAttention); only generated tokens cost n×. Sub-linear, and how sub-linear depends on your prompt-to-output ratio
`temperature=0`	Greedy decoding	Cheaper sampling step, negligible overall. Does not guarantee reproducibility — see below
`logprobs` / `prompt_logprobs`	Returns probabilities	Real serialisation and bandwidth cost; `prompt_logprobs` over a long prompt can dominate the response payload
`seed`	Reproducible sampling	Per-request RNG state; same caveat as `temperature=0`
`structured_outputs`	Constrains output to a grammar/schema	Constraint machinery per step; can restrict how sequences batch together

The n arithmetic is worth doing explicitly, because it's the one people misprice in both directions. A 2,000-token prompt with 100-token outputs at n=4:

naive assumption : 4 × (2000 + 100) = 8,400 token-slots
actual with COW  : 2000 + (4 × 100) = 2,400 token-slots     ← 3.5× cheaper than assumed

Flip it to a 50-token prompt with 1,000-token outputs and sharing buys you almost nothing. The saving is a function of your prompt-to-output ratio, which is exactly the workload shape you measured in Stage 0.

Why `temperature=0` isn't deterministic

The most useful thing on this page for anyone building an eval suite.

Greedy decoding is deterministic given identical logits. But logits come from floating-point reductions on a GPU, and floating-point addition is not associative — (a+b)+c can differ from a+(b+c) in the last bits. The order of those reductions depends on kernel selection and batch shape, and under continuous batching your request's batch composition depends on what other traffic is present at that instant.

So the same prompt, same seed, same temperature=0, on the same server, can produce different logits in the last few bits. Usually that changes nothing. Occasionally two candidate tokens are close enough that the argmax flips — and from there the sequences diverge completely.

The practical consequences:

Don't assert exact string equality in tests against a batching server.
Reproducibility needs batch-invariance, not just a seed — that's a property of the engine's kernels and scheduling, not of your request.
This is not a vLLM bug. It's inherent to batched GPU inference, and every engine that batches has it.

⚠️ Some engines offer batch-invariant or deterministic modes, usually at a throughput cost. Check whether your version has one before building a workflow that needs it.

Try It

Hardware: Colab T4 or any CUDA GPU.

vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 2048

Experiment 1 — watch the distribution, not the text

Reading about temperature is much less useful than seeing what it does to the candidate set.

# distribution.py — the actual probabilities the sampler is choosing from.
import math
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL  = "Qwen/Qwen2.5-0.5B-Instruct"
PROMPT = "The capital city of France is"

for temp in [0.0, 0.7, 1.5]:
    r = client.completions.create(
        model=MODEL, prompt=PROMPT, max_tokens=1,
        temperature=temp, logprobs=5,          # top 5 candidates for the next token
    )   # NOTE: needs the server started with --logprobs-mode processed_logprobs — see below
    top = r.choices[0].logprobs.top_logprobs[0]
    print(f"\ntemperature = {temp}")
    for tok, lp in sorted(top.items(), key=lambda kv: -kv[1]):
        print(f"   {tok!r:<12} p = {math.exp(lp):.4f}")

⚠️ Start the server with --logprobs-mode processed_logprobs for this experiment. By default V1 returns raw_logprobs — values taken from the model's raw output before any logit post-processing, including temperature scaling and top-p/top-k. With the default you will see the same numbers at every temperature, because you're looking at the distribution before temperature touched it. The four modes are raw_logprobs (default), processed_logprobs, raw_logits and processed_logits.

This is worth doing deliberately: run it once in each mode. Seeing raw logprobs refuse to move while the sampled text clearly changes is the most direct demonstration available that logprobs and sampling are separate stages.

What you should observe (with processed_logprobs): at temperature=0 the top candidate dominates; as temperature rises the probabilities flatten and the gap between first and fifth narrows. You're watching the sharpening and flattening directly, rather than inferring it from prose.

Now change one thing: add top_p=0.5 and re-run at temperature=1.5. The candidates below the cumulative-50% cut disappear from consideration entirely — flattened first by temperature, then truncated. That's the ordering dependency, visible.

Experiment 2 — what `n` actually costs

This is the systems experiment, and it tests the copy-on-write claim from PagedAttention.

# cost_of_n.py — is n=8 eight times the cost?
import time
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-0.5B-Instruct", max_model_len=2048)

LONG_PROMPT  = "Summarise the following text.\n\n" + ("The system allocates memory in blocks. " * 100)
SHORT_PROMPT = "Write a haiku."

for label, prompt in [("long prompt", LONG_PROMPT), ("short prompt", SHORT_PROMPT)]:
    print(f"\n{label}")
    for n in [1, 2, 4, 8]:
        params = SamplingParams(n=n, max_tokens=128, temperature=0.8)
        t0 = time.perf_counter()
        out = llm.generate([prompt], params)
        dt = time.perf_counter() - t0
        toks = sum(len(o.token_ids) for o in out[0].outputs)
        print(f"  n={n}: {dt:>5.2f}s  {toks:>4} output tokens  {dt/ (dt if n==1 else 1):>4}")

What you should observe:

Observation	What it proves
For the long prompt, going from `n=1` to `n=8` costs far less than 8×	Prompt blocks are shared by reference — copy-on-write, working
For the short prompt, the cost is much closer to linear in `n`	There's barely any prompt to share; the generated tokens dominate
Output tokens scale exactly with `n` in both cases	Sharing applies to the prompt only. Divergent generation always costs full price

The takeaway you can act on: n is cheap when prompts are long relative to outputs and expensive when they're not. That's a pricing rule you can defend, derived rather than guessed.

Experiment 3 — the determinism check (two minutes, worth it)

# determinism.py — is temperature=0 reproducible?
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

PROMPT = "List three causes of memory fragmentation, briefly."
outs = {client.completions.create(
            model="Qwen/Qwen2.5-0.5B-Instruct", prompt=PROMPT,
            max_tokens=200, temperature=0.0, seed=42,
        ).choices[0].text for _ in range(20)}

print(f"distinct outputs from 20 identical greedy requests: {len(outs)}")

What you should observe on an idle server: very likely 1 — no other traffic, stable batch shapes, identical logits. Now run it again while mixed_lengths.py from the previous page hammers the server, and see whether it's still 1.

Whatever result you get, you've learned something worth knowing: either your setup is stable enough to rely on, or it isn't — and you found out in a test rather than in a failing CI pipeline.

Dial It In

⚠️ Before any of this: you may not be starting where you think. vLLM applies the model's generation_config.json from its Hugging Face repository by default, so the temperature, top_p and top_k in effect when you set nothing are the model author's recommendations — not vLLM's defaults and not the values below. Pass generation_config="vllm" (offline) or --generation-config vllm (serve) to opt out, or set SamplingParams explicitly and make the question moot. See Offline Batch Inference.

Starting points by task. Treat these as defaults to depart from, not truths.

Task	`temperature`	`top_p`	Notes
Extraction, classification, tool calls	`0.0`	—	Greedy. Truncation is irrelevant when you take the argmax
RAG / factual Q&A	`0.0–0.3`	`0.9`	Low variance; let the retrieved context dominate
General chat	`0.7`	`0.9`	The common default, and a reasonable one
Code generation	`0.0–0.2`	`0.95`	Correctness beats variety
Creative writing	`0.9–1.1`	`0.95`	Raise temperature before loosening `top_p`

The parameters that decide your cost:

Parameter	Sane start	Rule of thumb
`max_tokens`	Your measured p99 output length, not the model's maximum	The most important one on the page. Enforce a server-side ceiling; never let clients set it unbounded
`stop`	Always set them for structured output	The cheapest throughput win available — it returns slots early
`n`	`1`	Only raise it with a long shared prompt, where COW makes it cheap
`logprobs`	Off	Turn on for debugging and evaluation; it's payload, not compute
`seed`	Set it for evaluation	Necessary for reproducibility, and not sufficient — see determinism above

Change one at a time. Temperature and top_p interact through the pipeline order, so moving both at once produces results you can't attribute. The usual sequence: fix top_p at 0.9, tune temperature until the style is right, and only then touch truncation.

Where It Bites You

Letting clients set max_tokens freely. A client that sends 4,096 "to be safe" has reserved the scheduler's worst case for a response that will average 150 tokens. Multiply by every client and your concurrency ceiling is set by imagination rather than traffic. Enforce a server-side maximum, and default it to your measured p99.

Forgetting stop sequences. Without them, generation runs until max_tokens or EOS. Models that drift past a natural ending will happily produce 3,000 tokens of continuation you discard — paying full price in slot time and KV blocks for output nobody reads.

Believing temperature=0 gives reproducibility. It gives greedy decoding. Reproducibility across runs additionally needs batch-invariant kernels, which batching servers generally don't guarantee. Tests asserting exact string equality against a live server will flake, and the flake will be blamed on the model.

Setting temperature and top_p aggressively at once. They're not independent — temperature reshapes the distribution before top_p truncates it. High temperature with tight top_p is a narrow slice of an artificially flattened distribution, which is not what either setting suggests in isolation.

Repetition penalties on structured output. JSON repeats ", : and , constantly; code repeats indentation and keywords. A repetition penalty tuned to stop prose looping will quietly corrupt syntax, and the failure looks like the model being bad at JSON. Use structured_outputs for structure; leave the penalties alone.

Assuming "no parameters set" means vLLM's defaults. It means the model author's, via generation_config.json. Change model and your effective sampling changes with no code change; leave the revision unpinned and it can change under you. This is also why two people running "the same" model with "no parameters" can get different behaviour.

Copying parameters between models. top_p=0.95 means "the 95% nucleus" of this model's distribution. Different models — and different fine-tunes of the same base — have differently shaped distributions, so the same numbers select differently-sized candidate sets. A value tuned elsewhere is a starting point, not a setting.

ignore_eos escaping from a benchmark. It's there so benchmarks can produce exactly N tokens. Left on in production, every request runs to max_tokens regardless of finishing, which is close to the worst possible configuration.

Assuming best_of still exists. It generated several candidates and returned the best one, and it appears throughout older vLLM and OpenAI-style examples. It was removed in V1 as a little-used feature (RFC #13361), and it is absent from the current SamplingParams. Porting older code that sets it will fail rather than silently degrade — which is the better outcome, but worth knowing before you plan around it.

Ignoring prompt_logprobs payload size. It returns probabilities for every prompt token. On a 4,000-token prompt with several alternatives each, the response can dwarf the generated text and cause client-side timeouts that look like server problems.

In Production

Sampling parameters are an API contract, so treat them like one. Decide which fields clients may set, which you pin server-side, and which you clamp. The usual split: clients own temperature and top_p; you own the ceiling on max_tokens and n; nobody in production gets ignore_eos. Enforce it at the gateway rather than trusting well-behaved callers — Security Posture.

max_tokens is your cost control, and it's the only hard one you have. Everything else affects quality; this affects the bill and the capacity plan directly. A per-tenant ceiling is the straightforward mechanism, and it's far better than discovering the limit as preemption.

Monitor the output-length distribution, not just the mean. It's the input to every capacity calculation in The KV Cache and the source of the variance that Continuous Batching depends on. A shift in it — a client changing their prompt template, a new use case arriving — changes your capacity without changing your configuration, and nothing else will alert you to it.

Pin parameters per endpoint rather than per request. /summarise and /chat want different settings, and encoding that server-side gives you one place to change them, a testable surface, and protection from a client shipping temperature=2.0 on a Friday. It also means your evaluation suite tests the same configuration production runs.

Version your sampling configuration alongside your model. A change to default temperature changes output quality as surely as a model swap, and it will be invisible in your model registry. Both belong in the same rollout and the same rollback — Shipping a Model Version.

What changes at 10× traffic. Nothing about the parameters themselves — but the carelessly set ones stop being free. At low load an over-generous max_tokens costs nothing because slots are plentiful. At 10× it directly reduces how many users you serve, because the scheduler is now rationing exactly the resource that parameter reserves.

Check Yourself

Recall the idea

What do sampling parameters actually operate on?

A probability distribution over the entire vocabulary — around 152,000 numbers for Qwen2.5 — produced at every decode step. They're the policy for collapsing that distribution into one token, and they split into two families: reshaping it (temperature, penalties) and truncating it (top_k, top_p, min_p).

Explain temperature without maths.

How much you trust the model's confidence ordering. Below 1 sharpens the distribution so the leading token dominates; above 1 flattens it so unlikely tokens get a real chance; at 0 you skip sampling entirely and take the most likely token every time.

Difference between top_k and top_p?

top_k keeps a fixed number of candidates regardless of how confident the model is. top_p keeps the smallest set whose probabilities sum to p, so it adapts — narrow when the model is confident, wide when it's uncertain. That adaptivity is why top_p is usually preferred.

Which parameters affect cost rather than text?

max_tokens (bounds the KV blocks and slot time the scheduler must plan for), stop and stop_token_ids (release the slot early), min_tokens and ignore_eos (prevent early release), and n (multiplies generated tokens while sharing the prompt). logprobs and prompt_logprobs cost payload rather than compute.

Explain the mechanics

Walk the pipeline from logits to a token.

Logits out of the forward pass → penalties adjust scores of already-seen tokens → temperature divides the logits, sharpening or flattening → truncation drops candidates via top_k, top_p or min_p → softmax renormalises the survivors → one token is sampled and appended, its KV cached.

Why don't temperature and top_p compose independently?

Temperature is applied first, so it changes the distribution that top_p then measures. At high temperature the distribution is flatter, so the 90% nucleus contains many more tokens than it would at low temperature. The same top_p value therefore selects different-sized candidate sets depending on temperature — which is why tuning both simultaneously produces results you can't attribute.

Why is temperature=0 not reproducible on a batching server?

Greedy decoding is deterministic given identical logits, but logits come from floating-point reductions whose order depends on kernel choice and batch shape. Floating-point addition isn't associative, so different batch compositions can produce last-bit differences. Under continuous batching your batch composition depends on concurrent traffic — so when two candidates are nearly tied, the argmax can flip and the sequences diverge from there. It's inherent to batched GPU inference, not a vLLM defect.

Compute the cost of n=4 with a 2,000-token prompt and 100-token outputs.

The prompt is stored once and shared by reference through copy-on-write, so it costs 2,000 token-slots regardless of n. Each completion's generated tokens are unique: 4 × 100 = 400. Total 2,400 rather than the naive 8,400 — about 3.5× cheaper than assumed. With a 50-token prompt and 1,000-token outputs the saving nearly vanishes, because there's almost nothing to share.

Reason about a trade-off

A client wants max_tokens=8192 by default "so responses are never truncated." Respond.

Establish the real distribution first: if p99 output is 400 tokens, the setting reserves twenty times the worst case for essentially no requests. Then make the cost concrete — it's the scheduler's worst-case planning bound, so it directly reduces how many users fit. The constructive answer is a ceiling near p99 plus headroom, a clear error when a response hits it, and an explicit opt-in for the rare long-form endpoint. What you're refusing is unbounded defaults, not long responses.

Your eval suite fails intermittently on exact-match assertions at temperature=0. Diagnose and fix.

It's almost certainly batch-composition non-determinism, not a model or server bug. Confirm by running the eval against an idle server and then under concurrent load — if it's stable alone and flaky under load, that's diagnostic. Fixes, in order of preference: stop asserting exact strings and assert on semantics or a similarity threshold; run evals against a dedicated instance with controlled concurrency; or use a batch-invariant mode if your engine offers one, accepting the throughput cost. Pinning the seed alone will not fix it.

When is raising n a good deal, and when is it a trap?

Good when the prompt is long relative to the outputs — a 2,000-token prompt with 100-token completions shares almost all its memory, so extra samples are close to free, which makes it genuinely attractive for reranking or best-of-N selection. A trap when outputs are long relative to the prompt, where sharing buys nothing and cost is essentially linear. The deciding number is your prompt-to-output ratio, and you already measured it.

A team reports the model "got worse" after a deploy with no model change. Where do you look?

Sampling configuration, first. Defaults may have shifted, a client may have started sending its own parameters, a chat template change may have altered stop tokens, or structured_outputs may have been added or removed. All of these change output quality as surely as a model swap and none appear in a model registry. The systemic fix is versioning sampling configuration alongside the model so this is visible in a diff rather than discovered by users.

Cheat Sheet

The pipeline

logits → penalties → temperature → truncation (top_k / top_p / min_p) → softmax → sample
                     └─ applied BEFORE truncation; they do not compose independently ─┘

Text knobs

Parameter	Does	Typical
`temperature`	Sharpen (<1) or flatten (>1); 0 = greedy	`0.0` factual · `0.7` chat · `1.0+` creative
`top_p`	Keep the smallest set summing to `p`. Adaptive	`0.9–0.95`
`top_k`	Keep the `k` highest. Fixed width	`40`, or unset
`min_p`	Keep ≥ `min_p` × top token's probability	`0.05`, robust at high temperature
`presence_penalty`	Flat penalty for any repeat	`0.0–0.5`
`frequency_penalty`	Penalty scaled by repeat count	`0.0–0.5`

Cost knobs — the ones this page exists for

Parameter	Systems effect
`max_tokens`	The scheduler's worst-case reservation. Your primary cost control
`stop`	Frees the slot and blocks early. Cheapest throughput win a client can give you
`n`	Prompt shared via copy-on-write; generated tokens cost `n`×. Sub-linear
`ignore_eos`	Forces every request to `max_tokens`. Benchmarks only
`min_tokens`	A floor on cost — prevents early slot release
`logprobs` / `prompt_logprobs`	Payload size, not compute. `prompt_logprobs` can be huge

The n arithmetic

cost ≈ prompt_tokens + (n × output_tokens)        not  n × (prompt + output)
→ cheap when prompts are long relative to outputs; near-linear when they aren't

Three things to remember

temperature=0 is greedy, not deterministic. Batch composition changes floating-point reduction order; near-ties can flip.
Temperature is applied before truncation. Tune one at a time.
max_tokens is a memory reservation, not a formatting preference. Cap it server-side.

Sources

vLLM SamplingParams API reference — docs.vllm.ai/en/stable/api/vllm/sampling_params.html
Holtzman et al., The Curious Case of Neural Text Degeneration — the nucleus (top_p) sampling paper — arxiv.org/abs/1904.09751

Cheat Sheet: Stage 1 complete

You can now compute what a model costs per token, predict how many users a GPU will hold, explain why throughput and per-user speed move in opposite directions, and read a request's parameters as a resource claim rather than a style preference.

Stage 2 — The Engine takes the next step: the component that has been implicit in all five pages — the scheduler actually making these decisions, step by step.

← Previous: Continuous Batching · Next: Stage 2 — The Engine →

⚠️ Verification checklist (delete before publishing)

Verified

The current SamplingParams field list was checked against the published API reference. It includes min_p, min_tokens, bad_words, allowed_token_ids, logit_bias, structured_outputs, repetition_detection, thinking_token_budget, prompt_logprobs — and does not include best_of, which is why the page flags it rather than documenting it.

Corrected after Stage 3

The page previously implied its recommended values were departures from vLLM's defaults. They aren't: vLLM applies the model's generation_config.json by default. A warning has been added to Dial It In and a new entry to Where It Bites You.

Needs verifying

The pipeline order (penalties → temperature → truncation → softmax → sample). This is the page's central mechanical claim and is asserted from general knowledge, not from reading vLLM's sampler. Confirm against the source for the pinned version; flagged inline but should be resolved rather than left flagged.
Confirm min_p's exact semantics (relative to the top token's probability).
Confirm repetition_penalty is multiplicative on logits while presence/frequency are additive — the page implies this distinction.
Confirmed removed in V1 — vLLM's V1 guide lists best_of under Removed Features, citing RFC #13361 ("removed due to limited usage"). The page now states this plainly.
repetition_detection and thinking_token_budget appear in the current API and are not covered on this page. Decide whether they belong here or in Stage 2.
Confirm Qwen2.5's vocabulary size — the page uses ~152,000 (config says 151,936 for the 0.5B and 152,064 for the 7B; the two differ, so the round number is fine but say "about").
Confirm whether vLLM currently offers a batch-invariant / deterministic mode, and name it if so. The page promises the reader to "check whether your version has one".

Code

Found a real bug in this experiment. V1 returns raw_logprobs by default — before temperature and top-p are applied — so the experiment as originally written could not show what it claimed. The page now requires --logprobs-mode processed_logprobs and turns the contrast into part of the lesson.
Run distribution.py in both modes; confirm logprobs=5 returns top_logprobs in the shape used, and capture output for each mode.
Run cost_of_n.py. The print statement's last column is malformed — dt/(dt if n==1 else 1) is nonsense left in by mistake. Replace with a proper ratio against the n=1 baseline before publishing.
Confirm out[0].outputs yields n completions and that len(o.token_ids) is the right accessor.
Run determinism.py idle and under load; record both results. If the idle case is not 1, the page's framing needs adjusting.

Rendering

One image generated and placed; row added to image-prompts.md.
The ASCII pipeline block renders correctly on the published site.
All relative links resolve once target files exist — this page links forward to Stage 2, 5 and 6, none of which exist yet.