05 · Speculative Decoding

Every other technique in this article makes the server better at serving many users. PagedAttention reclaims memory so more sequences fit; continuous batching keeps them all working; prefix caching skips repeated prefill. All of them are worth nothing when concurrency is one.

Speculative decoding is the exception. It is the only technique here that improves single-stream latency — the tokens-per-second one user sees when nobody else is on the server. That makes it the answer to a question the rest of the article can't address, and it comes with an economics that decides, from one number, whether it helps or actively hurts.

---

The Problem

You've read Prefill vs Decode, so you know why a single request is slow: decode reads every weight in the model to produce one token, and that read is memory-bandwidth bound. At batch size 1 your arithmetic intensity is about 1 against a ridge point in the hundreds — you're using a fraction of a percent of the GPU's compute.

The standard fix is batching. But:

• Your app has one user at a time. An internal tool, a desktop assistant, a coding agent. There is nothing to batch with.
• You bought a faster GPU and ITL barely moved, because you bought FLOPs and you were limited by bandwidth.
• Your agent makes twenty sequential LLM calls, each waiting on the last, so total latency is twenty decode loops end to end and no amount of concurrency helps a chain.
• You enabled speculative decoding because a blog post promised 2–3×, and throughput got worse.

That last one is the trap this page exists to prevent. The first three describe the gap it genuinely fills.

---

The Idea

Someone who finishes your sentences, and you check whether they were right.

You're writing by hand, slowly and carefully. An assistant beside you guesses the next several words and writes them in pencil. You glance at the whole guess at once — reading is fast, writing is slow — and keep the words they got right up to the first mistake. From there you write the next word yourself, and the assistant starts guessing again.

When they guess well, you gain several words for the price of one glance. When they guess badly, you lose almost nothing: you write that word yourself, exactly as you would have. The only true waste is the pencil time, which is cheap because the assistant is fast and sloppy rather than slow and careful.

The two properties that make this work:

• Checking many guesses costs the same as writing one word. That's the whole trick, and it's the same fact that makes prefill parallel: a transformer can evaluate many positions in one forward pass. Verification is a parallel operation; generation is a serial one.
• You never accept a wrong word. The output is what you would have written alone. Speculative decoding is not an approximation — done properly it is mathematically equivalent to sampling from the target model, which is why it can be enabled without changing your evaluations.

That second property is unusual in this article. Quantisation trades accuracy for memory; batching trades latency for throughput. This one is genuinely free — if the guesses are good enough.

---

Under the Hood

The draft-then-verify loop

1. DRAFT     a cheap proposer generates k candidate tokens, one at a time
             (fast, low quality — that's the point)
                                  │
                                  ▼
2. VERIFY    the target model evaluates all k candidates in ONE forward pass
             ← this is the trick: k positions checked for the price of one step
                                  │
                                  ▼
3. ACCEPT    keep the longest correct prefix, reject from the first mismatch,
             and emit one bonus token from the target's own distribution
                                  │
                                  └──► repeat

Because the verification uses the target model's own distribution to accept or reject, the tokens you emit are distributed exactly as if the target had generated them alone. Speculation changes how fast you get tokens, not which tokens you get.

Note the "bonus token": even when every draft token is rejected, the verification pass still produces one correct token. A completely wrong draft leaves you no worse off than plain decoding, apart from the drafting cost.

The three proposers

Method	What drafts	Cost	Where acceptance comes from
N-gram	No model at all — matches the recent context against earlier text and copies what followed	Effectively free	Output that repeats the input: summarisation, editing, code refactoring, RAG quoting
Draft model	A small model from the same family	Real, but small	General language modelling ability, at lower quality
EAGLE	A trained lightweight head that predicts the target's own next features	Small	Trained specifically against the target model, so acceptance is higher

Reported acceptance rates put standalone draft models around 40–60% and EAGLE-3 around 60–80% on in-distribution workloads. ⚠️ Secondary sources — verify before quoting.

The n-gram proposer deserves attention because it's the one people underrate. It has no model, no extra memory and no quality of its own — it simply notices that you're about to repeat something that appeared earlier in the context. On workloads where the output quotes the input heavily, that's a very good guess very cheaply. On creative generation it's useless.

The economics — one formula that decides everything

Let α be the per-token acceptance rate, k the number of tokens speculated per step, and c the draft's cost as a fraction of one target forward pass.

expected tokens per step  =  1 + α + α² + … + α^k      (accepted prefix, plus the bonus token)
cost per step             =  1 + k·c                    (one verify, plus k drafts)

speedup  =  (1 + α + α² + … + α^k) / (1 + k·c)

Working that through gives the table that should govern every decision on this page:

Speedup over plain decoding, c = 0.10 (a small draft model):

α ↓ / k →	1	2	3	4	6	8
0.2	1.09	1.03	0.96	0.89	0.78	0.69
0.4	1.27	1.30	1.25	1.18	1.04	0.93
0.6	1.45	1.63	1.67	1.65	1.52	1.37
0.8	1.64	2.03	2.27	2.40	2.47	2.40
0.9	1.73	2.26	2.65	2.93	3.26	3.40

Three readings, and the second is the one nobody tells you:

Low acceptance makes you slower. At α = 0.2 with k = 8 you get 0.69× — a 31% slowdown. You're paying for eight draft passes and accepting roughly one token.

More speculation is not better. The best k depends on α, and the bold entries above are the peaks. At α = 0.4 the optimum is k = 2; going to k = 8 takes you from 1.30× to 0.93×, turning a gain into a loss. Speculating further only pays if you're likely to get that far.

The break-even is a real threshold you can compute:

Draft cost c	k = 2	k = 4	k = 8
0.00 (n-gram)	any α	any α	any α
0.10	α > 0.17	α > 0.29	α > 0.44
0.20	α > 0.31	α > 0.46	α > 0.62
0.30	α > 0.42	α > 0.57	α > 0.72

With a free proposer, speculation can't lose. With a draft model that costs 30% of the target and k = 8, you need better than 72% acceptance just to break even — which is why an oversized draft model is a common and expensive mistake.

Why it stops working under load

The formula above assumes verification is free — that checking k tokens costs one forward pass. That holds when the GPU has spare compute, which at batch size 1 it overwhelmingly does.

Now recall from Prefill vs Decode that decode arithmetic intensity ≈ batch size. Speculating k tokens for each of B sequences makes the effective batch roughly B × (k+1). At B = 1 and k = 4 that's an effective batch of 5 — still far below the ridge point, still free. At B = 64 and k = 4 it's 320, which pushes you past the ridge point and into compute-bound territory.

Speculative decoding spends spare compute to buy latency. At low concurrency that compute is free, so it's close to pure gain. At high concurrency there is no spare compute — continuous batching is already using it — so speculation competes with real work and reduces throughput.

That's the mechanism behind "we enabled it and throughput got worse", and it's why the honest advice is a load-dependent decision rather than a global on/off.

---

Try It

Experiment 1 — model your own break-even (no GPU)

Before enabling anything, find out what acceptance rate you'd need.

# spec_economics.py — should speculative decoding help you at all?
def expected_tokens(alpha, k):
    return sum(alpha**i for i in range(k + 1))     # accepted prefix + bonus token

def speedup(alpha, k, c):
    return expected_tokens(alpha, k) / (1 + k * c)

C = 0.10        # draft cost as a fraction of one target forward pass
print(f"{'alpha':>7} " + " ".join(f"k={k:<6}" for k in (1, 2, 4, 8)))
for a in (0.2, 0.4, 0.6, 0.8, 0.9):
    print(f"{a:>7.1f} " + " ".join(f"{speedup(a,k,C):<7.2f}" for k in (1, 2, 4, 8)))

# Break-even acceptance rate, by bisection
print("\nbreak-even alpha:")
for k in (2, 4, 8):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if speedup(mid, k, C) < 1.0 else (lo, mid)
    print(f"  k={k}: alpha must exceed {lo:.3f}")

# DERIVED, not benchmarked — arithmetic from the speedup model above
  alpha k=1     k=2     k=4     k=8
    0.2 1.09    1.03    0.89    0.69
    0.4 1.27    1.30    1.18    0.93
    0.6 1.45    1.63    1.65    1.37
    0.8 1.64    2.03    2.40    2.40
    0.9 1.73    2.26    2.93    3.40

break-even alpha:
  k=2: alpha must exceed 0.171
  k=4: alpha must exceed 0.287
  k=8: alpha must exceed 0.445

Now change one thing: set C = 0.3, a draft model that's too large relative to the target. Watch the whole low-acceptance half of the table fall below 1.0, and the break-even at k = 8 climb past 0.72. The draft's cost matters as much as its accuracy — a more accurate but slower drafter can easily be a net loss.

Experiment 2 — acceptance rate is a property of your workload

Hardware: Colab T4. N-gram speculation needs no draft model, so it's the cheapest thing to test.

# Baseline
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096

# With n-gram speculation
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096 \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,
                         "prompt_lookup_min":3,"prompt_lookup_max":8}'

⚠️ The speculative_config surface has changed across versions. Confirm the exact flag spelling and JSON keys against vllm serve --help for your build before relying on this invocation.

Now measure ITL on two deliberately different workloads, using the harness from Prefill vs Decode:

Workload	Prompt	Expected acceptance
Copy-heavy	"Repeat the following text exactly, then summarise it: <800 tokens>"	High — the output quotes the input, which is exactly what n-gram matching finds
Creative	"Write an original poem about the sea."	Near zero — nothing in the context predicts the output

Observation	What it proves
ITL improves substantially on the copy-heavy prompt	N-gram speculation works when output echoes input
ITL is unchanged or slightly worse on the creative prompt	Acceptance is a property of the workload, not the config
The same server gives both results	There is no global "is speculation good" answer

Then change the one thing that matters most

Repeat the copy-heavy measurement at concurrency 1 and at concurrency 32.

What you should observe: the gain shrinks — and may invert — as concurrency rises. At batch 1 there's spare compute to spend on verification; at batch 32 there isn't, and speculation is competing with real requests. That single contrast is the page's central claim, and it's the one worth measuring on your own hardware before enabling anything in production.

---

Dial It In

Knob	What it does	Guidance
method	ngram, a draft model, or eagle	Start with ngram — no extra model, no memory, and it either helps or doesn't
num_speculative_tokens (k)	How many tokens to guess per step	Not "more is better". Match it to your acceptance rate: low α → k = 2; high α → k = 4–8
prompt_lookup_min / prompt_lookup_max	N-gram match window	Shorter minimum matches more often but less reliably. Defaults are a reasonable start
Draft model choice	The proposer for draft-model speculation	Must share the target's tokeniser. Keep it small — cost c is in the denominator
draft_tensor_parallel_size	TP degree for the drafter	EAGLE requires 1, even when the target uses tensor parallelism
Enable / disable by load	Whether speculation runs at all	The genuinely important one — see below

The decision procedure:

1. What's your concurrency? Consistently high — probably skip it. Low, or bursty with idle periods — investigate.
2. Does your output echo your input? Yes → n-gram, and it's nearly free. No → you need a draft model or EAGLE, with real cost.
3. Measure acceptance rate on your actual traffic.
4. Pick k from the table, not from a blog post.
5. Measure ITL at your real concurrency, not at batch 1.

---

Where It Bites You

Enabling it on a high-concurrency server. The most common way to lose throughput. Speculation spends spare compute, and a well-batched server doesn't have any. The benefit is largest exactly where you need it least.

Assuming more speculative tokens is better. The optimum k depends on acceptance rate, and past it you lose. At α = 0.4 with c = 0.1, going from k = 2 to k = 8 turns a 1.30× gain into a 0.93× loss.

Using a draft model that's too big. Every draft pass costs, and the cost sits in the denominator. A drafter at 30% of the target's cost needs >72% acceptance at k = 8 just to break even. Cheap and mediocre usually beats expensive and good.

Expecting n-gram speculation to help creative generation. It copies from context. If your output doesn't repeat your input, there's nothing to copy and acceptance is near zero.

Measuring throughput when the goal was latency. Speculation targets ITL at low concurrency. Judging it by aggregate tokens/sec under load will usually show a regression, correctly, while missing what it was for.

Assuming the acceptance rate from a paper applies to you. It's a property of your model, your drafter and your traffic. Published figures are measured on the authors' workload.

Forgetting the tokeniser must match. A draft model from a different family can't propose tokens the target understands. Use a small model from the same family.

Worrying that it changes output quality. It doesn't — properly implemented, the output distribution is identical to the target model's. This is the rare optimisation you don't need to re-evaluate for accuracy. Evaluate it for speed, at your real concurrency.

---

In Production

Monitor acceptance rate as a first-class metric. It's the single number that determines whether speculation is helping, and it drifts with your traffic. A prompt-template change that reduces copying will quietly turn a gain into a tax. If acceptance falls below your break-even, you're paying for nothing.

Consider making it load-dependent. The honest configuration isn't on or off — it's on when concurrency is low and off when it's high. That's awkward to express in most deployments, which is why a common pattern is a separate low-concurrency pool with speculation enabled for latency-sensitive interactive traffic, alongside a high-throughput pool without it.

It's a latency lever, so it belongs in a latency SLO conversation. From Prefill vs Decode, ITL is bounded below by memory bandwidth at batch 1 — and speculation is the only technique in this article that moves that bound. If you have a per-user speed target you cannot otherwise meet, this is the tool.

Agent workloads are the strongest case, and the most overlooked. A chain of twenty sequential calls has concurrency 1 by construction, no matter how many users you have — each step waits for the last. Total latency is twenty decode loops, batching cannot help, and agent trajectories are repetitive enough that n-gram acceptance is often high. Low effective concurrency plus high acceptance is exactly the regime where speculation wins.

What changes at 10× traffic. The benefit erodes and eventually inverts, because spare compute disappears. Speculation is one of the few settings in this article that should be reconsidered as you scale rather than simply re-tuned — and if you set it once at launch and never revisited it, it may now be costing you throughput.

---

Check Yourself

Recall the idea

What does speculative decoding do, in one sentence?

A cheap proposer guesses several tokens ahead; the target model verifies all of them in a single forward pass and keeps the longest correct prefix — trading spare compute for fewer sequential decode steps.

Why doesn't it change output quality?

Because verification accepts or rejects against the target model's own distribution, so the emitted tokens are distributed exactly as if the target had generated them alone. It changes how fast tokens arrive, not which tokens arrive.

Why is verifying k tokens cheaper than generating k tokens?

Generation is inherently serial — each token depends on the last. Verification is parallel: the transformer can evaluate all k candidate positions in one forward pass, the same property that makes prefill parallel. You convert k serial steps into one parallel one.

Which technique in this article improves single-user latency, and why is that unusual?

This one. Everything else — PagedAttention, continuous batching, prefix caching — improves throughput or capacity under concurrency, and does nothing when one user is alone on the server. Speculation targets exactly that case.

Explain the mechanics

Write the speedup formula and explain each term.

speedup = (1 + α + α² + … + α^k) / (1 + k·c). The numerator is the expected number of tokens per verification step: the accepted prefix plus the bonus token the verification always produces. The denominator is the cost: one target forward pass plus k draft passes, where c is the draft's cost relative to the target.

Why can speculation make a server slower?

When acceptance is low, you pay for k draft passes and accept roughly one token, so the denominator grows while the numerator doesn't. At α = 0.2, k = 8, c = 0.1 the speedup is 0.69× — a 31% slowdown.

Why is the optimal k a function of α?

Each additional speculated token adds a fixed cost c but contributes only α^k expected tokens, which shrinks geometrically. Once the marginal token's expected contribution falls below its cost, further speculation loses. High α keeps the contribution large for longer, so it supports deeper speculation; low α does not.

Why does the benefit shrink as concurrency rises?

Speculation works by spending spare compute on parallel verification. Decode arithmetic intensity is roughly the batch size, so speculating k tokens across B sequences gives an effective batch of about B × (k+1). At B = 1 that's still far below the ridge point and the compute is free; at high B it pushes past the ridge point into compute-bound territory, where verification competes with real work.

Reason about a trade-off

Your interactive coding assistant has p95 concurrency of 2 and ITL of 30 ms, which users call slow. Evaluate speculative decoding.

This is close to the ideal case. Concurrency is low, so there's spare compute and batching cannot help you. Code assistance is also repetitive — the model frequently reproduces identifiers, imports and surrounding code from context — so n-gram acceptance is likely high at zero draft cost. Start with n-gram at k = 4, measure acceptance and ITL at realistic concurrency, and tune k from the table. If acceptance disappoints, a small same-family draft model or EAGLE is the next step, with the break-even threshold checked first.

A colleague reports 3× on their benchmark. What do you ask?

At what concurrency — almost certainly 1, which is not your production regime. Then: what workload, since acceptance is workload-dependent and a copy-heavy benchmark inflates it dramatically. Then whether they measured ITL or throughput, since those move in opposite directions here under load. The number is probably real and probably doesn't transfer.

When is a draft model worth it over n-gram?

When your output doesn't echo your input, so n-gram acceptance is near zero, and you still need latency. Then you're paying real cost c for real acceptance, and the decision is arithmetic: check your break-even against the table before committing. EAGLE is attractive because it's trained against the target, giving higher acceptance at low cost — at the price of needing a matching trained head and draft_tensor_parallel_size = 1.

Should you enable speculation on a server that's sometimes idle and sometimes saturated?

Not globally, because the right answer differs between those states — a gain at low load becomes a throughput tax at high load. The clean resolution is two pools: a latency-optimised one with speculation for interactive traffic, and a throughput-optimised one without it for batch and high-concurrency work. Routing by traffic class is more work than a flag, and it's the only configuration that's correct in both regimes.

---

Cheat Sheet

The formula

speedup = (1 + α + α² + … + α^k) / (1 + k·c)

  α = acceptance rate   k = tokens speculated   c = draft cost ÷ target cost

Break-even acceptance rate

Draft cost c	k = 2	k = 4	k = 8
0.00 (n-gram)	always wins	always wins	always wins
0.10	0.17	0.29	0.44
0.20	0.31	0.46	0.62
0.30	0.42	0.57	0.72

Optimal k rises with α (at c = 0.10)

α	Best k	Speedup
0.2	1	1.09
0.4	2	1.30
0.6	3	1.67
0.8	6	2.47
0.9	8	3.40

The proposers

Method	Cost	Best for
N-gram	Free	Output that quotes input — summarisation, editing, code, RAG
Draft model	Small but real	General text; must share the tokeniser
EAGLE	Small	Highest acceptance; needs draft_tensor_parallel_size = 1

The three things to remember

1. It's a low-concurrency technique. It spends spare compute — a busy server has none.
2. Low acceptance makes you slower, and more speculation is not better. Pick k from α.
3. Output quality is unchanged. Evaluate it for speed at your real concurrency, not for accuracy.

---

Sources

• vLLM speculative decoding configuration (speculative_config, num_speculative_tokens, prompt_lookup_min/max, draft_tensor_parallel_size) — docs.vllm.ai/en/stable/api/vllm/config/speculative/
• vLLM, EAGLE Draft Models — docs.vllm.ai/en/latest/features/speculative_decoding/eagle/
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding — arxiv.org/abs/2211.17192
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling — arxiv.org/abs/2302.01318

---

← Previous: Quantisation · Next: Stage 3 — Running It Locally →

---

⚠️ Verification checklist (delete before publishing)

Derived and checked this session

• The speedup model, the c = 0.10 speedup table, the break-even thresholds and the optimal-k table were all computed from speedup = Σαⁱ / (1 + k·c), not taken from a source. They are labelled DERIVED in the page.
• The non-monotonicity result — that optimal k depends on α, and that k = 8 at α = 0.4 turns a 1.30× gain into 0.93× — falls out of the model and is the page's central practical claim.

The model's own assumptions — state or test these

• The model assumes verification of k tokens costs exactly one forward pass and that draft cost is linear in k. Both are first-order approximations. Consider stating them inline so the table isn't mistaken for a measurement.
• expected_tokens uses the standard geometric result including the bonus token. Confirm this matches vLLM's actual accept-reject implementation (some formulations exclude the bonus).
• The claim that a fully-rejected draft leaves you "no worse off apart from drafting cost" depends on the bonus token always being emitted. Verify.

vLLM specifics — could not fetch the primary doc

• The --speculative-config invocation in Try It is unverified. The feature docs would not fetch this session; the JSON keys come from the config API reference and secondary sources. Confirm the exact flag name and schema against vllm serve --help before publishing.
• Confirm method: "ngram" is the current spelling, and whether prompt_lookup_min/max are required or optional.
• Confirm EAGLE requires draft_tensor_parallel_size = 1 while the target may use TP > 1.
• Confirm vLLM exposes an acceptance-rate metric, and its name — In Production tells readers to monitor it as a first-class signal.
• The 40–60% (draft model) and 60–80% (EAGLE-3) acceptance figures are from secondary sources and flagged inline. Source them properly or drop the numbers.

Claims to verify

• That output distribution is provably identical to the target model's. True for the standard rejection-sampling formulation in the papers cited — confirm vLLM implements that rather than a lossy approximation, since the page tells readers they needn't re-evaluate accuracy.
• The effective-batch argument (B × (k+1) pushing past the ridge point) is reasoned from the roofline page, not measured. It's the page's explanation for why speculation hurts under load — worth measuring in Experiment 2's concurrency contrast.

Code

• spec_economics.py has been run; output is in the page.
• Run Experiment 2 on a T4: capture ITL for copy-heavy vs creative prompts, at concurrency 1 and 32. The concurrency inversion is the central claim and is currently unmeasured.
• Confirm a 0.5B model is a sensible target for n-gram speculation, or whether the effect only becomes visible on a larger target where decode is slower.

Rendering

• One image generated and placed; row added to image-prompts.md.
• All relative links resolve once target files exist — this page links forward to Stage 3, which doesn't exist yet.