Back to Articles/vLLM & LLM Inference Serving

04 · Streaming & Client Patterns

19 min read

Streaming looks like a presentation choice — text appearing word by word instead of all at once. It isn't. It changes what a request is: a response that arrives over seconds, can fail halfway through having already delivered half an answer, and can be abandoned by the client while the server is still working.

It's also the only way to measure TTFT, which is why every latency experiment in this article used it.

The Problem

You enabled streaming and the user still sees everything at once. Server-side metrics look perfect. Something between you and them is buffering.
A client hung for twenty minutes because nothing timed out, on either side.
A user pressed stop and your GPU kept generating the remaining 800 tokens, holding KV blocks the whole time.
You can't get token counts from a streaming response — the usage field is empty.
Occasional mojibake in the middle of streamed text.
A timeout triggered a retry, which timed out, which retried — and now you're generating three copies of an answer nobody is waiting for.

Every one of these is about the stream rather than the model. They're client and network problems, and they're invisible in any server-side dashboard.

The Idea

A letter versus a phone call.

A non-streaming request is a letter: it's composed in full, sent once, and arrives complete. Either it arrives or it doesn't. Handling it is simple because there's only one moment when anything happens.

Streaming is a phone call. It starts before either party knows how it ends, it unfolds over time, and crucially either side can hang up mid-sentence. That's not a rare edge case — it's the normal way calls end when someone gets what they needed.

Three consequences that don't exist for letters:

Partial success is a real outcome. A stream can deliver 200 useful tokens and then fail. Your client needs a position on whether that's a success, a failure, or something to retry — and retrying gets you the first 200 tokens again.
Hanging up must propagate. If the caller puts the phone down and the other end keeps talking, someone is paying for a conversation with nobody in it. That's your GPU.
Anything that buffers the call destroys it. A middlebox that waits for the last word before passing anything along has converted your phone call back into a letter, while every metric on both ends still says "streaming".

Under the Hood

The wire format

vLLM streams using server-sent events (SSE) — the same format OpenAI uses. Each chunk is a line prefixed with data: , followed by a blank line:

data: {"id":"cmpl-...","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"cmpl-...","choices":[{"index":0,"delta":{"content":"The"},"finish_reason":null}]}

data: {"id":"cmpl-...","choices":[{"index":0,"delta":{"content":" KV"},"finish_reason":null}]}

data: {"id":"cmpl-...","choices":[{"index":0,"delta":{"content":" cache"},"finish_reason":null}]}

data: {"id":"cmpl-...","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Four things worth noticing:

The first chunk usually carries the role and no content. A client that assumes every chunk has text will produce a stray empty token or crash on None.

The last content chunk carries finish_reason. That's where you learn why it stopped — stop for a stop token or EOS, length for hitting max_tokens. Distinguishing those matters: length means the answer was truncated, which is usually a bug in your max_tokens, not a complete response.

data: [DONE] is a literal sentinel, not JSON. Parsing every data: line as JSON without checking for it is a common client bug.

Chat and completions have different chunk shapes:

Endpoint	Where the text is
`/v1/chat/completions`	`choices[0].delta.content`
`/v1/completions`	`choices[0].text`

Streaming is the only way to measure TTFT

This is why it matters beyond user experience. Without streaming, the client observes one event: the complete response. You can measure end-to-end latency and nothing else.

From Prefill vs Decode, TTFT and ITL have different causes and different fixes — so collapsing them into one number destroys the information you need. Every measurement harness in this article streams for exactly this reason, even when nothing is displaying the tokens.

Usage data needs asking for

Streaming responses don't carry token counts by default — the usage block arrives at the end of a non-streaming response, and a stream has already ended by then. To get it:

stream = client.chat.completions.create(
    model="chat",
    messages=[...],
    stream=True,
    stream_options={"include_usage": True},     # ← adds a final chunk with usage
)

This appends a final chunk carrying prompt and completion token counts. Without it, streaming clients have no token accounting — which matters if you're billing, rate limiting, or doing the cost arithmetic from Cost per Token.

⚠️ Confirm stream_options support and the exact final-chunk shape against your vLLM version.

Detokenisation across chunk boundaries

From Lifecycle of a Request: tokens don't align to character boundaries. A multi-byte UTF-8 character — an emoji, an accented letter, most non-Latin scripts — can span two tokens.

vLLM handles this server-side with incremental detokenisation, so the chunks you receive contain valid text. The failure appears when you reassemble incorrectly: decoding bytes per chunk in isolation, or splitting on character counts for display. If you're building custom client-side handling, concatenate the strings and let the server own tokenisation.

Cancellation, and why it's a whole-path property

When a client disconnects mid-stream, the desirable behaviour is that the engine aborts the request and returns its KV blocks to the pool. Otherwise you're generating tokens nobody will read while holding blocks other requests need — paying twice for nothing.

The subtlety: this depends on the entire network path, not just the server. If a proxy between client and server buffers the response, it holds its upstream connection open regardless of what the client did. The server sees a healthy consumer and keeps generating. A correct engine behind a buffering proxy still wastes GPU on abandoned requests.

So cancellation is something you verify end to end, through the real path, not something you assume from a configuration flag.

⚠️ vLLM's disconnect-abort behaviour is asserted in three places in this article and has not been confirmed against a running server. It's the first item in this page's Try It for that reason.

Try It

Hardware: Colab T4 or any GPU with compute capability ≥ 7.5.

vllm serve Qwen/Qwen2.5-0.5B-Instruct --served-model-name chat --max-model-len 4096

Experiment 1 — see the raw wire format

curl -N -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"chat","messages":[{"role":"user","content":"Count to five."}],
       "max_tokens":40,"stream":true}'

-N disables curl's own buffering. Without it you'll see the entire stream arrive at once — a useful accident, because it's exactly the failure a buffering proxy causes.

What you should observe: individual data: lines arriving progressively, a first chunk carrying the role, a final chunk with finish_reason, and data: [DONE].

Now change one thing: drop the -N. The output arrives in one burst. That is what your users see if anything in your network path buffers, and every server-side metric will look perfect while it happens.

Experiment 2 — measure TTFT and ITL properly

# stream_timing.py — what streaming lets you measure that non-streaming can't.
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

def timed(stream: bool):
    t0 = time.perf_counter()
    if not stream:
        client.chat.completions.create(
            model="chat", messages=[{"role":"user","content":"Explain paged memory briefly."}],
            max_tokens=150, temperature=0.0)
        return {"E2E": time.perf_counter() - t0, "TTFT": None, "ITL": None}

    ttft, stamps = None, []
    for chunk in client.chat.completions.create(
            model="chat", messages=[{"role":"user","content":"Explain paged memory briefly."}],
            max_tokens=150, temperature=0.0, stream=True):
        if chunk.choices and chunk.choices[0].delta.content:
            now = time.perf_counter()
            if ttft is None:
                ttft = now - t0
            stamps.append(now)
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return {"E2E": time.perf_counter() - t0, "TTFT": ttft,
            "ITL": sum(gaps) / len(gaps) if gaps else None}

for mode in (False, True):
    r = timed(mode)
    label = "streaming" if mode else "non-streaming"
    ttft = f"{r['TTFT']*1000:6.0f} ms" if r["TTFT"] else "  — unmeasurable"
    itl  = f"{r['ITL']*1000:5.1f} ms" if r["ITL"] else "  —"
    print(f"{label:<14} E2E {r['E2E']:5.2f}s   TTFT {ttft}   ITL {itl}")

Observation	What it proves
E2E is roughly the same either way	Streaming doesn't make generation faster
TTFT is only available when streaming	You cannot separate the two phases without it
TTFT is a small fraction of E2E for a long answer	Users perceive responsiveness long before completion — the reason streaming exists

Experiment 3 — does cancellation actually free the GPU?

This is the experiment that resolves an open question in this article.

# cancel_test.py — abandon a long generation, then check whether the server noticed.
import threading, time
import httpx

BODY = {"model": "chat",
        "messages": [{"role": "user", "content": "Write an extremely long essay about memory."}],
        "max_tokens": 2000, "stream": True}

def abandon_after(seconds: float):
    with httpx.Client(timeout=None) as c:
        with c.stream("POST", "http://localhost:8000/v1/chat/completions", json=BODY) as r:
            t0 = time.perf_counter()
            for _ in r.iter_lines():
                if time.perf_counter() - t0 > seconds:
                    print(f"client disconnecting after {seconds}s")
                    return          # closing the context manager drops the connection

abandon_after(2.0)
print("client gone — now watch the server for ~30s")
time.sleep(30)

What to watch on the server side, in its log output and metrics:

If cancellation works	If it doesn't
Running-request count drops promptly after disconnect	The request keeps running to `max_tokens`
KV cache usage falls back	Cache stays occupied for tens of seconds
An abort or cancellation line appears in the log	Nothing; generation completes normally

Then run the same test through a proxy — nginx with default settings is the realistic case. If cancellation worked directly and stops working through the proxy, you've reproduced the failure from Under the Hood, and you know it's a path problem rather than a server one.

Whatever you observe, it settles a claim this article currently makes in three places on the strength of reasoning alone.

Dial It In

Knob	Where	Guidance
`stream=True`	Request	Always for interactive use; also required to measure TTFT
`stream_options={"include_usage": True}`	Request	When you need token counts from a stream
Client `timeout`	Client	Set it. The `openai` client's default may be longer than you want
Client `max_retries`	Client	Default retry behaviour plus a long generation is a way to triple your load
`proxy_buffering off`	nginx	Required, or streaming silently degrades to non-streaming
`proxy_read_timeout`	nginx	Must exceed your longest generation, not your average
`max_tokens`	Request	Bounds the worst case — from Sampling Parameters, it's a memory reservation

Timeouts need to be set at every layer, and they need to be consistent. A client that gives up at 30 s behind a proxy that waits 60 s behind a server with no limit produces the worst combination: the client retries while the original request is still generating, so load rises exactly when the system is already slow.

Where It Bites You

A proxy that buffers the response. nginx and several ingress controllers buffer by default. Your stream arrives in one piece, TTFT as the user experiences it becomes E2E, and every server-side metric still looks perfect. Verify streaming through the full path, never against the server directly.

No client timeout. A hung connection with no timeout waits forever. Set one, and set it longer than your longest legitimate generation — which you know from max_tokens and your measured ITL.

Automatic retries on long generations. The openai client retries by default. A timeout on a 90-second generation that then retries has you generating the same answer twice, concurrently. Under load this is a genuine amplification mechanism — the system is slow, so requests time out, so clients retry, so it gets slower.

Treating a completed stream as a successful response. Check finish_reason. length means you hit max_tokens and the answer is truncated mid-thought, which is a different outcome from stop and usually indicates a configuration problem rather than a complete reply.

Parsing data: [DONE] as JSON. It's a literal sentinel. Check for it before parsing.

Assuming every chunk has content. The first chunk typically carries the role with no content, and the final chunk carries finish_reason with no content. chunk.choices[0].delta.content can be None — guard it.

Reassembling tokens yourself. Concatenate the strings the server gives you. Decoding per chunk or splitting on byte counts breaks multi-byte characters, which shows up as mojibake in exactly the languages you tested least.

Assuming cancellation works because you closed the connection. It depends on the whole path. A buffering proxy holds the upstream connection open and the server never learns the client left.

In Production

Verify streaming end to end before launch, through the real ingress. This is a five-minute check that catches a failure invisible to every metric you have — and one that will otherwise be reported as "the model feels slow" long after launch.

Set consistent timeouts at every layer, ordered sensibly. Client timeout ≤ proxy read timeout, and both comfortably above your worst-case generation time. Then make retry policy deliberate: retries on a streaming endpoint are expensive, because a retry regenerates from scratch and you've already paid for the abandoned attempt.

Make cancellation a tested property. For interactive products it's a real cost saving — users abandon long answers routinely, and those are exactly the requests holding the most KV blocks. Test it through the deployed path and monitor for its absence: running-request counts that don't fall when traffic does are the symptom.

Instrument TTFT and ITL from the client, not just the server. Server-side timing excludes everything in Where It Bites You. The client's view is the user's view, and it's the only place a buffering proxy is visible.

What changes at 10× traffic. Streaming makes the API server's job heavier in a way the GPU doesn't feel: from Lifecycle, detokenisation and SSE writes are per-token CPU work in the API server process. Ten times the traffic is ten times the socket writes, and that's where --api-server-count and CPU provisioning start to matter. Retry storms also become dangerous rather than merely wasteful, because they add load precisely when the system is struggling.

Check Yourself

Recall the idea

What's the SSE wire format, and what terminates it?

Lines of the form data: {json} separated by blank lines, terminated by the literal data: [DONE]. The final content chunk carries finish_reason; the first typically carries the role with no content.

Why is streaming the only way to measure TTFT?

Without it the client observes a single event — the complete response — so it can measure end-to-end latency and nothing else. Streaming exposes the arrival time of the first token and of every subsequent one, which is what separates TTFT from ITL.

Where is the text in a streaming chunk?

choices[0].delta.content for chat completions, choices[0].text for completions. Both can be absent or None on the first and last chunks.

How do you get token counts from a stream?

stream_options={"include_usage": True}, which appends a final chunk carrying usage. Streams don't include it by default, because the usage block normally arrives with a complete response.

Explain the mechanics

Your streaming endpoint delivers everything at once. Diagnose.

Something in the path is buffering — most likely a reverse proxy with response buffering on by default. Confirm by testing directly against the server with curl -N: if it streams there and not through ingress, the proxy is the cause. For nginx that's proxy_buffering off, plus a proxy_read_timeout longer than your longest generation.

Why is a client-side retry on a streaming endpoint expensive?

The abandoned attempt already consumed prefill and however many decode steps it completed, and the retry regenerates from scratch. You pay twice for one answer. Worse, retries triggered by timeouts arrive when the system is already slow, so they add load exactly when it can least absorb it.

Why does finish_reason matter?

It distinguishes a complete answer from a truncated one. stop means a stop token or EOS — the model finished. length means it hit max_tokens and was cut off mid-thought. Treating both as success means shipping truncated answers and never noticing.

Why can cancellation fail even when the server implements it correctly?

Because it's a property of the whole path. A buffering proxy consumes the upstream response independently of the downstream client, so when the client disconnects the proxy keeps reading and the server sees a healthy consumer. The engine never learns the request was abandoned.

Reason about a trade-off

Should an internal batch job stream?

Not for its own sake — there's no user perceiving responsiveness, and streaming adds per-token CPU work in the API server. But if you want TTFT and ITL measurements from it, streaming is the only way to get them, so a monitoring or benchmarking job should stream even though nothing displays the output. For pure bulk generation, the offline LLM class is simpler than either (Offline Batch Inference).

How would you choose a client timeout?

Work from the worst case rather than the average: max_tokens × your measured p99 ITL, plus p99 TTFT, plus headroom. On a T4 with 30 ms ITL and max_tokens=2000, that's roughly a minute before you've allowed anything for queueing. Then set the proxy's read timeout above the client's, so the client gives up first and the proxy doesn't sever a connection the client still wants. And cap max_tokens server-side, because otherwise your worst case is unbounded and no timeout can be principled.

A user reports the answer "stops halfway". Walk through it.

First check finish_reason — if it's length, the answer isn't stopping, it's being truncated by max_tokens, and the fix is configuration. If it's stop, the model chose to end and that's a prompt or model question. If the stream ends with neither — no finish_reason, no [DONE] — then the connection broke, and you're looking at a proxy timeout, a network fault, or a server restart. Three quite different causes, distinguished by one field.

Is it worth implementing cancellation propagation for an internal tool with ten users?

Probably not on cost grounds — ten users won't move your GPU bill. But it's worth knowing whether it works, because the same code path serves the product later, and it's much easier to verify now than to retrofit under load. The cheap version is one test through the real path plus a note in the runbook. The expensive version — custom abort plumbing — waits until you have traffic that justifies it.

Cheat Sheet

The wire format

data: {"choices":[{"delta":{"role":"assistant"}}]}      ← first chunk: role, no content
data: {"choices":[{"delta":{"content":"The"}}]}         ← content chunks
data: {"choices":[{"delta":{},"finish_reason":"stop"}]} ← why it ended
data: [DONE]                                            ← literal sentinel, not JSON

Endpoint	Text field
`/v1/chat/completions`	`choices[0].delta.content`
`/v1/completions`	`choices[0].text`

Client essentials

stream = client.chat.completions.create(
    model="chat", messages=[...], stream=True,
    stream_options={"include_usage": True},      # token counts
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:   # guard: may be None
        print(chunk.choices[0].delta.content, end="", flush=True)

finish_reason

Value	Meaning
`stop`	Stop token or EOS — a complete answer
`length`	Hit `max_tokens` — truncated, usually a config problem
absent, stream just ends	Connection broke — proxy, network, or restart

Proxy settings that are not optional

proxy_buffering off;                 # or your stream becomes a letter
proxy_read_timeout 300s;             # longer than your longest generation

Three things to remember

Test streaming through the real path, not against the server. Buffering is invisible in server-side metrics.
You cannot measure TTFT without streaming. That's why every harness in this article uses it.
Check finish_reason. A completed stream is not necessarily a complete answer.

Sources

vLLM, Online Serving — endpoints and OpenAI compatibility — github.com/vllm-project/vllm/blob/main/docs/serving/online_serving/README
OpenAI API streaming semantics, which vLLM implements — chunk shape, finish_reason, stream_options — platform.openai.com/docs/api-reference

← Previous: The OpenAI-Compatible Server · Next: The Getting-Started Notebook →

⚠️ Verification checklist (delete before publishing)

This page owns an open question

Does vLLM abort a request when the client disconnects? This article asserts it in three places — The KV Cache, Lifecycle and here — on reasoning alone. cancel_test.py in Try It is designed to settle it. Run it, capture what the server does, and correct all three pages to match. Highest-priority item on the page.
Then run the same test through nginx with default settings, to confirm (or refute) that buffering breaks cancellation.

Needs verifying

stream_options={"include_usage": True} support in vLLM, and the exact shape of the final usage chunk. Flagged inline.
The exact SSE chunk JSON — the example is written from the OpenAI spec, not captured from a running vLLM server. Replace with real captured output.
Whether the first chunk always carries role and no content, or whether that varies.
The openai Python client's default timeout and max_retries, so the Dial It In advice can name real numbers rather than "may be longer than you want".
Confirm vLLM emits an identifiable log line on client disconnect — the Try It observation table tells readers to look for one.

Code

Run stream_timing.py and paste real numbers. The claim that E2E is roughly equal streaming and non-streaming is central and unmeasured.
Run Experiment 1 with and without -N and confirm the buffering contrast is visible.
cancel_test.py uses httpx — confirm it's available, or rewrite with requests (stream=True + r.close()) to avoid an extra dependency.
Confirm that closing the httpx stream context actually drops the TCP connection promptly rather than draining it first. If it drains, the experiment silently tests nothing.

Rendering

No diagram yet. Candidate: the letter-versus-phone-call contrast as two timelines, showing where TTFT is measurable in one and not the other.
All relative links resolve once target files exist.