Background
Back to Articles/vLLM & LLM Inference Serving
Sections
IntroductionOrientation01 · Why Inference Servers Exist02 · Where vLLM Sits03 · Anatomy of a vLLM SetupStage 1 — Core Concepts01 · The KV Cache02 · Prefill vs Decode03 · PagedAttention04 · Continuous Batching05 · Sampling ParametersStage 2 — The Engine01 · The Scheduler & Block Manager02 · Lifecycle of a Request03 · Prefix Caching04 · Quantisation05 · Speculative DecodingStage 3 — Running It Locally01 · Installing vLLM02 · Offline Batch Inference03 · The OpenAI-Compatible Server04 · Streaming & Client Patterns05 · The Getting-Started NotebookStage 7 — Case Studies01 · RAG Chatbot at Scale02 · Offline Batch Summarisation03 · Coding Agent on Long Context04 · Multi-Tenant LoRA on One GPU

04 · Quantisation

24 min read

Everything so far has treated the model's weights as fixed — 2 bytes per parameter, non-negotiable, the entry fee before you get any KV cache at all. Quantisation is the lever that moves that number, and it moves it a long way: a 7B model that cannot fit on a 16 GB card in 16-bit fits comfortably in 4-bit, with room left over for the cache.

The cost is accuracy, and the cost is real but small if you choose well and measure. The bigger practical hazard isn't accuracy at all — it's that the format you pick may simply not run on the GPU you have, and the failure arrives at startup rather than in a benchmark.

The Problem

  • Your model doesn't fit. From Stage 0: Qwen2.5-7B is 15.23 GB of weights against a 14.40 GB budget on a 16 GB card. Negative KV cache. It will not start.
  • It fits, but barely, and you have no concurrency. Weights eat the card and the remaining cache holds a handful of sequences.
  • You enabled FP8 on an A100 and it failed. A100 is a datacenter GPU that costs more than a car deposit, and it cannot do FP8 weight-activation quantisation. This surprises nearly everyone.
  • You're facing an alphabet soup — AWQ, GPTQ, FP8, INT8, W8A8, W4A16, GGUF, bitsandbytes, Marlin — with no framework for telling them apart.
  • Someone says "quantisation makes it faster" and someone else says "quantisation makes it slower", and both have benchmarks.
  • You quantised, quality dropped, and nobody can say by how much because there was never an evaluation to compare against.

The Idea

Write down 3.14159265 as "3.14". You've lost precision and saved space, and for most purposes it doesn't matter.

Now write down a million such numbers, all clustered between 3.1 and 3.2. Storing each as "3.14" is wasteful — every value starts with "3.1". Instead, record one shared note saying "all these are 3.1 plus something", and then store just the something, in far fewer digits.

That shared note is a scale, and it's the entire trick of modern quantisation:

  • Fewer bits per number — 16-bit becomes 8-bit or 4-bit.
  • A shared scale per group of numbers, kept at higher precision, that says how to expand them back.
  • Group size matters. One scale for the whole model would be useless, because weights in different layers live on wildly different ranges. One scale per small group of weights keeps the digits you kept meaningful.

That's why 4-bit quantisation isn't "4 bits per weight" exactly — it's four bits per weight plus a little shared bookkeeping, which is why a 4-bit checkpoint is somewhat larger than params ÷ 2 bytes.

And the thing worth understanding before any of the acronyms: quantisation is a memory optimisation that sometimes buys speed as a side effect. From Prefill vs Decode, decode is memory-bandwidth bound — it reads every weight to produce one token. Halve the bytes and you halve the read. That's why smaller weights can decode faster. It is not because the arithmetic got cheaper.

Under the Hood

Reading the names: WxAy

Most of the alphabet soup resolves once you can read one notation. W is weights, A is activations, and the number after each is bits:

Notation Weights Activations Meaning
W16A16 16-bit 16-bit Unquantised — fp16 or bf16, the baseline
W4A16 4-bit 16-bit Weights compressed, maths still in 16-bit. AWQ, GPTQ
W8A8 8-bit 8-bit Both compressed — the maths itself runs in 8-bit. FP8, INT8

The distinction between those last two is the one that matters operationally:

W4A16 shrinks memory. Weights are stored in 4 bits and expanded back to 16-bit to compute. You get a much smaller model — more room for KV cache, and less to read per decode step — but the arithmetic is unchanged.

W8A8 shrinks memory and changes the arithmetic. The multiplications happen in 8-bit, which needs hardware that can do 8-bit maths fast. That's why W8A8 has hardware requirements W4A16 doesn't, and it's the source of the A100 surprise.

What can be quantised — three separate decisions

People conflate these constantly, and they have different formats, different hardware requirements and different trade-offs:

What Typical formats What it buys
Weights AWQ, GPTQ (W4A16); FP8, INT8 (W8A8) Smaller model → more KV cache, faster decode reads
Activations FP8, INT8 (the A in W8A8) Faster compute, on hardware that supports it
KV cache --kv-cache-dtype fp8 ~Half the cache per token → more concurrency

The third is independent of the first two. You can run 16-bit weights with an FP8 KV cache, or 4-bit weights with a 16-bit cache. They're separate flags solving separate problems — weights are a fixed cost, KV cache is a per-user cost (The KV Cache).

The hardware matrix — check this before you choose

This is the table that prevents the startup failure. Architecture names map to compute capability: Volta 7.0 · Turing 7.5 · Ampere 8.0/8.6 · Ada 8.9 · Hopper 9.0.

Implementation Volta Turing (T4) Ampere (A100) Ada Hopper AMD x86 CPU
AWQ ❌ ✅ ✅ ✅ ✅ ❌ ✅
GPTQ ✅ ✅ ✅ ✅ ✅ ❌ ✅
Marlin (GPTQ/AWQ/FP8/FP4 kernels) ❌ ✅* ✅ ✅ ✅ ❌ ❌
INT8 W8A8 ❌ ✅ ✅ ✅ ✅ ❌ ✅
FP8 W8A8 ❌ ❌ ❌ ✅ ✅ ✅ ❌
bitsandbytes ✅ ✅ ✅ ✅ ✅ ❌ ❌
GGUF ✅ ✅ ✅ ✅ ✅ ✅ ❌

*Turing does not support Marlin MXFP4.

Three readings that matter:

FP8 W8A8 needs Ada or Hopper. Not Ampere — so an A100 cannot do FP8 weight-activation quantisation. This is the single most surprising row in the table, and it catches people who reasonably assume a datacenter GPU supports everything. On Ampere and Turing, the 8-bit option is INT8.

A T4 has good options. AWQ, GPTQ, INT8 W8A8, bitsandbytes and GGUF all work on Turing. Only FP8 is out. The article's baseline hardware is not a second-class citizen here.

AMD inverts the picture. FP8 works; AWQ and GPTQ don't. If you're on ROCm, most quantisation advice written for NVIDIA is wrong for you.

The two different FP8s — a distinction worth being pedantic about

"FP8" refers to two separate features with different hardware requirements, and conflating them produces incorrect advice:

FP8 W8A8 FP8 KV cache
What it quantises Model weights and activations The KV cache only
Flag --quantization / a pre-quantised checkpoint --kv-cache-dtype fp8
Hardware Ada or Hopper (and AMD) fp8_e4m3: CUDA 11.8+ and ROCm · fp8_e5m2: CUDA 11.8+
Buys you Smaller weights, faster 8-bit maths ~2× the cacheable tokens

The KV cache variant is documented against a CUDA version, not a compute-capability floor — so it is not automatically ruled out on older cards the way FP8 W8A8 is.

⚠️ This distinction corrects an over-broad claim made earlier in this article: "FP8 is a no-op on the T4." That is right for FP8 W8A8 and possibly wrong for FP8 KV cache. Verify FP8 KV cache on your own T4 before relying on either statement — see the run sheet.

The accuracy trap hiding in FP8 KV cache

Quantising the cache needs scales, and how you obtain them is a configuration choice with three options — one of which is the default and is also the worst:

Approach Config Quality
No calibration kv_cache_dtype="fp8", calculate_kv_scales=False All scales are 1.0. Nothing is fitted to your model
Random-token calibration calculate_kv_scales=True Scales estimated from one batch of random tokens at warmup, then fixed
Dataset calibration via llm-compressor Recommended. Scales fitted against a curated dataset

Enabling FP8 KV cache without calibration means every scale is 1.0 — you've halved your cache and accepted whatever accuracy that produces, with no fitting at all. It may be fine. You have no way of knowing without measuring, which is the theme of the whole page.

Try It

Experiment 1 — does quantisation change your capacity? (no GPU)

Before downloading anything, redo the Stage 0 arithmetic with quantised weights. This decides whether quantisation solves your problem at all.

# quantised_capacity.py — what quantisation buys, in concurrent users.
GPU_GB, UTIL, OVERHEAD, MAX_LEN = 16.0, 0.90, 1.0, 4096
KV_PER_TOKEN = 57_344          # Qwen2.5-7B: 2 x 28 layers x 4 kv_heads x 128 head_dim x 2 bytes
PARAMS       = 7_615_616_512   # verified from the Hub API

SCHEMES = {
    "bf16 (W16A16)":  2.0,
    "INT8  (W8A8)":   1.0,
    "AWQ/GPTQ (W4A16)": 0.5,   # plus scale/zero-point overhead — see note
}

budget = GPU_GB * UTIL
print(f"{'scheme':<20} {'weights':>9} {'KV left':>9} {'concurrent @4096':>18}")
for name, bytes_per_param in SCHEMES.items():
    w  = PARAMS * bytes_per_param / 1e9
    kv = budget - w - OVERHEAD
    if kv <= 0:
        print(f"{name:<20} {w:>8.2f}G {kv:>8.2f}G {'DOES NOT FIT':>18}")
    else:
        print(f"{name:<20} {w:>8.2f}G {kv:>8.2f}G "
              f"{kv*1e9/KV_PER_TOKEN/MAX_LEN:>18,.0f}")
# DERIVED, not benchmarked — arithmetic from verified parameter counts
scheme                 weights   KV left   concurrent @4096
bf16 (W16A16)           15.23G    -1.83G       DOES NOT FIT
INT8  (W8A8)             7.62G     5.78G                 25
AWQ/GPTQ (W4A16)         3.81G     9.59G                 41

Three stacked horizontal bars comparing memory layouts for the same model under different quantisation schemes. Each bar divides into model weights, overhead and KV cache. Under bf16 the weights segment overflows past the card's memory limit and the row is marked as not fitting. Under INT8 the weights halve and a KV cache segment appears, giving roughly 25 concurrent sequences. Under 4-bit the weights shrink to a quarter and the KV cache segment is largest, giving roughly 41. An arrow down the right-hand side reads "reclaimed weight memory becomes concurrency"

What this shows: quantisation doesn't merely make the model fit — it converts weight memory into concurrency. Going from bf16 to 4-bit takes Qwen2.5-7B from "won't start on a T4" to roughly 41 concurrent 4k-token sequences.

The 0.5 bytes/param figure ignores scale and zero-point overhead, so real 4-bit checkpoints are somewhat larger — expect the true number to be a little below the estimate. Check the actual checkpoint size on the Hub before planning against it.

Now change one thing: set KV_PER_TOKEN to half (28,672), simulating --kv-cache-dtype fp8. Concurrency roughly doubles again. Weight quantisation and KV cache quantisation compose, and they attack different halves of the memory budget.

Experiment 2 — run a quantised model on a T4

Hardware: Colab T4. AWQ works on Turing; FP8 W8A8 does not — that's the point of the comparison.

# Baseline: the 0.5B model in bf16
vllm serve Qwen/Qwen2.5-0.5B-Instruct --max-model-len 4096

# Then an AWQ 4-bit 7B model — which would not fit at all unquantised
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --max-model-len 4096 --quantization awq

Capture from each startup log: the reported GPU blocks (your real cache capacity) and the attention/quantisation backend chosen. Then measure throughput with the harness from Prefill vs Decode.

Observation What it proves
The AWQ 7B model starts at all on a 16 GB card Quantisation moved the fixed cost enough to leave a workable cache
GPU blocks are far higher than the unquantised 7B would allow Reclaimed weight memory became KV cache — the Experiment 1 arithmetic, confirmed
Decode throughput per sequence may improve over an unquantised 7B Decode is bandwidth-bound; fewer weight bytes means a faster read
Prefill may not improve, or may worsen Prefill is compute-bound, and W4A16 still computes in 16-bit after dequantising

Then try the failure: attempt FP8 W8A8 on the T4.

vllm serve <an-fp8-checkpoint> --quantization fp8      # expect this to fail on Turing

Capture the error. A format your hardware can't run fails at startup, not silently — which is the good outcome, and worth seeing once so you recognise it.

Dial It In

Knob What it does Guidance
Pre-quantised checkpoint Download a model someone already quantised The usual path. Look for -AWQ, -GPTQ, -FP8, -INT8 variants on the Hub
--quantization <method> Names the method explicitly Often auto-detected from the checkpoint; set it when detection is ambiguous
--kv-cache-dtype fp8 Quantises the KV cache — independent of weight quantisation Roughly doubles cacheable tokens. Calibrate the scales
calculate_kv_scales=True Estimates KV scales from a warmup batch Better than the default 1.0; worse than dataset calibration
llm-compressor Quantise a model yourself, with dataset calibration When no pre-quantised checkpoint exists, or you need calibrated FP8 KV scales
--dtype Activation dtype for unquantised models On a T4, bf16 has no fast path — fp16 is the sensible choice

Choosing a scheme, in priority order:

  1. Does it run on your GPU? Consult the matrix. This filter eliminates more options than any performance consideration.
  2. Is the constraint weights or cache? A model that won't fit needs weight quantisation. A model that fits but serves too few users may only need --kv-cache-dtype fp8.
  3. On Turing/Ampere: AWQ or GPTQ for 4-bit; INT8 W8A8 for 8-bit.
  4. On Ada/Hopper: FP8 W8A8 is the natural choice — good accuracy, hardware-accelerated.
  5. Then measure quality, against an evaluation you wrote before quantising.

Where It Bites You

Assuming a newer or more expensive GPU supports everything. An A100 cannot do FP8 W8A8. Ampere predates the hardware FP8 support that arrived with Ada. Check the matrix rather than the price.

Conflating FP8 weights with FP8 KV cache. Different features, different flags, different hardware requirements. "We can't use FP8" is usually a statement about W8A8 and may not apply to the cache.

Enabling FP8 KV cache without calibration. The default is uncalibrated scales of 1.0. That's not a tuned quantisation, it's an unfitted one — and the failure mode is quietly worse outputs, not an error.

Expecting quantisation to speed up prefill. W4A16 dequantises to 16-bit to compute, so compute-bound prefill sees little benefit and can be marginally slower from the dequantisation step. The gain is in decode, which is bandwidth-bound. A RAG workload — prefill-dominated — benefits far less than a chat workload.

Skipping evaluation because "the outputs look fine." They will look fine. Quantisation damage is usually subtle: slightly worse reasoning, slightly more hallucination, degradation concentrated in rare cases. You need an evaluation from before the change, or you have no baseline and no way to attribute a later complaint.

Assuming 4-bit means params ÷ 2 bytes. Scales and zero-points add real overhead. Check the actual checkpoint size rather than trusting the arithmetic.

Quantising an already-small model. A 0.5B model in bf16 is 1 GB. Quantising it saves 500 MB and adds accuracy risk and operational complexity for a rounding error on a 16 GB card. Quantisation is for models that are large relative to your GPU.

Using bf16 on a T4. Turing has no fast bfloat16 path. It may run, slowly, or fall back. Use fp16 on Turing — and this is the general lesson: read your card's capabilities rather than copying a flag from a blog post written on an H100.

In Production

Pin the quantised checkpoint by revision, like any other model. A quantised model is a distinct artifact with its own quality characteristics, produced by someone else's calibration run against someone else's dataset. Treat it as a model version in its own right, not as a flag applied to a model you already trust.

Evaluate before and after, on your own tasks. Published quantisation benchmarks use standard evaluations that may not resemble your workload. The comparison that matters is your task, your prompts, your quality bar — and you can only make it if you measured the unquantised baseline first.

Weight quantisation changes your capacity model. Every number in The KV Cache shifts: weights shrink, so KV cache grows, so concurrency rises. Redo the arithmetic rather than assuming the old operating point still holds — and expect max_num_seqs to need raising, since the memory constraint moved.

Quantisation is a lever against cost, and it's the biggest one you have. Roughly speaking it lets you serve a given model on a smaller GPU, or serve far more users on the same one. That's the framing for Cost per Token, and it usually dominates the tuning flags elsewhere in this article.

What changes at 10× traffic. Nothing about the quantisation itself — it's a static property of the loaded model. What changes is that the reclaimed memory becomes the thing standing between you and preemption, so the benefit compounds with load precisely when you need it.

Check Yourself

Recall the idea

Decode W4A16 and W8A8.

W4A16 is 4-bit weights with 16-bit activations: weights are stored compressed and expanded to 16-bit to compute, so it's a pure memory optimisation. W8A8 is 8-bit weights and 8-bit activations, so the arithmetic itself runs in 8-bit — which requires hardware support the first doesn't.

What is a scale, and why is group size important?

A higher-precision shared multiplier that converts quantised values back to their real range. Group size matters because weights across different layers occupy very different ranges — a single global scale would waste most of the available precision, so scales are kept per small group.

Name the three independent things you can quantise.

Weights, activations, and the KV cache. Weight/activation quantisation reduces a fixed cost; KV cache quantisation reduces a per-user cost. They're separate flags and compose freely.

Why can quantisation speed up decode but not prefill?

Decode is memory-bandwidth bound — it reads every weight to produce one token — so halving weight bytes halves the dominant cost. Prefill is compute-bound, and W4A16 dequantises to 16-bit before computing, so the arithmetic is unchanged and there's little to gain.

Explain the mechanics

Why can't an A100 run FP8 W8A8?

FP8 requires hardware support for 8-bit floating-point arithmetic, which arrived with Ada (compute capability 8.9) and Hopper (9.0). A100 is Ampere (8.0), which predates it. Ampere's 8-bit option is INT8 W8A8, which is supported.

A T4 user asks which quantisation to use. What are the options and the one exclusion?

AWQ, GPTQ, INT8 W8A8, bitsandbytes and GGUF all work on Turing, plus Marlin kernels with the exception of MXFP4. The single exclusion is FP8 W8A8, which needs Ada or newer. For a 4-bit weight-only scheme, AWQ or GPTQ is the standard answer.

Someone says "we can't use FP8, we're on Turing." What should you clarify?

Which FP8. FP8 W8A8 is indeed unavailable on Turing. FP8 KV cache is documented against a CUDA version (11.8+) rather than a compute-capability floor, so it may well be available — and it solves a different problem, halving per-user cache cost rather than shrinking the weights. Worth testing before writing it off.

What happens if you enable FP8 KV cache without calibration?

All quantisation scales default to 1.0 — nothing has been fitted to the model's actual activation ranges. It still runs and still halves the cache, but the accuracy cost is unmanaged and unmeasured. The better options are warmup-batch estimation (calculate_kv_scales=True) or, best, dataset calibration via llm-compressor.

Reason about a trade-off

A 7B model won't start on your 16 GB card. Walk through the options.

Weights are 15.23 GB against a 14.40 GB budget, so the weights themselves are the problem and no scheduler flag helps. Options in rough order of preference: AWQ or GPTQ 4-bit, which takes weights to roughly 4 GB and leaves ~9.5 GB of cache — around 41 concurrent 4k sequences, and it works on Turing. INT8 W8A8 halves weights to ~7.6 GB, a smaller quality risk with less headroom, ~25 concurrent. A smaller model, which is often the right answer and rarely the popular one. A bigger GPU, which works and costs money. What doesn't help: max_num_seqs, max_model_len, or anything else that rations a cache you don't have yet.

Your model fits but serves too few users. Weight quantisation or KV cache quantisation?

Start with KV cache quantisation, because it targets the actual constraint. If the model already fits, weights are a sunk fixed cost and shrinking them is indirect; halving per-token cache cost roughly doubles concurrency directly, and it doesn't touch the weights' accuracy at all. Weight quantisation becomes attractive as a second step, or when you also want the decode-speed benefit. Do the Experiment 1 arithmetic for both before choosing.

How would you decide whether a quantised model is good enough?

Define "good enough" before quantising, as an evaluation on your own tasks with a numeric threshold — otherwise you're comparing against a memory of how it used to feel. Run the unquantised baseline, run the quantised model, compare on the same inputs. Pay attention to variance and to rare cases: quantisation damage concentrates in the tail, so aggregate scores can look unchanged while specific capabilities degrade. If you have no evaluation, building one is the prerequisite, not an optional extra.

Is quantisation worth the operational complexity for a 0.5B model on an A100?

Almost certainly not. The weights are 1 GB on an 80 GB card — quantising saves a fraction of a percent of your memory budget while adding a second artifact to version, a calibration to trust and an accuracy risk to evaluate. Quantisation earns its complexity when weights are large relative to the GPU. Here they aren't, and the constraint is elsewhere.

Cheat Sheet

Reading the notation

W4A16  →  4-bit weights, 16-bit activations   →  memory win, compute unchanged   (AWQ, GPTQ)
W8A8   →  8-bit weights, 8-bit activations    →  memory AND compute              (FP8, INT8)

Hardware, condensed

Your GPU Compute capability 4-bit 8-bit
T4 (Turing) 7.5 AWQ, GPTQ INT8 — no FP8
A100 (Ampere) 8.0 AWQ, GPTQ INT8 — no FP8
L4 / L40S (Ada) 8.9 AWQ, GPTQ INT8 or FP8
H100 (Hopper) 9.0 AWQ, GPTQ INT8 or FP8
AMD (ROCm) — GGUF FP8 — no AWQ/GPTQ

FP8 W8A8 needs Ada or Hopper. An A100 cannot do it.

The two FP8s

FP8 W8A8 FP8 KV cache
Flag --quantization fp8 / checkpoint --kv-cache-dtype fp8
Needs Ada / Hopper / AMD CUDA 11.8+ (fp8_e4m3 also ROCm)
Shrinks Weights + activations The cache only

Capacity impact — Qwen2.5-7B on a 16 GB card

Scheme Weights KV left Concurrent @4k
bf16 15.23 GB −1.83 GB won't start
INT8 7.62 GB 5.78 GB ~25
AWQ/GPTQ 4-bit 3.81 GB 9.59 GB ~41

Flags

--quantization awq                 # or gptq, fp8, bitsandbytes — often auto-detected
--kv-cache-dtype fp8               # independent of weight quantisation; ~2x cacheable tokens
--dtype float16                    # on Turing: no fast bf16 path

The three things to remember

  1. Check the hardware matrix first. It eliminates more options than performance does.
  2. Weights and KV cache are separate levers on separate halves of the memory budget, and they compose.
  3. Quantise, then evaluate — against a baseline you measured first. "It looks fine" is not a measurement.

Sources

← Previous: Prefix Caching · Next: Speculative Decoding →

⚠️ Verification checklist (delete before publishing)

Verified against vLLM's docs this session

  • The full hardware compatibility matrix, including that FP8 W8A8 is ❌ on Volta, Turing AND Ampere — supported only on Ada, Hopper and AMD. The A100 exclusion is real.
  • Architecture → compute capability mapping (Volta 7.0, Turing 7.5, Ampere 8.0/8.6, Ada 8.9, Hopper 9.0).
  • Turing supports AWQ, GPTQ, INT8 W8A8, bitsandbytes, GGUF and Marlin (except MXFP4).
  • AMD supports FP8 and GGUF but not AWQ or GPTQ.
  • FP8 KV cache options: fp8_e4m3 (CUDA 11.8+ and ROCm), fp8_e5m2 (CUDA 11.8+) — stated against CUDA version, not compute capability.
  • The three KV-scale calibration approaches, and that the default is uncalibrated scales of 1.0.
  • Qwen2.5-7B parameter count (7,615,616,512) and KV-per-token (57,344 B) — verified earlier this session from the Hub API and config.json.

Correction to propagate

  • The KV Cache says FP8 KV cache "Needs Ada/Hopper — NOT a T4". That requirement belongs to FP8 W8A8; the KV cache variant is documented against CUDA 11.8+. Either confirm FP8 KV cache runs on a T4 and fix that page, or confirm it doesn't and fix this one. The landing page's "FP8 quantisation is a no-op on the T4" line needs the same treatment. This is the highest-priority item on the page.

Needs verifying

  • Confirm --quantization awq is the right invocation for Qwen2.5-7B-Instruct-AWQ, and that the checkpoint exists under that name.
  • Confirm the real on-disk size of a 4-bit Qwen2.5-7B checkpoint, so the "somewhat larger than params ÷ 2" caveat can be quantified rather than hand-waved.
  • Confirm that FP8 W8A8 on Turing fails at startup with a clear error rather than silently falling back — the Try It failure case depends on it.
  • Confirm the claim that W4A16 can be marginally slower at prefill due to dequantisation overhead. Reasoned, not sourced.
  • Confirm bf16 has no fast path on Turing and what vLLM actually does if you request it.

Code

  • quantised_capacity.py output is arithmetic I computed, not a benchmark — confirm the three rows and consider labelling the block as derived rather than measured.
  • Run Experiment 2 on a T4: capture GPU blocks for the AWQ 7B model and compare against the predicted ~41 concurrent. The gap between prediction and reality is itself worth publishing.
  • Capture the actual FP8-on-Turing error message.

Rendering

  • Diagram added (04-quantisation-capacity.png) showing the memory-budget shift across bf16 / INT8 / 4-bit, with a row added to image-prompts.md.
  • Consider a second diagram decoding the WxAy notation if the table alone proves insufficient.
  • All relative links resolve once target files exist.