> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/concepts/speculative-decoding.md). # Speculative decoding A small helper guesses the next few tokens, and the model checks all of them in one pass. The check is built so that the answer stays the model's own and only the time changes. ## What it is Without speculative decoding, the model produces one token per pass. Each pass reads the model's weights from GPU memory, and with a few users that reading is what takes the time (see [Why it is fast](/concepts/why-fast.md)). With speculative decoding: 1. A small, fast helper (the *draft*) guesses the next few tokens. 2. The model checks all the guesses in one pass. 3. It keeps the guesses it agrees with, up to the first one it would not have written, and adds its own token at that point.

Diagram of one step. After the text The cat, the draft guesses four tokens: sat, on, a, mat. The model checks all four in one pass: it keeps sat and on, writes its own token the instead of a, and drops mat. One pass of the model writes three tokens. — An example step: two guesses kept, the model's own token where the third guess differs, the rest dropped.

Checking several guesses costs about as much as producing one token, because the weights are read once for all of them. Every guess the model keeps is a token that cost almost nothing extra. Think of a colleague drafting your next sentence: you only nod or correct. The words are still yours. Qwen3.8-27B drafts with a draft head (MTP), a small extra layer; on NVIDIA GPUs it drafts with it at every load, and new installs there get our own version of it, made for speed only. ## Why drafts are built not to change the answer The draft only proposes. The model decides every token. * **Greedy decoding** (always the most likely token): the model keeps a guess only if it is exactly the token it would have picked itself. So the answer is the one the model would write without drafts. The one exception is a near-tie between the model's two most likely tokens, where tiny numeric differences can tip the choice. That happens with or without drafts. * **Sampling** (temperature above 0): the accept-or-reject step is built so that each token is still drawn from the model's own probabilities. The text differs from run to run, as sampling always does. vLLM and SGLang use the same principle. Why our quality comparisons use statistics rather than matching texts: [How we benchmark](/concepts/benchmarking.md#how-quality-is-compared). ## What it gives The gain depends on how many guesses the model keeps. That depends on the text. Predictable text such as code or maths keeps more guesses. Sampling at a high temperature keeps fewer than greedy decoding. Our serving benchmarks sample on every server, with the model maker's settings, so their results are for sampled requests ([How we benchmark](/concepts/benchmarking.md#how-a-serving-number-is-taken)). It gains most where speed per user matters: with a few users per GPU. Deeper drafts, which guess more tokens per step, are not always better. Each extra guess costs work, and with many users the GPU is already busy. So for Qwen3.8-27B on NVIDIA GPUs, NVCL AI picks its draft depth by load for you, with nothing to tune. Fixed depths show why: in our runs on one H200 from 16 to 256 users, vLLM 0.32 FP8 with fixed depth-7 drafts wrote less [total output](/benchmarks/benchmarks.md#how-to-read-the-tables) than with depth 3 or its [dynamic depth](/concepts/benchmarking.md#what-we-compare-against) ([Every setting](/benchmarks/speed-h200.md#every-setting)). Which draft settings each comparison uses: [How we benchmark](/concepts/benchmarking.md#what-we-compare-against).