> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/concepts/benchmarking.md).

# How we benchmark

The rules behind every output and quality comparison in these docs. Every result is measured by our team, with the same client and method for every server. In short: for output, the same cards, the same prompts sent as the same tokens, and each competitor in every [draft](/concepts/speculative-decoding.md) setting we ran, with its [fastest setting at each load](/benchmarks/benchmarks.md#how-to-read-the-tables) shown next to ours; a gain within run-to-run noise counts as level, not as a lead. For quality, every server answers the same questions, and we compare them one by one with a statistical test.

## What we compare against

We compare against the two open-source servers that inference providers use most: **vLLM** and **SGLang**. Each result names its competitor versions; a newer version needs a new measurement before we compare against it.

* **Card for card.** One of our cards against one of theirs, two against two. Never two of ours against one of theirs. On two cards every server ran in turn on the same machine.
* **Each tier against its own quality class.** Each competitor runs the full-precision original (BF16) and the official 8-bit checkpoint (FP8), which writes more [total output](/benchmarks/benchmarks.md#how-to-read-the-tables). Standard, our full-precision-class tier, is compared with BF16; our FP8-class tiers with FP8. A tier is never set against a competitor of a higher [quality class](/concepts/quality.md#formats-and-their-quality-class).
* **Drafts on both sides.** NVCL AI runs once per load, on its automatic settings: its draft depth follows the load by default. A competitor's draft setting is chosen when it starts (vLLM's dynamic depth, too, follows a schedule set at start), so the competitors run every draft setting in turn: on one H200, vLLM 0.32 with drafts off, depth 3, depth 7 and its dynamic depth, and SGLang 0.5.22 FP8 with depth 3 and BF16 with drafts off and with depth 3. Headlines compare drafts on both sides. Bulk generation runs with drafts off on every server.
* **Our reference vLLM setting is its dynamic depth.** vLLM changes its draft depth with the number of running requests, so one setting serves every load, as a deployment must. Headlines compare with it, on one H200.
* **Every competitor setting counts.** Every run is kept, with its software versions, launch command and settings, and a total-output or first-token number goes into these docs only after we have checked it against its run record or measured it again. The tables show the reference setting and, at each load, the competitor's [fastest setting at that load](/benchmarks/benchmarks.md#how-to-read-the-tables), whatever its server, draft setting or request limit. It can be a different setting at each load and it favours the competitor ([Every setting](/benchmarks/speed-h200.md#every-setting)). A setting that fails to start is recorded as not run, never quietly replaced by a weaker one, and we claim no advantage over it: SGLang with depth-7 drafts did not start with our settings.

## How a serving number is taken

* **One machine, one server at a time.** The cards are empty before a server starts and after it stops: another process on a card, even an idle one, takes memory and time.
* **The same power cap.** Serving benchmarks run on NVIDIA H200 NVL cards capped at 600 W, one or two per server, the same for every server. Quality runs name their machines in their conditions.
* **The same prompts, as token ids.** The prompts are turned into tokens once, with the model's own chat template, and every server receives exactly these tokens, so differences in how servers apply chat templates cannot change the work.
* **A realistic mix.** Prompts of about 4,000 tokens: 40% code, 40% English, 10% Russian, 10% maths; the model thinks before it answers. Drafts guess some kinds of text better than others, so a test on one kind of text would flatter or punish them.
* **A steady load.** N users each send a request, wait for the full answer and send the next at once (closed loop): 1 to 768 users, 3 requests per user, with at least 8 and at most 1,536 in all (so 8 at one user, 4 each at two users, 2 each at 768 users). Answers of up to 2,048 tokens, sampled with the model maker's settings (temperature 1.0, top-p 0.95, top-k 20), on every server.
* **One count for every server.** The benchmark client counts the answer tokens itself, with the model's own tokenizer, from the text it received (thinking included), the same way for every server. A server's own count can be wrong.
* **What is measured.** For the whole server, total output and the test time; per request, the first token, as a median (and a 95th percentile where stated). Each is defined in [How to read the tables](/benchmarks/benchmarks.md#how-to-read-the-tables). Per-request speed (how fast each answer streams once it has started, and request times) is planned for these runs.
* **Prefix cache by workload.** Off for every server in the serving tests; on for every server in the coding-agent tests and in bulk generation.
* **Capacity under heavy load.** At high user counts we also record how many chats are writing answers at the same moment, counted by the benchmark client (a server's own gauge of requests held means something else, so we never compare it), and the time to finish the same requests.
* **Read the curve, not one point.** Two identical servers of ours differed by about 4%, runs of ours with different settings by up to 6%, and three vLLM FP8 runs were within 1.5%. So a gain above about 6% for one run is a lead, given with what it means (1.10x the total output is 10% more output from the same GPUs and a 9% cheaper token); within the noise it is level.
* **Comparing with other numbers.** Shorter prompts, greedy decoding (always the most likely token), one kind of text, a burst of requests instead of a steady load, a server's own token count, or one user's streaming speed set against total output: each of these changes the result. Before you set a number against ours, check its conditions, or measure both servers the way this page describes.

## How quality is compared

* **Statistics, not matching texts.** Under batching, greedy answers are not guaranteed to repeat bit for bit from run to run on any server (ours, vLLM or SGLang): the order in which the GPU adds numbers can change with what else is in the batch. So we never judge quality by whether two answers match word for word; we compare scores question by question.
* **Paired tests.** Both servers answer exactly the same questions, and we compare them question by question, not just the totals. A paired statistical test then tells us whether a gap is larger than luck would explain.
* **"No loss detected", never "better".** If a difference could be luck, we report no difference, even when our count is higher.
* **Measured noise.** We repeat the same setup to see how much the score moves on its own. On our 1,015-task release suite a plain repeat changes the outcome of 64-70 items, and the total moves by about ±15-20 tasks from run to run; a total difference inside that band is noise.
* **Chance wins do not count.** Across many tests, a few come out "significant" by chance. We do not treat those as findings.
* **Scoring problems are fixed for both sides.** Where a public test's scoring had a problem, it was rescored or re-run the same way for both servers.
* **Model-written code runs in a sandbox**: an isolated container with no network access.

## How we decide

The pass mark for each experiment is written before it runs and not moved afterwards. If our quality is significantly below the reference, neither total output nor speed is published as an advantage.

Results with their conditions: [Benchmarks](/benchmarks/benchmarks.md).