> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/benchmarks/benchmarks.md).

# Benchmarks

Total output, latency, quality and agent results for NVCL AI, with the conditions of every run.

{% hint style="info" %}
All results are measured by our team, on Qwen3.8-27B unless a table says otherwise. How we run the comparisons is on [How we benchmark](/concepts/benchmarking.md); the summary in plain words is on [Performance at a glance](/overview/performance.md).
{% endhint %}

## How to read the tables

| Term                          | Meaning                                                                                                                                                                                                                                                         |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ratio                         | Above 1 always means NVCL AI is ahead. Total output: NVCL AI's ÷ the competitor's (1.5x = 1.5 times their total output). Times (first token, time to finish the same requests): the competitor's time ÷ NVCL AI's ("sooner").                                   |
| Total output, tok/s           | All users' answer tokens per second together, thinking included, counted by the benchmark client with the model's own tokenizer. It includes reading prompts and waiting in the queue, so it measures capacity and cost per token, not the speed one user sees. |
| Users                         | Requests in flight at all times: each user sends its next request as soon as its answer ends.                                                                                                                                                                   |
| Writing at once               | How many requests receive answer tokens at the same moment, on average, counted by the benchmark client the same way for every server.                                                                                                                          |
| Test time                     | From the first request sent to the last answer received.                                                                                                                                                                                                        |
| First token                   | From sending a request to the first streamed piece of the answer, queueing and reading the prompt included.                                                                                                                                                     |
| Median                        | Half the requests waited less.                                                                                                                                                                                                                                  |
| Drafts                        | Speculative decoding; see [Speculative decoding](/concepts/speculative-decoding.md).                                                                                                                                                                            |
| MTP                           | A draft head built into the model.                                                                                                                                                                                                                              |
| Depth 3, depth 7              | Drafts that guess 3 or 7 tokens per step.                                                                                                                                                                                                                       |
| Dynamic depth                 | vLLM changes its draft depth with the number of running requests, by a schedule set when it starts. Our reference vLLM setting.                                                                                                                                 |
| Fastest setting at that load  | The competitor run with the most total output among all its settings in that precision at that load (on one H200: every vLLM 0.32 setting and the SGLang 0.5.22 settings we ran).                                                                               |
| BF16                          | The original full-precision model.                                                                                                                                                                                                                              |
| FP8                           | The official 8-bit checkpoint.                                                                                                                                                                                                                                  |
| Standard, Compact, Throughput | NVCL AI's weight formats (tiers); Standard is the default. See [Weight formats](/reference/formats.md).                                                                                                                                                         |
| TP2                           | One copy of the model split across two cards.                                                                                                                                                                                                                   |
| DP2                           | A full copy of the model on each card, behind one address.                                                                                                                                                                                                      |
| p                             | The chance of seeing a gap at least this big if the two setups were really equal; above 0.05 means not significant.                                                                                                                                             |
| Paired test                   | Both setups answer the same questions, and the test looks at the questions only one of them got right (McNemar or sign test).                                                                                                                                   |
| Run-to-run noise              | How much a result moves when exactly the same setup runs again.                                                                                                                                                                                                 |
| Level                         | Within run-to-run noise: for total output, see [Read the curve](/concepts/benchmarking.md#how-a-serving-number-is-taken); for quality, a gap that is not significant.                                                                                           |