> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/overview/performance.md).

# Performance at a glance

Where NVCL AI gets more out of every GPU than vLLM and SGLang, in plain words.

## More output from every GPU

On one NVIDIA H200, NVCL AI's total server output was higher than vLLM's at every load we measured, in each quality class, with NVCL AI on its automatic settings with drafts and vLLM 0.32 with dynamic depth:

* **Full precision: 1.34-2.52x the total output of vLLM 0.32 BF16 with dynamic depth**, Standard tier, 1-32 users (one and two users measured with a separate, publicly available draft model; re-measurement with the current default, a draft head (MTP), is planned).
* **FP8 class: 1.16-1.85x the total output of vLLM 0.32 FP8 with dynamic depth**, Throughput tier, 1-768 users. From 32 users, 1.16-1.27x: 16-27% more output from the same GPU, a 14-21% cheaper token. Quality of the Throughput tier: FP8 class, by our full 1,015-task run.

**Terms** (more in [How to read the tables](/benchmarks/benchmarks.md#how-to-read-the-tables)):

* **Total server output** = all users' answer tokens per second together. It measures capacity and cost per token, not the speed one user sees.
* **Users** = requests in flight at all times: each user sends the next request as soon as its answer ends.
* **Drafts** = [speculative decoding](/concepts/speculative-decoding.md). **Depth 3** = drafts guessing 3 tokens per step. **Dynamic depth** = vLLM changes its draft depth with the number of running requests.
* **Tiers.** Standard, NVCL AI's default, is compared with **BF16**, the original full-precision model; Throughput, an opt-in tier, with **FP8**, the official 8-bit checkpoint.

<figure><picture><source srcset="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-04977a8c530fc4b80412b54f6207381b56c83e9e%2Fchart-total-output-dark.svg?alt=media" media="(prefers-color-scheme: dark)"><img src="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-e07cc457c3d205262a49ee9f3c4e2fe7b9defdaa%2Fchart-total-output-light.svg?alt=media" alt="Line charts of total output by users at once on one H200. FP8 class: NVCL AI Throughput above vLLM 0.32 FP8 with dynamic depth at every load from 1 to 768 users, 1.85x at one user and 1.25x at 768. Full precision: NVCL AI Standard above vLLM 0.32 BF16 with dynamic depth at every load from 1 to 32 users, 2.52x at one user and 1.34x at 32."></picture><figcaption><p>Total server output by users at once: NVCL AI on its automatic settings against vLLM 0.32 with dynamic depth, on one H200. Conditions under the table.</p></figcaption></figure>

| What                          | Result                                                                    | Conditions                                                                                                                                                                                                                        |
| ----------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Against SGLang                | **1.17-2.12x** the total output of SGLang FP8; **1.46-2.81x** SGLang BF16 | SGLang 0.5.22 with depth-3 drafts and up to 64 requests at once. Throughput, 1-256 users; Standard, 1-32 users.                                                                                                                   |
| The same work finishes sooner | **1.23x** sooner                                                          | 1,536 requests at 512 users: 749 s against 923 s for vLLM FP8 with dynamic depth. Throughput tier.                                                                                                                                |
| More chats served at once     | **107-108** chats writing answers at once, about 22 tokens/s each         | Throughput tier, 512-768 users, against 59 for vLLM FP8 with dynamic depth at 768 users, counted by the benchmark client. About 22 tokens/s each is a sizing rule, not a guarantee: the gain is more chats and more total output. |
| First token at light load     | about **0.3 s** median                                                    | From sending a request to the first streamed piece of the answer, queue included; median: half the requests waited less. Standard, 1-32 users, against 0.4-0.7 s for vLLM 0.32 BF16 with dynamic depth.                           |

Qwen3.8-27B on one NVIDIA H200 (600 W), mixed prompts of about 4,000 tokens, answers up to 2,048 tokens with the model thinking before it answers, NVCL AI on its automatic settings with drafts, one run per load, measured by our team on 24-25 October 2026. Full tables, with every vLLM and SGLang setting we ran: [One H200](/benchmarks/speed-h200.md). Method: [How we benchmark](/concepts/benchmarking.md).

## More places NVCL AI does well

Qwen3.8-27B on two H200, measured by our team.

| What                          | Result                              | Conditions                                                                                                                                                                    |
| ----------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bulk generation (a batch job) | **9-14%** more output than vLLM FP8 | Throughput tier, 2,048 short prompts with 512 at once, drafts off on both sides, against vLLM 0.31 FP8 on the same cards. [Two H200](/benchmarks/two-gpus.md#bulk-generation) |
| Ready to serve                | **21-30 s**                         | From starting the server to the first answer it accepts, model files already in memory: a restart costs about half a minute.                                                  |

## Our own quantization

Smart quantization with lossless quality: fewer bits than FP8, the quality of full precision. On Qwen3.8-27B the default tier, **Standard**, uses about 7.13 bits per weight, fewer than FP8's 8 and about 2.2x smaller than BF16, with no measurable quality loss on our 1,015-task suite (the differences stay within its run-to-run noise). Need the original bit for bit? **Exact** keeps it in 12 bits per weight, a quarter smaller than BF16. These formats run only on NVCL AI. Tiers in [Weight formats](/reference/formats.md), scores in [Quality](/benchmarks/quality.md).

Coding agents on Standard: no significant difference in the share of tasks solved, within noise, against vLLM 0.31 BF16 with depth-3 drafts (DeepSWE and Terminal-Bench 2.0, one H200). [Agents](/benchmarks/agents.md)

## Big models on fewer GPUs

**DeepSeek-V4.1-Flash** runs on two H200 plus server memory, on DeepSeek's published weights, unchanged, where vLLM's published recipe needs four: 302 tokens/s for one chat and 399 in total for two (the model's own drafts, greedy, 6 short prompts, 512-token answers, measured by our team on 23 October 2026). [Other models](/benchmarks/other-models-and-hardware.md)

**GLM-5.3**, a 750B model, runs on two cards plus server memory, where the usual setups use eight. Its request options: [API, Per model](/reference/api.md#per-model).