> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/benchmarks/speed-h200.md).

# One H200

Total output, time to finish, chats served at once and first token for Qwen3.8-27B on one NVIDIA H200, from 1 to 768 users: NVCL AI on its automatic settings against every vLLM 0.32 and SGLang 0.5.22 setting we ran.

Terms are explained in [How to read the tables](/benchmarks/benchmarks.md#how-to-read-the-tables).

## Every setting

On Qwen3.8-27B, NVCL AI wrote more total output than every vLLM and SGLang setting we ran, at every load we measured. Even against the competitor's fastest setting at each load, it wrote 1.14-1.85x the total output on the Throughput tier against FP8 (1-768 users) and 1.34-2.52x on the Standard tier against BF16 (1-32 users; one and two users measured with a separate, publicly available draft model; re-measurement with the current default, a draft head (MTP), is planned). Quality of the Throughput tier: FP8 class, by our full 1,015-task run.

One H200, 1-768 users, mixed prompts of about 4,000 tokens; NVCL AI on its automatic settings with drafts, Throughput against FP8 and Standard against BF16; vLLM 0.32 with drafts off, depth 3, depth 7 and dynamic depth; SGLang 0.5.22 FP8 with depth 3 and BF16 with drafts off and depth 3. Each cell is NVCL AI's total output ÷ the competitor's at that load.

<figure><picture><source srcset="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-bfa88747eb5baef9cff8cc4fc7f3f1209d0500a7%2Fchart-every-setting-dark.svg?alt=media" media="(prefers-color-scheme: dark)"><img src="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-cfeb37afa1d54eed782cf1d3241776bdbce920f1%2Fchart-every-setting-light.svg?alt=media" alt="Small line charts, one per competitor setting, of NVCL AI&#x27;s total output divided by the competitor&#x27;s at each number of users on one H200; every line stays above 1x. FP8 class, Throughput against vLLM FP8 with drafts off, depth 3, depth 7 and dynamic depth and SGLang FP8 with depth 3: 1.85-4.88x at one user, 1.15-1.39x at the last load measured. Full precision, Standard against vLLM BF16 with drafts off, depth 3, depth 7 and dynamic depth and SGLang BF16 with drafts off and depth 3: 2.52-7.27x at one user, 1.34-2.04x at 32 users."></picture><figcaption><p>One panel per competitor setting. vLLM FP8 with depth 7 and SGLang FP8 were measured up to 256 users, the BF16 settings up to 32; conditions at the end of this page.</p></figcaption></figure>

<details>

<summary>Every setting as a table</summary>

| Their setting, users →                                                  |     1 |     2 |     4 |     8 |    16 |    32 |    64 |   128 |   256 |   512 |   768 |
| ----------------------------------------------------------------------- | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: |
| **NVCL AI, Throughput, total output, tok/s**                            |   412 |   515 |   668 | 1,075 | 1,810 | 2,103 | 2,432 | 2,562 | 2,542 | 2,506 | 2,510 |
| vLLM FP8, drafts off                                                    | 4.88x | 3.34x | 2.59x | 2.35x | 2.11x | 1.76x | 1.36x | 1.23x | 1.18x | 1.24x | 1.24x |
| vLLM FP8, depth 3                                                       | 2.05x | 1.43x | 1.40x | 1.23x | 1.23x | 1.16x | 1.16x | 1.14x | 1.16x | 1.14x | 1.15x |
| vLLM FP8, depth 7                                                       | 1.85x | 1.36x | 1.39x | 1.35x | 1.31x | 1.35x | 1.30x | 1.41x | 1.39x |     - |     - |
| **vLLM FP8, dynamic depth (reference)**                                 | 1.85x | 1.36x | 1.39x | 1.35x | 1.21x | 1.16x | 1.17x | 1.27x | 1.26x | 1.24x | 1.25x |
| SGLang FP8, depth 3                                                     | 2.12x | 1.56x | 1.42x | 1.27x | 1.24x | 1.17x | 1.18x | 1.26x | 1.26x |     - |     - |
| **Fastest FP8 setting at that load**                                    | 1.85x | 1.36x | 1.39x | 1.23x | 1.21x | 1.16x | 1.16x | 1.14x | 1.16x | 1.14x | 1.15x |
| **NVCL AI, Standard, total output, tok/s**                              |   429 |   595 |   687 | 1,133 | 1,730 | 1,938 |     - |     - |     - |     - |     - |
| vLLM BF16, drafts off                                                   | 7.27x | 5.52x | 4.10x | 3.50x | 2.64x | 2.04x |     - |     - |     - |     - |     - |
| vLLM BF16, depth 3                                                      | 2.87x | 2.23x | 1.93x | 1.76x | 1.50x | 1.34x |     - |     - |     - |     - |     - |
| vLLM BF16, depth 7                                                      | 2.56x | 2.04x | 1.86x | 1.74x | 1.57x | 1.51x |     - |     - |     - |     - |     - |
| **vLLM BF16, dynamic depth (reference; also the fastest BF16 setting)** | 2.52x | 2.00x | 1.82x | 1.70x | 1.46x | 1.34x |     - |     - |     - |     - |     - |
| SGLang BF16, drafts off                                                 | 6.82x | 5.36x | 3.64x | 3.33x | 2.62x | 2.00x |     - |     - |     - |     - |     - |
| SGLang BF16, depth 3                                                    | 2.81x | 2.45x | 1.92x | 1.84x | 1.65x | 1.46x |     - |     - |     - |     - |     - |

</details>

## Each load: Throughput against vLLM FP8 with dynamic depth

| Users | Total output, tok/s (NVCL AI / vLLM) | Ratio | Test time, s (NVCL AI / vLLM) | Finished sooner | First token, median s (NVCL AI / vLLM) | Writing at once (NVCL AI / vLLM) |
| ----: | ------------------------------------ | ----: | ----------------------------- | --------------: | -------------------------------------- | -------------------------------- |
|     1 | 412 / 223                            | 1.85x | 26 / 44                       |           1.73x | 0.3 / 0.3                              | 1 / 1                            |
|     2 | 515 / 379                            | 1.36x | 20 / 26                       |           1.30x | 0.3 / 0.3                              | 2 / 2                            |
|     4 | 668 / 479                            | 1.39x | 21 / 29                       |           1.39x | 0.3 / 0.3                              | 3 / 3                            |
|     8 | 1,075 / 797                          | 1.35x | 28 / 38                       |           1.39x | 0.3 / 0.3                              | 6 / 6                            |
|    16 | 1,810 / 1,501                        | 1.21x | 35 / 43                       |           1.22x | 0.3 / 0.3                              | 12 / -                           |
|    32 | 2,103 / 1,816                        | 1.16x | 55 / 64                       |           1.17x | 0.3 / 0.5                              | 26 / -                           |
|    64 | 2,432 / 2,084                        | 1.17x | 103 / 125                     |           1.21x | 0.3 / 2.7                              | 53 / -                           |
|   128 | 2,562 / 2,013                        | 1.27x | 190 / 237                     |           1.25x | 8.9 / 40.8                             | 97 / -                           |
|   256 | 2,542 / 2,023                        | 1.26x | 374 / 466                     |           1.25x | 69.7 / 116.5                           | 104 / -                          |
|   512 | 2,506 / 2,013                        | 1.24x | 749 / 923                     |           1.23x | 194.7 / 267.9                          | 107 / -                          |
|   768 | 2,510 / 2,010                        | 1.25x | 749 / 924                     |           1.23x | 316.0 / 419.3                          | 108 / 59                         |

## Each load: Standard against vLLM BF16 with dynamic depth

| Users | Total output, tok/s (NVCL AI / vLLM) | Ratio | Test time, s (NVCL AI / vLLM) | Finished sooner | First token, median s (NVCL AI / vLLM) |
| ----: | ------------------------------------ | ----: | ----------------------------- | --------------: | -------------------------------------- |
|     1 | 429 / 170                            | 2.52x | 25 / 58                       |           2.27x | 0.3 / 0.4                              |
|     2 | 595 / 297                            | 2.00x | 16 / 36                       |           2.19x | 0.3 / 0.4                              |
|     4 | 687 / 377                            | 1.82x | 20 / 38                       |           1.93x | 0.3 / 0.4                              |
|     8 | 1,133 / 666                          | 1.70x | 26 / 46                       |           1.75x | 0.3 / 0.5                              |
|    16 | 1,730 / 1,188                        | 1.46x | 35 / 57                       |           1.61x | 0.3 / 0.5                              |
|    32 | 1,938 / 1,447                        | 1.34x | 59 / 82                       |           1.40x | 0.3 / 0.7                              |

## Chats served at once

At 512-768 users NVCL AI Throughput writes answers for 107-108 chats at once, against 59 for vLLM 0.32 FP8 with dynamic depth at 768 users, and the same 1,536 requests finish 1.23x sooner (749 s against 923 s at 512 users). vLLM's fastest FP8 setting at these loads, fixed depth-3 drafts, wrote answers for 106 chats at once at 768 users; NVCL AI still wrote 1.14-1.15x its total output at 512-768 users.

NVCL AI sizes how many chats it serves at once so that each gets about 22 tokens/s ([how](/concepts/why-fast.md#more-output-as-the-load-grows)). That is a sizing rule, not a guarantee: the gain is more chats and more total output, not faster chats.

## First token

With 1-32 users the median first token is about 0.3 s on NVCL AI in both tiers. On the Standard tier that is against 0.4-0.7 s for vLLM 0.32 BF16 with dynamic depth. Above about 128 users neither server gets more output, and first token there is queueing: those points test capacity, not a service level.

<details>

<summary>Conditions of every table on this page</summary>

* Qwen3.8-27B on one NVIDIA H200 NVL at 600 W.
* Mixed prompts of about 4,000 tokens (40% code, 40% English, 10% Russian, 10% maths), sent to every server as the same token ids; answers up to 2,048 tokens; thinking on; sampling at temperature 1.0, top-p 0.95, top-k 20; a 131,072-token context; prefix cache off.
* Each user sends its requests one after another: 3 per user, with at least 8 and at most 1,536 in all (so 8 at one user, 4 each at two users, 2 each at 768 users).
* NVCL AI on its automatic settings: it picks how many requests it serves at once (112 for Throughput) and its draft depth, by load. Throughput measured on 25 October 2026 (32-64 users: same build, 128-request cap), Standard on 24 October 2026.
* Competitors on the same card, 24 October 2026: vLLM 0.32.0 with up to 768 requests at once (FP8 depth 7: best of 128 and 768) and SGLang 0.5.22 with up to 64. The FP8 runs use 8-bit weights and activations and an 8-bit attention cache. SGLang with depth-7 drafts did not start with our settings.
* "-": not measured (vLLM FP8 with depth 7 and SGLang at 512-768 users; the full-precision settings and Standard above 32 users), or, under writing at once, a run that did not record each request's times.
* Total output: from 16 users over the busy part of the test, at 1-8 users over the whole test. The one-user totals include reading the 4,000-token prompt, so they are lower than per-user output speeds that start the clock at the first token.
* First token: with thinking on, the first streamed piece is the start of the thinking.
* Run-to-run: two identical servers of ours differed by about 4%, runs of ours with different settings by up to 6%, and three vLLM FP8 runs were within 1.5%. Each point was measured once.

</details>