> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/benchmarks/two-gpus.md).

# Two H200

Qwen3.8-27B on two NVIDIA H200 cards. On the default Standard tier NVCL AI reaches up to 4,952 tokens/s of total output (one copy per card, 256 users; conditions below), in the layout that fits the load. Below: both layouts at each load, the two tiers, and a bulk job against vLLM.

Terms are explained in [How to read the tables](/benchmarks/benchmarks.md#how-to-read-the-tables).

## Which layout to use

How to pick the layout is on [Install](/get-started/install.md#gpu-layout). NVCL AI, Standard tier, total output in tok/s at each load:

<figure><picture><source srcset="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-7dfb05d440c2f6393e29d9144691f976581df1df%2Fchart-two-gpu-layouts-dark.svg?alt=media" media="(prefers-color-scheme: dark)"><img src="https://3133985975-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FDZCnCH3aMU9wYLt2Z4wR%2Fuploads%2Fgit-blob-8ebeed54e0a7e33f6414cd804dce987423a53e5f%2Fchart-two-gpu-layouts-light.svg?alt=media" alt="Line chart of NVCL AI Standard&#x27;s total output on two H200 by users at once, with one copy split across both cards (TP2) and one copy per card (DP2). The split copy writes more from 1 to 8 users, the two are level at 16 and 32 users, and one copy per card writes more from 64 to 768 users: 4,829 tok/s against 3,941 at 768 users."></picture><figcaption><p>NVCL AI Standard on two H200, on its automatic settings with drafts; conditions below. Split copy: one and two users measured with a separate, publicly available draft model; re-measurement with the current default, a draft head (MTP), is planned. Level: within run-to-run noise.</p></figcaption></figure>

<details>

<summary>The numbers</summary>

| Users | One copy split across both cards (TP2) | One copy per card (DP2) | Layout with more output |
| ----: | -------------------------------------: | ----------------------: | ----------------------- |
|     1 |                                    613 |                     428 | split, 1.43x            |
|     2 |                                    934 |                     756 | split, 1.24x            |
|     4 |                                  1,086 |                     807 | split, 1.35x            |
|     8 |                                  1,777 |                   1,396 | split, 1.27x            |
|    16 |                                  2,748 |                   2,735 | level                   |
|    32 |                                  3,259 |                   3,353 | level                   |
|    64 |                                  3,777 |                   4,274 | copy per card, 1.13x    |
|   128 |                                  4,060 |                   4,549 | copy per card, 1.12x    |
|   256 |                                  3,994 |                   4,952 | copy per card, 1.24x    |
|   512 |                                  3,939 |                   4,755 | copy per card, 1.21x    |
|   768 |                                  3,941 |                   4,829 | copy per card, 1.23x    |

</details>

At 768 users the same 1,536 requests finish in 405 s with one copy per card, against 481 s with the split copy.

<details>

<summary>Conditions</summary>

* Qwen3.8-27B, Standard tier, on two NVIDIA H200 NVL cards at 600 W, linked by NVLink.
* Prompts, answers, sampling, closed loop and how total output is counted as on [One H200](/benchmarks/speed-h200.md); prefix cache off.
* NVCL AI on its automatic settings, with drafts.
* Split copy (TP2): 25 October 2026; NVCL AI chose 176 requests at once.
* One copy per card (DP2): 22-23 October 2026, 128 requests at once per card (an earlier version of the router; a cache-aware router reproduced these rows at 1-128 users within about 3% on 25 October).
* One run per point. Runs of ours with different settings differed by up to 6%, so the rows marked level are within noise.

</details>

## Standard and Throughput on two cards

With the model split across both cards, Throughput and Standard write about the same up to 64 users (Standard is ahead at 2 users), and Throughput writes 7-8% more from 128 users. Quality of the Throughput tier: FP8 class, by our full 1,015-task run.

Conditions: as for the split copy above; NVCL AI chose 192 requests at once for Throughput.

## Bulk generation

NVCL AI writes 9-14% more output on bulk jobs (Throughput, two H200, 2,048 short prompts, 512 at once, drafts off on both sides, vLLM 0.31 FP8): an 8-12% cheaper token, and the Russian set finished in 183 s against 207 s.

A bulk job answers a large set of short prompts as fast as possible, so the cards spend almost all their time writing answers: compare these numbers only within this table.

| Prompt set   | NVCL AI, Throughput, tok/s | vLLM 0.31 FP8, tok/s | Ratio |
| ------------ | -------------------------: | -------------------: | ----: |
| Russian      |                      9,230 |                8,128 | 1.14x |
| English chat |                      8,288 |                7,491 | 1.11x |
| Code         |                      7,675 |                7,041 | 1.09x |

<details>

<summary>Conditions</summary>

* Qwen3.8-27B split across two NVIDIA H200 NVL cards at 600 W (TP2); 8-bit attention cache and prefix cache on both servers.
* NVCL AI on its automatic settings: 512 requests at once (480 for English chat and code); vLLM 0.31 FP8 512.
* Per set: 2,048 prompts (Russian about 212 tokens on average), answers up to 1,024 tokens, temperature 1.0, thinking off.
* Total output: answer tokens ÷ test time.
* One run per set and server, 22-23 October 2026. SGLang was not run.

</details>