> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/benchmarks/quality.md).
# Quality
Task scores for Qwen3.8-27B on NVCL AI: the Standard tier against the full-precision model on vLLM and SGLang, and the Compact and Throughput tiers against the official FP8 checkpoint on vLLM. Why quality holds is explained in [Why quality holds](/concepts/quality.md).
**Lossless quality** with the Standard tier, NVCL AI's default: no measurable quality loss on our 1,015-task suite (the differences stay within its run-to-run noise); Exact when you need the original bit for bit. Through the chat template, Standard is level with the full-precision BF16 model on the release suite, standard benchmarks, multi-turn instruction following (Multi-IF) and long context up to 128K tokens. **p** is the chance of seeing a gap at least this big if the two setups were really equal; above 0.05 means not significant.
Conditions of every table on this page
* Qwen3.8-27B, prompts through the chat template.
* NVCL AI: Standard tier, build of 2026-10-16 (release suite: 2026-10-14).
* Competitors: vLLM 0.31 and SGLang 0.5.21 with the BF16 model; FP8 rows use the official FP8 checkpoint.
* Greedy, thinking at effort `medium`; answers up to 32,768 tokens on the release suite, Multi-IF and LongBench v2.
* Each section below lists only what differs.
## Release suite (1,015 tasks)
| Setup | Score |
| ------------------------------ | --------------- |
| NVCL AI, Standard (three runs) | 732 / 732 / 730 |
| vLLM BF16 | 726 |
| SGLang BF16 | 726 |
| vLLM FP8 / SGLang FP8 | 723 / 721 |
* Standard against the BF16 servers: no per-task difference is significant (all p ≥ 0.25).
* NVCL AI's default fast prompt processing shows no task-level loss: 732, the same as exact prompt processing, with 72 changed items against 70 between two plain repeats.
Conditions
* Tasks: IFBench 300, AIME 2026 + HMMT Feb 2026 63, LiveCodeBench v6 60, AA-LCR 50, EvalPlus 542.
* On H200 SXM machines. Measured 2026-10-14.
* The three Standard runs: two with different NVCL AI settings and a repeat. Standard scored 732 again on a build of 2026-10-21.
* Noise: a repeat of the same setup flips 64-70 items (both ways together), and the total moves by about ±15-20 tasks from run to run. Total differences inside that band are noise.
## Compact and Throughput against FP8 (1,015 tasks, two H200)
| Setup | Two runs | Mean |
| ---------------------------------------------------- | --------- | ----: |
| NVCL AI, Compact | 722 / 722 | 722.0 |
| vLLM 0.32 FP8 (official checkpoint) | 715 / 722 | 718.5 |
| vLLM 0.32, Unsloth NVFP4 (about 5.9 bits per weight) | 721 / 719 | 720.0 |
* Compact is in the FP8 class with about 5.8 bits per weight, fewer than FP8's 8: no significant difference from the official FP8 checkpoint, or from Unsloth's NVFP4 checkpoint of about the same size.
* Quality of the Throughput tier: FP8 class, by our full 1,015-task run: two runs, no significant difference from the official FP8 checkpoint.
Conditions
* NVCL AI's Compact and Throughput tiers; competitors on vLLM 0.32.
* 2x H200 NVL at 600 W, the model split across both cards; drafts of depth 7 and an 8-bit attention cache on every server.
* The release suite as above; AA-LCR answers judged by vLLM BF16.
* Paired exact McNemar tests: p ≥ 0.50 for every pair of setups.
* Noise: a repeat of the same setup flips 25-46 tasks in each direction.
* Measured 2026-10-22 to 2026-10-23.
## Standard benchmarks
| Benchmark | NVCL AI, Standard | vLLM BF16 |
| ---------------------------- | ----------------: | --------: |
| BBH | 0.918 | 0.916 |
| MGSM | 0.824 | 0.822 |
| IFEval | 0.889 | 0.876 |
| AIME 2024+2025 (4 samples) | 0.829 | 0.812 |
| GSM8K-Platinum | 0.895 | 0.894 |
| MMLU-ProX Lite, 10 languages | 0.694 | 0.691 |
| MMLU-ProX Lite, Russian | 0.716 | 0.711 |
Also level on MMLU-Pro (full), MMLU-Redux, MATH-500, GSM8K and Russian MGSM (English and Russian prompts). No difference is significant.
Conditions
* lm-eval 0.4.13; paired McNemar or sign tests.
* Thinking at effort `medium` for IFEval, MATH-500 and AIME; off for MMLU-Pro, MMLU-Redux, BBH, MGSM, MMLU-ProX and GSM8K-Platinum.
* AIME sampled 4 times at T 0.6. GSM8K 5-shot through chat.
* MATH-500 and MGSM were rescored, and BBH re-run with a fixed stop rule, the same way for both servers.
* No per-language MMLU-ProX difference is significant (p 0.11-1.0). On MGSM, 2 of 22 language tests reached p < 0.05, both in NVCL AI's favour; that is what chance predicts over 22 tests.
* Measured 2026-10-16 to 2026-10-18.
## Multi-turn instruction following (Multi-IF)
| Setup | Score, mean of 3 turns | p against NVCL AI, turns 1 / 2 / 3 |
| ----------------- | ---------------------: | ---------------------------------- |
| NVCL AI, Standard | 0.826 | |
| vLLM BF16 | 0.823 | 0.55 / 0.01 / 0.54 |
| SGLang BF16 | 0.822 | 0.06 / 0.04 / 0.96 |
Standard is level with the BF16 servers and shows no deficit on any turn. On turn 2 it scored a little higher than both (p 0.01 and 0.04); among six tests a result like that can come from chance, so we count it as level, not as a lead.
Conditions
* Multi-IF: 4,501 three-turn conversations in 8 languages (English, French, Hindi, Portuguese, Spanish, Russian, Italian, Chinese). Each turn adds instructions and carries the model's own earlier answers.
* The official protocol and checkers; the score is the official mean over the three turns. Paired McNemar tests per turn.
* vLLM 0.31.0 on one H200 SXM, one server after the other; SGLang on one H200 NVL, same protocol.
* Measured 2026-10-19 (vLLM) and 2026-10-20 (SGLang).
## Long context (RULER)
| | NVCL AI, Standard | vLLM BF16 |
| -------------------------------------- | ----------------: | ----------: |
| Needle retrieval, through the template | 1.000 | 1.000 |
| Needle retrieval, raw prompts | 0.994-0.995 | 0.994-0.995 |
Also level on RULER at 32K and 128K tokens; no difference is significant.
Conditions
* lm-eval RULER, 13 subtasks (650 items at 32K, 325 at 128K).
* Thinking off; raw prompts skip the chat template.
* Measured 2026-10-16 to 2026-10-17.
## Long real documents (LongBench v2)
| Setup | Right, of 503 |
| ----------------- | ------------: |
| NVCL AI, Standard | 302 (60.0%) |
| vLLM BF16 | 300 (59.6%) |
Standard is level with vLLM BF16 and SGLang BF16. No difference is significant: against vLLM BF16 p 0.89 (29 against 27 questions right on one side only), against SGLang BF16 p 0.58.
Conditions
* LongBench v2: 503 multiple-choice questions on real long documents. The median prompt is about 102.5K tokens; longer prompts are cut to 120K tokens the official way.
* The official prompt. Every server got the same requests, 8 at a time, prefix cache off.
* vLLM 0.31.0 with BF16 weights and attention cache, on one H200 SXM, one server after the other; SGLang on one H200 NVL.
* NVCL AI: the Standard tier, engine build of 16 October 2026, served through its OpenAI-compatible API.
* Measured 2026-10-19 (vLLM) and 2026-10-20 (SGLang).