> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/concepts/why-fast.md). # Why it is fast With a few users per GPU, speed is set by GPU memory: how many bytes it reads for each read of the model's weights, and how many tokens each read gives. NVCL AI reads fewer bytes, keeps the memory busy and gets several tokens out of each read, while keeping answer quality. What that gives against vLLM and SGLang is on [Performance at a glance](/overview/performance.md). ## Every token means reading the weights A language model is billions of numbers, called weights. To write the next token, the GPU reads them from its memory: all of them in Qwen3.8-27B; in a mixture-of-experts model such as DeepSeek-V4.1-Flash, only the part used for that token, about 16B of its 552B parameters. One such read is a *pass*, and it gives the next token for every chat in it. A GPU calculates very fast but reads memory comparatively slowly. With only a few users, the arithmetic waits for the reading. So with a few users the GPU's output comes down to one line: **tokens per second ≈ memory speed ÷ bytes read per pass × tokens per pass** That gives three ways to get more tokens out of the same GPU, and NVCL AI works on all three: 1. **Fewer bytes per pass:** its own weight formats. 2. **Memory kept busy:** its own GPU code keeps the memory reading. 3. **More tokens per pass:** speculative decoding, and many chats in the same pass. ## Fewer bytes, full-precision quality NVCL AI stores weights in its own formats, which only NVCL AI runs. The original model stores each weight in 16 bits (BF16). Our default tier on NVIDIA GPUs, **Standard**, stores each weight in about 7.13 bits, a little fewer than the 8 of FP8, the official 8-bit checkpoint that most servers run. The GPU reads a little under half the bytes for every pass. That is the main reason NVCL AI writes more than vLLM serving the full-precision model when a few users are writing. Against FP8 the bytes are about the same. There the difference is quality: Standard stays in the full-precision [quality class](/concepts/quality.md#formats-and-their-quality-class), a class above FP8. The hard part is keeping that quality. Rounding weights carelessly makes the answers worse. The real work is storing each weight in fewer bits while the model's answers barely change, and checking that after every change. See [Why quality holds](/concepts/quality.md) and [Weight formats](/reference/formats.md). ## Custom CUDA kernels NVCL AI is our own inference engine, written in C++. It is not a fork of vLLM or SGLang. On NVIDIA GPUs, the code that runs the model uses custom CUDA kernels; like any CUDA program, it also uses standard NVIDIA libraries. On Macs with Apple Silicon it runs on its own Metal engine. NVCL AI is built for a few chosen models on chosen GPUs, to get the most out of each GPU for them. ## More tokens per pass * **Speculative decoding.** A small helper guesses the next few tokens, and the model checks all of them in one pass, for about the cost of producing one. The check is built so that the answer stays the model's own and only the time changes. See [Speculative decoding](/concepts/speculative-decoding.md). * **Many chats in one pass.** The weights are read once for all the chats writing at that moment, so every extra chat adds tokens to the same read. That is where the output under load comes from (next section). ## More output as the load grows With many users at once, each pass carries many chats, so the reading is shared and the limit moves to the GPU's arithmetic and to the memory that holds the conversations. For Qwen3.8-27B on NVIDIA GPUs, NVCL AI sizes how many chats it serves at once so that each still gets about 22 tokens per second when all are writing; the other requests wait for a place instead of slowing everyone down. That is a sizing rule, not a guarantee. The gain under heavy load is more chats at once and more [total output](/benchmarks/benchmarks.md#how-to-read-the-tables), not faster chats. Under heavy load there is the opt-in **Throughput** tier, a second 7-bit format for many users at once on Hopper cards. Quality of the Throughput tier: FP8 class, by our full 1,015-task run. See [Weight formats](/reference/formats.md). Capacity results: [Chats served at once](/benchmarks/speed-h200.md#chats-served-at-once). ## Less waiting on a busy server * **Short questions do not wait behind long documents.** When one user sends a 30,000-word contract and another asks a one-line question a moment later, the question does not wait for the whole contract to be read: it is worked in between, and the contract's reading finishes only a fraction of a second later. The order changes when a request is computed, not what is computed. * **A returning conversation is not read again.** For Qwen3.8-27B, NVCL AI keeps recent conversations' state in GPU memory and in the server's RAM. When a user comes back with a conversation that is still cached, it continues from there instead of reading the whole conversation again. ## The combination is ours What sets NVCL AI apart is the combination: our own weight formats, the GPU code written to read them, and the cache, speculative decoding and scheduling built around them.