> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/reference/api.md). # OpenAI-compatible API The server that `nvcl-ai` starts speaks the OpenAI Chat Completions, Completions and Responses APIs and the Anthropic Messages API on one port, with streaming, tool calls, answers held to a JSON schema, logprobs, token counting and Prometheus metrics. Some of these depend on the model: see [Endpoints](#endpoints), [Chat request parameters](#chat-request-parameters) and [Per model](#per-model). To start it, see [Start the server](/get-started/run.md). Clients that use OpenAI Chat Completions work unchanged, for example the OpenAI SDK, litellm, `vllm bench serve`, AIPerf, EvalPlus and mini-swe-agent. Clients built for the OpenAI Responses API (Codex, the OpenAI Agents SDK) or the Anthropic Messages API (Claude Code-style harnesses) connect directly, without a proxy. Answer-length defaults, the `length` finish reason and the context-overflow error follow vLLM. ## Endpoints | Endpoint | What it does | | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /v1/chat/completions` | Chat, streaming (SSE) or not. Tool calls, a separate reasoning field, usage in every answer. Up to 64 answers per request (`n`) | | `POST /v1/completions` | Text completion without the chat template, on text or token-id prompts, one or a batch (see [Text completions](#text-completions)). Qwen3.8-27B | | `POST /v1/responses`; `GET`, `DELETE /v1/responses/{id}` | The OpenAI Responses API: text input, function tools, reasoning items; JSON output on Qwen3.8-27B on NVIDIA GPUs | | `POST /v1/messages`, `POST /v1/messages/count_tokens` | The Anthropic Messages API: text, tool calls (`auto`, `none`, `any` or a named tool), thinking with a budget, token counting. Token counting on Qwen3.8-27B; `any`, a named tool and the enforced budget on Qwen3.8-27B on NVIDIA GPUs | | `POST /tokenize`, `POST /detokenize` | Count the tokens of a text, or of chat messages as vLLM's (messages on Qwen3.8-27B); turn token ids back into text | | `GET /v1/models`, `GET /v1/models/{name}` | The served model, with `max_model_len` and the tier | | `GET /health` | 200 `ok` when serving; 503 while loading or after the engine stopped | | `GET /version` | The API server's version | | `GET /`, `GET /v1` | On NVIDIA GPUs, a plain-text page about this address: the base URL for OpenAI clients, the endpoints and a `curl` example to copy | | `GET /metrics` | Prometheus metrics (see [Monitoring](/reference/monitoring.md)) | With `nvcl-ai start --public`, the server needs the API key that `nvcl-ai` generated (it prints it, and keeps it in `~/.nvcl-ai/api-key`), sent as `Authorization: Bearer ` or as `x-api-key: ` (Anthropic clients). Only `/health`, `/version`, `/metrics` and, on NVIDIA GPUs, `GET /` and `GET /v1` answer without it. Without `--public`, the server listens on this machine only and asks for no key. A request may name its conversation with the `X-NVCL-Session` header (up to 512 characters); with one copy per GPU (DP2) it helps the router reuse the conversation cache. ## Chat request parameters | Parameter | Models | Notes | | -------------------------------------------------------------------------------- | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `model` | all | The model's `nvcl-ai` name (see [Supported models](/reference/models.md)); another name gets 404 | | `messages` | all | `system`, `developer`, `user`, `assistant` (with `tool_calls` and `reasoning_content` or `reasoning`), `tool`. Text (see [Supported models](/reference/models.md)) | | `max_tokens`, `max_completion_tokens` | all | Default: the rest of the context, as vLLM | | `temperature`, `top_p`, `top_k`, `seed` | Qwen3.8-27B, DeepSeek-V4.1-Flash, GLM-5.3 | `temperature` 0 is greedy. `top_k` -1 or 0 turns it off; its range differs by model (see [Per model](#per-model)) | | `n` | all | 1 to 64 answers; with a `seed`, answer *i* samples with seed + *i*. `best_of` equal to `n` is accepted | | `stop`, `include_stop_str_in_output` | all | Stop strings; the second is vLLM's | | `ignore_eos` | all | vLLM's: generate to the length limit | | `stream`, `stream_options.include_usage` | all | Server-sent events; the usage chunk on request | | `chat_template_kwargs.enable_thinking` | all | Thinking on or off | | `thinking: {"type": "enabled" \| "disabled"}` | all | DeepSeek-style switch; wins over `enable_thinking` | | `reasoning_effort`, `chat_template_kwargs.reasoning_effort` | all | Values differ by model | | `tools`, `tool_choice`, `parallel_tool_calls` | Qwen3.8-27B, DeepSeek-V4.1-Flash, GLM-5.3 | `tool_choice` `auto` or `none`; on Qwen3.8-27B on NVIDIA GPUs also `required` or a named function, and tools marked `strict` follow their schema. `parallel_tool_calls: false` keeps one call | | `response_format` | Qwen3.8-27B on NVIDIA GPUs | `text`, `json_object` or `json_schema` (the answer follows the schema). With thinking on, the reasoning stays free and the schema applies to the answer | | `stop_token_ids` | Qwen3.8-27B | vLLM's: more stop tokens | | `thinking_token_budget` | Qwen3.8-27B on NVIDIA GPUs | vLLM's: caps the reasoning length; enforced | | `min_p` (0 to 1), `min_tokens` | Qwen3.8-27B (`min_tokens` on NVIDIA GPUs only) | `min_tokens` is vLLM's | | `logit_bias` | Qwen3.8-27B on NVIDIA GPUs | Up to 1,024 token ids, values -100 to 100; up to 64 conversations with a bias at once | | `prompt_logprobs` (0-20) | Qwen3.8-27B | The whole prompt is read again, without the conversation cache; not with `stream` | | `logprobs`, `top_logprobs` (0-20) | Qwen3.8-27B, one GPU | Raw logprobs before penalties, as vLLM's default | | `presence_penalty`, `frequency_penalty` (-2 to 2), `repetition_penalty` (0 to 2] | Qwen3.8-27B, one NVIDIA GPU | vLLM's formula; up to 32 conversations with penalties at once | ## Per model Qwen3.8-Flash-Next runs on the same server code as Qwen3.8-27B and takes the same request controls. | | Qwen3.8-27B | DeepSeek-V4.1-Flash | GLM-5.3 | | --------------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | Thinking on/off per request | yes | yes | yes | | Reasoning effort | `low`, `medium`, `xhigh` (`high` = `xhigh`, `minimal` = `low`) | `low`, `medium`, `high`, `max`, or a number from 1 to 100 (a hint, not a hard limit) | `low`, `high`; `minimal`, `medium`, `xhigh`, `max` = max; other values refused | | Default | thinking on, `xhigh` | thinking on, `high` | thinking on, `max` (`high` with thinking off) | | Tool calls | Qwen3 format | DeepSeek's own format | GLM's own format | | `top_k` when sampling | 1 to 64 (above 64 refused); off samples from the 64 most likely tokens | off only: samples from all tokens | off only: samples from all tokens under `top_p` | ## Text completions `POST /v1/completions` takes `prompt` as a text, a list of texts, a list of token ids, or a list of such lists. Token ids run exactly as sent, so a benchmark can send every engine the same prompt tokens. The answer is raw text in `choices[i].text`, including the model's `` tags. Qwen3.8-27B. * Prompts × `n` up to 64. * `max_tokens` defaults to 16, as vLLM's completions; `null` means the rest of the context. * `temperature`, `top_p`, `top_k`, `seed`, `ignore_eos`, `stop`, `stop_token_ids`, `logprobs` work as in chat. * `echo` puts the prompt before the answer; with `logprobs` it adds the prompt's logprobs, and with `max_tokens` 0 and `logprobs` it scores the prompt without writing an answer, as log-likelihood evaluations do. * `usage.prompt_tokens` counts each prompt once. ## Refused with 400 The server refuses what the engine cannot honour, rather than ignoring it: * `best_of` other than `n`, more than 64 answers; * `response_format` other than `text`, `json_object` or `json_schema`, and vLLM's older guided-decoding fields; * on DeepSeek-V4.1-Flash and GLM-5.3: what the tables above give for Qwen3.8-27B only, and `tool_choice` `required` or a named function; * predicted outputs, audio, web search, moderation and chat-template overrides; * on `/v1/completions`: `suffix`, ids outside the vocabulary; * on the Responses and Anthropic Messages APIs: hosted and server-side tools (web search, file search, code interpreter, MCP), files and documents, Anthropic structured output. Fields that change nothing (`user`, `metadata`, caching hints) and fields the server does not know are ignored, as vLLM does. A prompt that does not fit `max_model_len`, or a `max_tokens` that does not fit after the prompt, gets 400 with the code `context_length_exceeded`; the message opens with vLLM's sentence. ## Answers and usage | Field | Meaning | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `choices[].message.content` | The answer | | `choices[].message.reasoning_content`, `choices[].message.reasoning` | The reasoning, in both fields | | `choices[].message.tool_calls` | Parsed tool calls (Qwen3.8-27B, DeepSeek-V4.1-Flash, GLM-5.3) | | `choices[].logprobs` | With `logprobs` | | `finish_reason` | `stop` (stop token or string), `tool_calls`, `length` (cut at `max_tokens`, at the end of the context or by `ignore_eos`) | | `usage.prompt_tokens`, `completion_tokens`, `total_tokens` | Token counts | | `usage.prompt_tokens_details.cached_tokens` | Prompt tokens served by the prefix cache | | `usage.completion_tokens_details.reasoning_tokens` | Output tokens inside the reasoning | Answers from `/v1/responses` and `/v1/messages` follow the OpenAI Responses and Anthropic Messages formats, streams included. The server keeps the last 256 Responses answers in memory, so a client can continue with `previous_response_id`; `store: false` opts out, and a restart forgets them. In a stream, an error before the first token is an ordinary JSON error with its HTTP status; after it, an `error` event followed by `[DONE]`. A client that closes the stream cancels its request.