> For the complete documentation index, see [llms.txt](https://docs.nvcl.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nvcl.ai/reference/monitoring.md).

# Monitoring

Health checks and Prometheus metrics of the server that `nvcl-ai` starts.

## Health

```bash
curl -s localhost:8000/health
```

| Answer   | Meaning                                           |
| -------- | ------------------------------------------------- |
| 200 `ok` | The model is loaded and serving                   |
| 503      | The model is still loading, or the engine stopped |

With a full copy of the model on each card (DP2), `/health` answers 503 until both copies have loaded; if one copy stops later, it answers 200 `ok (degraded)` while the other serves.

8000 is the default port; `nvcl-ai status` shows the address. `/v1/models` also answers 503 while the model loads. `/health`, `/metrics` and `/version` answer without the API key, so a load balancer or Prometheus can poll them.

A hang watchdog guards the engine: if it makes no progress for 180 s while requests are in flight, the server ends those requests with an error instead of leaving clients waiting, and stops. While a model loads, the limit without progress is longer: `nvcl-ai` sets it by the model's size, at least 30 minutes. A model started at boot (`--boot`) then restarts by itself; otherwise `nvcl-ai start` starts it again. `nvcl-ai logs` shows what happened.

## Prometheus metrics

`GET /metrics` returns Prometheus text. The names follow vLLM's, with the prefix `nvcl:`.

| Metric                                                                                                                       | What it counts                                                                                                             |
| ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `nvcl:num_requests_running`, `nvcl:num_requests_waiting`                                                                     | Requests holding a slot now; requests waiting for one                                                                      |
| `nvcl:request_success_total`, `nvcl:request_failure_total`, `nvcl:request_cancelled_total`                                   | Finished, refused or failed, and cancelled-by-client requests                                                              |
| `nvcl:prompt_tokens_total`, `nvcl:generation_tokens_total`, `nvcl:reasoning_tokens_total`                                    | Prompt tokens, generated tokens, and generated tokens inside the reasoning                                                 |
| `nvcl:prompt_tokens_cached_total`                                                                                            | Prompt tokens the prefix cache served; divide by `prompt_tokens_total` for the hit rate                                    |
| `nvcl:prompt_tokens_cache_lost_total`                                                                                        | Prompt tokens a bigger prefix cache would have served                                                                      |
| `nvcl:num_preemptions_total`                                                                                                 | Requests paused to free attention-cache memory                                                                             |
| `nvcl:spec_decode_num_cycles_total`, `nvcl:spec_decode_num_draft_tokens_total`, `nvcl:spec_decode_num_accepted_tokens_total` | Draft steps, proposed draft tokens and accepted draft tokens; accepted ÷ steps is the extra tokens a step gives on average |

Histograms, in seconds:

| Histogram                           | Measures                           |
| ----------------------------------- | ---------------------------------- |
| `nvcl:time_to_first_token_seconds`  | Request accepted to first token    |
| `nvcl:request_queue_time_seconds`   | Time waiting for a slot            |
| `nvcl:request_prefill_time_seconds` | Reading the prompt                 |
| `nvcl:request_kv_wait_time_seconds` | Waiting for attention-cache memory |
| `nvcl:request_resume_time_seconds`  | Resuming a paused request          |
| `nvcl:inter_token_latency_seconds`  | Mean time between tokens           |
| `nvcl:e2e_request_latency_seconds`  | Request accepted to finished       |

Two more histograms count tokens per request: `nvcl:request_prompt_tokens`, `nvcl:request_generation_tokens`.

A scrape config:

```yaml
scrape_configs:
  - job_name: nvcl
    static_configs:
      - targets: ["localhost:8000"]
```