Observability with Prometheus and Grafana
Run Prometheus and a provisioned Grafana dashboard next to the orchestrator with the observability compose profile, or scrape the orchestrator's /metrics from your own monitoring.
The orchestrator serves Prometheus metrics at GET /metrics (no auth, the same port as /v1). A venom checkout ships a Prometheus scrape config and a Grafana dashboard for it under deploy/observability/, wired into compose behind the observability profile.
Start it
make up SERVICES="orchestrator prometheus grafana"
open http://127.0.0.1:3001 # Grafana, the "Venom orchestrator" dashboard
open http://127.0.0.1:9090 # PrometheusOr with compose directly: docker compose --profile observability up -d. Both bind to 127.0.0.1 only. Grafana opens anonymously as a Viewer on the provisioned dashboard; sign in as admin with GRAFANA_ADMIN_PASSWORD to edit.
Settings
| Variable | Default | What it does |
|---|---|---|
PROMETHEUS_PORT |
9090 |
Prometheus's host port. |
PROMETHEUS_RETENTION |
15d |
How long Prometheus keeps samples (--storage.tsdb.retention.time). |
GRAFANA_PORT |
3001 |
Grafana's host port. |
GRAFANA_ADMIN_PASSWORD |
admin |
Grafana's admin password. Change it before you publish Grafana anywhere. |
Data lives in the prometheus-data and grafana-data volumes; make down keeps them.
Grafana runs with usage reporting, update checks, plugin update checks, the news feed and plugin preinstalls turned off, so it makes no outbound calls.
What is scraped
deploy/observability/prometheus.yml scrapes every 5 seconds:
| Job | Target | Notes |
|---|---|---|
venom-orchestrator |
orchestrator:8787/metrics |
Every orchestrator metric. |
vllm |
vllm:8000/metrics |
vLLM's prefix-cache counters for the "Engine prefix cache" panel. Shows as down while the gpu profile is off; that is expected. |
SGLang has no compose service. For one on the host started with --enable-metrics, add a job:
- job_name: sglang
static_configs:
- targets: ["host.docker.internal:30000"]The dashboard
The provisioned "Venom orchestrator" dashboard (Grafana's home dashboard, read from deploy/observability/grafana/dashboards/venom.json) has: requests per second, error rate, p95 latency, p95 time to first token, cache hit ratio, active streams, request rate by model, error rate by status, latency and TTFT p50/p95, tokens per second, the upstream circuit breaker, upstream health and in-flight requests, and the vLLM engine prefix cache. The data source is provisioned as Prometheus (uid venom-prometheus) and is not editable from the UI.
Your own monitoring
Point any Prometheus-compatible scraper at the orchestrator:
scrape_configs:
- job_name: venom-orchestrator
metrics_path: /metrics
static_configs:
- targets: ["venom.example.com:8787"]Every metric and its labels are listed in Orchestrator operations. Label cardinality is bounded by your config: unknown model names are recorded as (unknown). On Kubernetes the same endpoint is on the venom-orchestrator Service (Kubernetes). Traces go to your own OpenTelemetry collector with OTEL_EXPORTER_OTLP_ENDPOINT (Orchestrator operations).
Related
- Docker Compose and profiles
- Orchestrator operations: probes, metrics, the profiler and tracing.