Skip to content
Docs menu

Operations - probes, metrics, local models and mesh

Monitor the orchestrator with health probes, Prometheus metrics and the profiler, manage local models, connect it to Venom Cloud, and replay traffic from cassettes.

View .md

Health probes

Endpoint Auth Answer
GET /healthz none Liveness: 200 {"status":"ok"} while the process serves.
GET /readyz none 200 with at least one healthy upstream; 503 {"status":"no healthy upstream"} otherwise, or "checking upstreams" with checking_upstreams > 0 right after a start or reload while upstreams await their first probe.

An upstream counts as unhealthy until its first probe passes. A probe that only timed out keeps a healthy upstream healthy if live traffic got a 2xx from it since the last probe, otherwise it takes two in a row; an error status or a refused connection counts at once. In Kubernetes, point the liveness probe at /healthz and readiness at /readyz.

GET /health and /metrics on port 9797 belong to the separate public-mesh auditor, not the orchestrator (Environment variables).

Metrics

GET /metrics serves Prometheus text format 0.0.4, no auth. From a checkout, make up SERVICES="orchestrator prometheus grafana" runs Prometheus (http://localhost:9090) and Grafana (http://localhost:3001) with a provisioned dashboard.

Metric Labels
venom_requests_total model, upstream, code (0 = no upstream reached, 499 = client closed)
venom_request_duration_seconds, venom_ttft_seconds model
venom_tokens_total model, type (prompt, completion, cached, cache_write)
venom_cost_usd_total model
venom_cache_hit_ratio, venom_cache_breaks_total model, reason
venom_wasted_tokens_total, venom_wasted_usd_total model, reason (abandoned, ttft_timeout, client_disconnect, retried)
venom_cost_anomaly, venom_cost_anomalies_total scope, metric
venom_response_cache_hits_total, venom_response_cache_stores_total kind, how
venom_dedup_total how (joined, replayed, lagged, alone, mismatch)
venom_tool_call_repairs_total upstream, kind
venom_upstream_healthy, venom_upstream_inflight, venom_upstream_circuit_state (0 closed, 1 open, 2 half open), venom_upstream_latency_ewma_seconds upstream
venom_upstream_attempts_total, venom_upstream_retries_skipped_total, venom_upstream_retry_budget_available upstream, attempt or reason
venom_upstream_quota_exhausted upstream, model
venom_upstream_requests_waiting, venom_upstream_kv_cache_usage upstream (engine load)
venom_upstream_shaping_queue_depth, venom_upstream_shaping_requests_total, venom_upstream_shaping_rejected_total, venom_upstream_shaping_wait_seconds upstream, outcome or reason
venom_route_target_available route, upstream, model
venom_key_spend_usd, venom_key_budget_usd, venom_budget_soft_limit_crossed_total, venom_budget_exceeded_total key, key_id, period, limit
venom_request_log_rows, venom_request_log_dropped_total, venom_active_streams, venom_uptime_seconds none
go_goroutines, go_heap_objects_bytes, go_memory_total_bytes, go_gc_cycles_total, process_open_fds, process_cpu_seconds_total none

model is the requested alias or id; unknown names are recorded as (unknown), so label cardinality is bounded by the config.

Profiler

Go's profiler is served behind the admin token at /admin/v1/debug/pprof/: the standard net/http/pprof paths (the /debug/pprof/ index, /debug/pprof/profile, /debug/pprof/trace, /debug/pprof/block, /debug/pprof/mutex, /debug/pprof/cmdline, /debug/pprof/symbol, plus heap, allocs, goroutine) under the /admin/v1 prefix. Block and mutex sampling are switched on only for the duration of a ?seconds=N profile.

curl -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -o cpu.pb.gz \
  'http://127.0.0.1:8787/admin/v1/debug/pprof/profile?seconds=10'
go tool pprof -top cpu.pb.gz

Tracing and logs

Set OTEL_EXPORTER_OTLP_ENDPOINT to export a span per request over OTLP/HTTP; a client that sends traceparent (the venom CLI does) gets the orchestrator's span joined to its trace. Spans never carry prompt or reply content. Logs go to stderr; LOG_LEVEL and LOG_JSON=1 set the level and format.

Model catalog and guardrail pages

  • GET /api/models/providers/{id} (no key, not under /v1) serves <VENOM_MODEL_CATALOG>/<id>.json, a vetted model-catalog shard, with a strong ETag (If-None-Match → 304). An unknown id or no directory is 404.
  • GET /g/{code} (no key) is a guardrail code's page, where a 403 guardrail_blocked and the x-venom-guardrail header link. HTML, or JSON with Accept: application/json. Without the admin token it shows only the public summary.

Local models

/admin/v1/local/* is what the console's Local models page and venom models drive: this machine's hardware, which models fit, and pulling, selecting and serving them. Nothing here is on the /v1 path.

Endpoint What it does
GET /admin/v1/local Hardware, llmfit status, backends (Ollama, vLLM: reachable, models), the current selection per tier, strain and running pulls.
GET /admin/v1/local/catalog[?refresh=1] Recommended models for this hardware (default and fast tier) and the catalog with fit estimates.
GET /admin/v1/local/suggest[?days=7&gpu_usd_per_hour=X] Which cloud side-role traffic (classifier, summary, reflex, workers) a local model could take, and the savings.
POST /admin/v1/local/pull {model, select?: "default"|"fast"}: start (or join) an Ollama pull; 202.
POST /admin/v1/local/select {model, tier: "default"|"fast"}: put that model behind venom-default or venom-fast (sets VENOM_OLLAMA_MODEL or VENOM_OLLAMA_FAST_MODEL and reloads; kept in $VENOM_DATA_DIR/local-models.json).
POST /admin/v1/local/setup Pull and select the recommended models in one step; 202.
POST /admin/v1/local/serve {model}: one-click vLLM in Docker (VENOM_VLLM_IMAGE); 202.
DELETE /admin/v1/local/serve Stop it; 404 not_running.
DELETE /admin/v1/local/models/{model} Delete an Ollama model; 409 selected while it is selected.
POST /admin/v1/local/unload {model}: free its VRAM now.

Hardware and fit come from llmfit (VENOM_LLMFIT_URL, else VENOM_LLMFIT_BIN); without it the catalog still works.

Mesh mode

In mesh mode the orchestrator is a Venom Cloud daemon: it dials out to the cloud, serves the cloud's jobs with its own upstreams, and keeps provider keys local. Turn it on with VENOM_CLOUD_URL and a credential, or from the dashboard:

orchestrator mesh join https://cloud.usevenom.dev -      # reads the vjt_… join token from stdin
orchestrator mesh status
orchestrator mesh rotate
orchestrator mesh leave
Endpoint What it does
GET /admin/v1/mesh source (env, dashboard, none), state (off, connecting, connected, revoked, error), cloud URL, name, ids, region, paused, running jobs and tokens_per_s.
POST /admin/v1/mesh/connect {join_token, cloud_url?, name?, region?}: enroll and start mesh mode without a restart (cloud_url defaults to https://cloud.usevenom.dev). 409 mesh_managed_by_env when VENOM_CLOUD_URL controls it, 409 mesh_already_connected, 502 mesh_join_failed.
POST /admin/v1/mesh/disconnect Drain running jobs, deregister, remove the credential.
POST /admin/v1/mesh/pause, POST /admin/v1/mesh/resume Stay connected but take no new jobs (kept across restarts); 409 mesh_off with no connection.

The mesh variables are listed in Environment variables.

Desktop daemon

When venom-daemon runs the orchestrator, these drive its dashboard and menu bar app (404 not_managed otherwise):

Endpoint What it does
GET /admin/v1/daemon managed, version, OS, start-at-login state, whether Ollama is installed and where to get it.
PUT /admin/v1/daemon/autostart {enabled}: add or remove the login item.
GET /admin/v1/daemon/projects The project folders and workspaces and whether venom serve runs: {folders: [{path, exists}], max, serve: {running, url?, error?}, managed, workspaces?}.
POST /admin/v1/daemon/projects {path}: add a folder and restart venom serve. 400 invalid_project.
DELETE /admin/v1/daemon/projects?path= Remove a folder and restart venom serve. 404 project_not_found.
DELETE /admin/v1/daemon/projects?workspace= Delete a workspace and its files. 404 workspace_not_found.
GET /admin/v1/daemon/update Auto-update status: auto_update, state, current and latest version, last and next check, progress. No network.
PUT /admin/v1/daemon/update {auto_update}: off by default; on, checks run about every 6 hours.
POST /admin/v1/daemon/update/check One fetch of the releases' SHA256SUMS; 502 check_failed.
POST /admin/v1/daemon/update/apply Download, verify, wait for an idle window, swap and restart (or roll back); 202. 409 up_to_date, update_busy, not_supported.
GET /admin/v1/heal?limit=50 The self-heal supervisor: state (ok, healing, degraded), failing components with attempts and next try, and recent repair events. 404 not_available without one.

Cassettes

VENOM_CASSETTE_MODE=record writes every complete 200 the orchestrator relays to $VENOM_CASSETTE_DIR/<key>.json (status, headers, exact body bytes, plus every attempt made for it with secrets redacted). VENOM_CASSETTE_MODE=replay answers from those files and never reaches an upstream: nothing is spent and the request log gets a zero-cost row; a replayed answer carries x-venom-cassette: replay. A request with no cassette is 409 cassette_miss. The key is a hash of the endpoint, route profile, Accept and the normalised body (not the client key), so cassettes are portable and can be committed for deterministic tests of an app built on venom. Failures, partial streams and audio are not recorded.