Operations - probes, metrics, local models and mesh
Monitor the orchestrator with health probes, Prometheus metrics and the profiler, manage local models, connect it to Venom Cloud, and replay traffic from cassettes.
Health probes
| Endpoint | Auth | Answer |
|---|---|---|
GET /healthz |
none | Liveness: 200 {"status":"ok"} while the process serves. |
GET /readyz |
none | 200 with at least one healthy upstream; 503 {"status":"no healthy upstream"} otherwise, or "checking upstreams" with checking_upstreams > 0 right after a start or reload while upstreams await their first probe. |
An upstream counts as unhealthy until its first probe passes. A probe that only timed out keeps a healthy upstream healthy if live traffic got a 2xx from it since the last probe, otherwise it takes two in a row; an error status or a refused connection counts at once. In Kubernetes, point the liveness probe at /healthz and readiness at /readyz.
GET /health and /metrics on port 9797 belong to the separate public-mesh auditor, not the orchestrator (Environment variables).
Metrics
GET /metrics serves Prometheus text format 0.0.4, no auth. From a checkout, make up SERVICES="orchestrator prometheus grafana" runs Prometheus (http://localhost:9090) and Grafana (http://localhost:3001) with a provisioned dashboard.
| Metric | Labels |
|---|---|
venom_requests_total |
model, upstream, code (0 = no upstream reached, 499 = client closed) |
venom_request_duration_seconds, venom_ttft_seconds |
model |
venom_tokens_total |
model, type (prompt, completion, cached, cache_write) |
venom_cost_usd_total |
model |
venom_cache_hit_ratio, venom_cache_breaks_total |
model, reason |
venom_wasted_tokens_total, venom_wasted_usd_total |
model, reason (abandoned, ttft_timeout, client_disconnect, retried) |
venom_cost_anomaly, venom_cost_anomalies_total |
scope, metric |
venom_response_cache_hits_total, venom_response_cache_stores_total |
kind, how |
venom_dedup_total |
how (joined, replayed, lagged, alone, mismatch) |
venom_tool_call_repairs_total |
upstream, kind |
venom_upstream_healthy, venom_upstream_inflight, venom_upstream_circuit_state (0 closed, 1 open, 2 half open), venom_upstream_latency_ewma_seconds |
upstream |
venom_upstream_attempts_total, venom_upstream_retries_skipped_total, venom_upstream_retry_budget_available |
upstream, attempt or reason |
venom_upstream_quota_exhausted |
upstream, model |
venom_upstream_requests_waiting, venom_upstream_kv_cache_usage |
upstream (engine load) |
venom_upstream_shaping_queue_depth, venom_upstream_shaping_requests_total, venom_upstream_shaping_rejected_total, venom_upstream_shaping_wait_seconds |
upstream, outcome or reason |
venom_route_target_available |
route, upstream, model |
venom_key_spend_usd, venom_key_budget_usd, venom_budget_soft_limit_crossed_total, venom_budget_exceeded_total |
key, key_id, period, limit |
venom_request_log_rows, venom_request_log_dropped_total, venom_active_streams, venom_uptime_seconds |
none |
go_goroutines, go_heap_objects_bytes, go_memory_total_bytes, go_gc_cycles_total, process_open_fds, process_cpu_seconds_total |
none |
model is the requested alias or id; unknown names are recorded as (unknown), so label cardinality is bounded by the config.
Profiler
Go's profiler is served behind the admin token at /admin/v1/debug/pprof/: the standard net/http/pprof paths (the /debug/pprof/ index, /debug/pprof/profile, /debug/pprof/trace, /debug/pprof/block, /debug/pprof/mutex, /debug/pprof/cmdline, /debug/pprof/symbol, plus heap, allocs, goroutine) under the /admin/v1 prefix. Block and mutex sampling are switched on only for the duration of a ?seconds=N profile.
curl -H "Authorization: Bearer $VENOM_ADMIN_TOKEN" -o cpu.pb.gz \
'http://127.0.0.1:8787/admin/v1/debug/pprof/profile?seconds=10'
go tool pprof -top cpu.pb.gzTracing and logs
Set OTEL_EXPORTER_OTLP_ENDPOINT to export a span per request over OTLP/HTTP; a client that sends traceparent (the venom CLI does) gets the orchestrator's span joined to its trace. Spans never carry prompt or reply content. Logs go to stderr; LOG_LEVEL and LOG_JSON=1 set the level and format.
Model catalog and guardrail pages
GET /api/models/providers/{id}(no key, not under/v1) serves<VENOM_MODEL_CATALOG>/<id>.json, a vetted model-catalog shard, with a strongETag(If-None-Match→304). An unknown id or no directory is404.GET /g/{code}(no key) is a guardrail code's page, where a403 guardrail_blockedand thex-venom-guardrailheader link. HTML, or JSON withAccept: application/json. Without the admin token it shows only the public summary.
Local models
/admin/v1/local/* is what the console's Local models page and venom models drive: this machine's hardware, which models fit, and pulling, selecting and serving them. Nothing here is on the /v1 path.
| Endpoint | What it does |
|---|---|
GET /admin/v1/local |
Hardware, llmfit status, backends (Ollama, vLLM: reachable, models), the current selection per tier, strain and running pulls. |
GET /admin/v1/local/catalog[?refresh=1] |
Recommended models for this hardware (default and fast tier) and the catalog with fit estimates. |
GET /admin/v1/local/suggest[?days=7&gpu_usd_per_hour=X] |
Which cloud side-role traffic (classifier, summary, reflex, workers) a local model could take, and the savings. |
POST /admin/v1/local/pull |
{model, select?: "default"|"fast"}: start (or join) an Ollama pull; 202. |
POST /admin/v1/local/select |
{model, tier: "default"|"fast"}: put that model behind venom-default or venom-fast (sets VENOM_OLLAMA_MODEL or VENOM_OLLAMA_FAST_MODEL and reloads; kept in $VENOM_DATA_DIR/local-models.json). |
POST /admin/v1/local/setup |
Pull and select the recommended models in one step; 202. |
POST /admin/v1/local/serve |
{model}: one-click vLLM in Docker (VENOM_VLLM_IMAGE); 202. |
DELETE /admin/v1/local/serve |
Stop it; 404 not_running. |
DELETE /admin/v1/local/models/{model} |
Delete an Ollama model; 409 selected while it is selected. |
POST /admin/v1/local/unload |
{model}: free its VRAM now. |
Hardware and fit come from llmfit (VENOM_LLMFIT_URL, else VENOM_LLMFIT_BIN); without it the catalog still works.
Mesh mode
In mesh mode the orchestrator is a Venom Cloud daemon: it dials out to the cloud, serves the cloud's jobs with its own upstreams, and keeps provider keys local. Turn it on with VENOM_CLOUD_URL and a credential, or from the dashboard:
orchestrator mesh join https://cloud.usevenom.dev - # reads the vjt_… join token from stdin
orchestrator mesh status
orchestrator mesh rotate
orchestrator mesh leave| Endpoint | What it does |
|---|---|
GET /admin/v1/mesh |
source (env, dashboard, none), state (off, connecting, connected, revoked, error), cloud URL, name, ids, region, paused, running jobs and tokens_per_s. |
POST /admin/v1/mesh/connect |
{join_token, cloud_url?, name?, region?}: enroll and start mesh mode without a restart (cloud_url defaults to https://cloud.usevenom.dev). 409 mesh_managed_by_env when VENOM_CLOUD_URL controls it, 409 mesh_already_connected, 502 mesh_join_failed. |
POST /admin/v1/mesh/disconnect |
Drain running jobs, deregister, remove the credential. |
POST /admin/v1/mesh/pause, POST /admin/v1/mesh/resume |
Stay connected but take no new jobs (kept across restarts); 409 mesh_off with no connection. |
The mesh variables are listed in Environment variables.
Desktop daemon
When venom-daemon runs the orchestrator, these drive its dashboard and menu bar app (404 not_managed otherwise):
| Endpoint | What it does |
|---|---|
GET /admin/v1/daemon |
managed, version, OS, start-at-login state, whether Ollama is installed and where to get it. |
PUT /admin/v1/daemon/autostart |
{enabled}: add or remove the login item. |
GET /admin/v1/daemon/projects |
The project folders and workspaces and whether venom serve runs: {folders: [{path, exists}], max, serve: {running, url?, error?}, managed, workspaces?}. |
POST /admin/v1/daemon/projects |
{path}: add a folder and restart venom serve. 400 invalid_project. |
DELETE /admin/v1/daemon/projects?path= |
Remove a folder and restart venom serve. 404 project_not_found. |
DELETE /admin/v1/daemon/projects?workspace= |
Delete a workspace and its files. 404 workspace_not_found. |
GET /admin/v1/daemon/update |
Auto-update status: auto_update, state, current and latest version, last and next check, progress. No network. |
PUT /admin/v1/daemon/update |
{auto_update}: off by default; on, checks run about every 6 hours. |
POST /admin/v1/daemon/update/check |
One fetch of the releases' SHA256SUMS; 502 check_failed. |
POST /admin/v1/daemon/update/apply |
Download, verify, wait for an idle window, swap and restart (or roll back); 202. 409 up_to_date, update_busy, not_supported. |
GET /admin/v1/heal?limit=50 |
The self-heal supervisor: state (ok, healing, degraded), failing components with attempts and next try, and recent repair events. 404 not_available without one. |
Cassettes
VENOM_CASSETTE_MODE=record writes every complete 200 the orchestrator relays to $VENOM_CASSETTE_DIR/<key>.json (status, headers, exact body bytes, plus every attempt made for it with secrets redacted). VENOM_CASSETTE_MODE=replay answers from those files and never reaches an upstream: nothing is spent and the request log gets a zero-cost row; a replayed answer carries x-venom-cassette: replay. A request with no cassette is 409 cassette_miss. The key is a hash of the endpoint, route profile, Accept and the normalised body (not the client key), so cassettes are portable and can be committed for deterministic tests of an app built on venom. Failures, partial streams and audio are not recorded.