Public mesh hardening
How the public mesh checks that a daemon run by a stranger serves the registered model at the promised quality, what was measured, and what is still weak.
On the public mesh, anyone can host a daemon and earn per token. This page summarises how venom checks such a daemon and where the checks are still weak. The normative contract is docs/public-mesh.md and the decision record is ADR 0023; where they differ from this summary, they win.
The threat
A public daemon's operator can earn the same money more cheaply by serving a smaller model, a coarser quantisation or a cheaper engine setting (an FP8 KV cache); by editing, truncating or caching prompts and answers; by inflating token counts; by faking latency to attract traffic; or by behaving only when it thinks it is tested. The daemon's code is open, so no check may depend on a secret the daemon can see.
The pipeline
- Verifiable sampling (
venom-sampler/1). Every public request carries a secret 16-byte nonce. The engine picks each token as the Gumbel-max over the top-k/top-p candidates, with noise fromHMAC-SHA256(nonce, position ‖ token). Given the model's log-probabilities, anyone can replay the choice. A vLLM logits-processor plugin implements it. - Receipts. The daemon signs a receipt binding the prompt token hash, the output token ids, the streamed text and the counts.
- Audits. An auditor replays sampled jobs on reference hardware and compares the daemon's tokens with the ones the registered model would pick.
- Sequential evidence. One audit is tuned to a one-in-a-million false-positive rate, which makes it blind to an artifact that is only slightly wrong. The auditor therefore accumulates evidence across audits (a CUSUM-style statistic), so a small, steady excess, such as a one-step-coarser quantisation, is caught over many jobs.
- Witnessed latency. The cloud stamps time to first token on the first streamed line that carries a real output token, so an empty opening chunk sent at once no longer looks fast. A "first-token feint" (streaming a guessed opener before prefill ends, then sampling honestly) is caught by the sequential evidence at its second audited occurrence.
- Preflight. A registry entry can carry up to 8 golden cases. At start-up and every 6 hours the daemon runs them through its public-job path, locally, and advertises a model for public traffic only when they pass. The cloud never sees a preflight.
- llama.cpp without a plugin. A shim (
venom-llamacpp) puts llama.cpp behind the sampler contract so it can be audited too.
Known shortfalls
- Verdict results remain visible to the daemon after each epoch, so a daemon learns that it failed an audit, though not how close it is to the alarm.
- A non-streamed job's time to first token is its whole duration; a stream padded with over 64 KiB of tokenless data falls back to the done time and counts as an SLO miss, not as abuse.
- The preflight's thresholds come from a toy model and have not been run against real vLLM or llama.cpp.
- The llama.cpp path replays every prompt position, which is slow on long prompts, and approximates the rank of rare prompt tokens.
- Region verification, an attested trust class, payouts and KYC are out of scope.
Reproducing
In a venom checkout, the adversarial toy suite is go test ./internal/audit -run Adversarial -v in services/go-orchestrator, and the shared vectors run in go test ./internal/provenance -run Cusum and make test-cloud. make public-mesh-e2e runs the whole public mesh on loopback.