Skip to content

Compare

How Venom compares, with a source for every line

First the workspace next to the tools it replaces. Then the venom CLI, which runs any model you point it at with no account and no telemetry, next to Claude Code and Qwen Code, and the daemon, which puts all your backends behind one endpoint for 49 µs a request at p50, next to Ollama and vLLM.

The workspace

Instead of Notion, Linear, Calendly and the rest

One workspace, priced per organisation and by usage, where agents work under the same roles as people. Tool names only, so you can find yours; each row links the docs for the Venom side.

Venom products next to the tools they replace
Notion, ConfluenceLinked notes, a wiki and databases. Imports a Notion export or a Confluence space with links kept. Docs
Linear, JiraIssues, projects and cycles, with work handed to agents. Imports from Linear and Jira. Docs
CalendlyBooking pages on your real availability across every calendar, plus team event types and routing forms. Docs
HubSpot, PipedriveContacts, deals, a forecast and follow-up sequences. Imports HubSpot and Pipedrive contacts. Docs
ZendeskA shared inbox for customer email, with replies an agent drafts from your notes and a person approves. Docs
SlackChannels, DMs and threads where agents are members, with the rest of Venom posting into it. Docs
Typeform, TallyForms with conditional logic, shared by link, with answers sent where the team works. Docs
Sentry, PagerDuty, StatuspageError tracking that keeps your Sentry SDKs, on-call schedules, and a status page fed by your own checks. Docs
LaunchDarkly, OptimizelyFeature flags with percentage rollouts, and A/B tests read on those flags. Docs
An answering serviceAn AI agent that answers your number, and messages and calls through your own provider account. Docs

More on replacing your SaaS stack. For the model side, see running your own model mesh.

Coding agents

venom CLI, Claude Code and Qwen Code

All three have permission modes, plan mode, subagents, MCP and an OS sandbox. They differ in which models they run, what they send home and what they need from you. Other products' columns are from their own docs, read on 2026-10-18.

  • License1,2,3

    venom CLI
    MIT, free download
    Claude Code
    Proprietary, under Anthropic's Commercial Terms
    Qwen Code
    Apache-2.0
  • Models4,5,6,3

    venom CLI
    Any OpenAI-compatible endpoint: Ollama, vLLM, SGLang, llama.cpp, Venom Cloud or a provider
    Claude Code
    ~Claude models, through Anthropic or Amazon Bedrock, Google Cloud and Microsoft Foundry
    Qwen Code
    OpenAI, Anthropic, Gemini and Qwen APIs, any OpenAI-compatible endpoint, local models
  • Account needed1,5,3

    venom CLI
    None. Point it at a model and start
    Claude Code
    A Claude Pro, Max, Team, Enterprise or Console account, or a cloud provider
    Qwen Code
    A provider key, or a local model
  • With the network unplugged4,5

    venom CLI
    Works, with a local model
    Claude Code
    –Docs list an internet connection as required
    Qwen Code
    Not checked
  • Usage telemetry by default4,7,8

    venom CLI
    Off. No telemetry and no update checks
    Claude Code
    On with the Claude API (no code or prompts), off on cloud providers; one variable turns it off
    Qwen Code
    On; one setting turns it off
  • Automatic updates4,5,8

    venom CLI
    None. You choose when to upgrade
    Claude Code
    Native installs update in the background
    Qwen Code
    Checks and installs on startup by default
  • Permission modes and plan mode9,10,8

    venom CLI
    plan, default, accept-edits, yolo; each part of a compound shell command is judged on its own
    Claude Code
    default, acceptEdits, plan, auto, dontAsk, bypassPermissions
    Qwen Code
    plan, default, auto-edit, auto, yolo
  • OS sandbox for shell commands9,10,11

    venom CLI
    sandbox-exec on macOS, bubblewrap on Linux, with a per-host network allowlist; on whenever yolo runs bash
    Claude Code
    Seatbelt on macOS, bubblewrap on Linux and WSL2; off by default
    Qwen Code
    Seatbelt, Docker or Podman, Linux bwrap; opt-in
  • Subagents12,13,11

    venom CLI
    Explorer, worker, verifier and your own; workers can get their own git worktree
    Claude Code
    Qwen Code
  • Plugins and MCP14,13,11

    venom CLI
    pi packages and Claude Code plugins load as they are; MCP
    Claude Code
    Plugins, hooks, skills, MCP
    Qwen Code
    Extensions, including Claude Code marketplace plugins; MCP
  • Reads a Claude Code project as it is14,13

    venom CLI
    CLAUDE.md, .claude skills, commands, agents, settings, hooks, .mcp.json
    Claude Code
    Its own format
    Qwen Code
    Not checked
  • Prompt-cache stability15,13

    venom CLI
    Fixed prompt held under a 2,910-token budget a test enforces; the prefix is append-only and the same across sessions
    Claude Code
    Automatic prompt caching
    Qwen Code
    Not checked

“Not checked” means we couldn't confirm it in that product's docs, not that it is missing. More on the venom CLI.

Inference servers

The Venom daemon, or a server on its own

The daemon doesn't replace Ollama or vLLM. It sits in front of them, and of your provider keys, and adds routing, fallback, cache affinity and keys.

  • OpenAI-compatible API16,17,18

    Venom daemon
    Chat, responses, completions, embeddings and models, with streaming
    Ollama on its own
    ~A subset of the OpenAI API
    vLLM on its own
    Completions, chat and more
  • Several backends behind one endpoint16,17,19

    Venom daemon
    vLLM, SGLang, llama.cpp, Ollama and cloud providers, by model alias
    Ollama on its own
    Its own models, plus Ollama's hosted cloud models
    vLLM on its own
    One server; the vLLM project ships separate routers for many
  • Fallback and circuit breaking16,17,19

    Venom daemon
    Fallback before the first byte, circuit breakers per target
    Ollama on its own
    Not documented
    vLLM on its own
    ~In the separate vllm-project/router
  • Prefix-cache affinity16,20,17,18,19

    Venom daemon
    Session-sticky routing, and longest-prefix routing for new sessions
    Ollama on its own
    Not documented
    vLLM on its own
    ~Automatic prefix caching on an instance; session and cache-aware routing in the separate routers
  • Client keys, limits and budgets16,17

    Venom daemon
    Per-key rate and token limits and spend budgets, usage and cost in SQLite
    Ollama on its own
    The local API needs no authentication
    vLLM on its own
    Not checked
  • Listens on localhost by default16,17

    Venom daemon
    127.0.0.1
    Ollama on its own
    127.0.0.1:11434
    vLLM on its own
    Not checked
  • Reachable by your team without an inbound port21,17,18

    Venom daemon
    Optional: it dials out to Venom Cloud after you paste a join token
    Ollama on its own
    Not documented
    vLLM on its own
    Not documented

More on the daemon.

Speedups and optimizations

What we measured

These numbers come from our own benchmarks, evals and live runs, each with its conditions. They show what each change bought. They are not head-to-head runs against other agents, and we have none to show.

The agent

97.6%22

pass@1 on Venom's eval suite

42 tasks, 3 trials each: 95.2% passed all 3. The safety slice passed 18 of 18 with no permission-engine violations. 126 trials cost $1.23, with 74.4% of input tokens cached.

Self-authored tasks, gpt-5.4-mini through the orchestrator. Not a public benchmark.

206 ms23

to the first frame of the TUI

Bundling the CLI halved startup: 417 ms to 206 ms for the first frame, and 514 ms to 243 ms to the first model request.

Apple M2 Pro, medians of 15 runs.

37.5% → 100%24

long sessions in a 16k window

Compaction masks old tool output first, keeps pinned facts word for word and recovers from overflow, so long tasks finish in small local-model windows.

Eval longsession slice, k=2.

9/18 → 18/1825

prompt-injection trials passed

Outside content is spotlighted, and an advisory detector flags injection attempts.

Eval safety slice, k=3.

< 1 ms26

sandbox cost per command on Linux

Each sandboxed command gets its own empty /tmp and /run, which closed a socket leak and cut the cost from about 90 ms.

A Linux host with a busy /tmp.

0.42 s27

to update the code index

After 10 changed files in the Kubernetes repository, down from 60 s. A cold build went from 171 s to 45 s.

26,918 files, a machine loaded by other runs.

The orchestrator and the cloud

+49 µs28

added per request, p50

Auth, alias resolution, quirks, usage accounting and session affinity on the path. Streaming adds +65 µs p50, and nothing is buffered.

make bench on an Apple M2 Pro with a local fake upstream.

1,00029

concurrent streams, 0 errors

+0.4 ms TTFT at p50, throughput 0.997× of the upstream alone, 163 MB peak memory.

Fake vLLM upstream on a shared M2 Pro.

0% → 95%30

cached on a new session's first request

Sessions share a small pool of OpenAI cache keys, so a new session finds the common prompt prefix already cached.

Live, eight new sessions with the same 5k-token prefix.

0% → 88-95%31

prompt reused on llama.cpp

The orchestrator leaves slot choice to builds with the host-RAM prompt cache instead of pinning, so interleaved sessions keep their cache.

llama.cpp b11459, measured through the orchestrator.

75.5% → 79.7%20

prefix-cache hit rate, simulated

A new session goes to the backend that holds the longest prefix of its prompt, unless that backend is busy.

Simulation on 3 fake vLLM targets with tight caches.

7 µs32

for Venom Cloud to pick a daemon, p50

11 µs at p95 across 10 eligible daemons, 65 µs across 100. Routing is not where a request's time goes.

Wall clock on Node 24.

Smaller changes, also measured

  • Subagents: 16 agents at a concurrency cap of 16 finish twice as fast (11 to 22 tasks a second).33
  • apply_patch for GPT-5 models cut requests from 645 to 526 on 27 tasks and kept pass@1 at 100%.34
  • Handing a subagent a 42 KB doc by citation costs about 2.9k prompt tokens instead of 17.6k pasted.35
  • A compaction summary reuses the session's cached prefix: 5,923 of 7,138 prompt tokens cached, about 45% less input cost.36
  • Images go at the size the provider uses: 82% fewer image tokens on a local Qwen2.5-VL, 18% fewer on Anthropic.37
  • Test runners, grep and find output fold to summaries: 72% fewer characters, with failures kept verbatim.38
  • Checkpoints for /undo on a 30k-file workspace: a mutating turn went from about 740-890 ms to about 265-300 ms.39
  • A trigram index for grep in large repositories: nine patterns went from 4.9 s to 1.5 s.40
  • The injection detector and secret scrubs find their literals in one Aho-Corasick pass: 1.65 to 3.4 times faster, with identical findings.41

How it works

The techniques behind the numbers

What the agent and the router do differently, by name, with the result each one measured.

The agent

  1. Append-only prompt prefix

    Every turn is a cache hit on the turn before15,42,22

    The system prompt and tool list never change during a session, reminders ride on the newest message, and two sessions in one workspace send the same prefix. A test fails the build if the fixed prompt grows past its token budget, which is how a week of features was cut back from 4,243 tokens to 2,776.

    74.4%

    of input tokens cached across 126 eval trials; the fixed prompt is 35% smaller than at its peak

    Eval baseline on gpt-5.4-mini, and the prompt measured on main.

  2. Deferred MCP tool schemas

    A big MCP server doesn't cost you on every request43

    When a server's tool schemas would be large, venom sends only their names and a tool_search tool. The model loads the schemas it needs once, and the prefix stays the same.

    15,368 → 1,863

    tokens in the fixed prompt with a 60-tool MCP server (−88%)

    A fake MCP server of 60 GitHub-sized tools, in a test.

  3. Staged compaction

    Long tasks finish in a small local model's window24,36

    Old tool output is masked to a blob store first, pinned facts are carried word for word by code, and a summary is written only when masking isn't enough. The summary request re-sends the session's prefix byte for byte, so it is itself a cache read.

    37.5% → 100%

    long-session tasks passed in a 16k-token window

    Eval longsession slice, k=2, before and after.

  4. Layered patch matching

    Half as many failed edits44,45,34

    For GPT-5 models, venom edits with apply_patch. Each hunk is matched exactly, then without trailing whitespace, then without indentation, then with Unicode quotes and dashes folded, and a patch applies whole or rolls back.

    8.3% → 4.3%

    of edit calls failed; requests across the 27 tasks fell from 645 to 526, and pass@1 stayed at 100%

    27 eval tasks on gpt-5.4-mini.

  5. Parsed shell commands

    Rules judge what a command does, not how it's spelled9,26

    A compound command is parsed and each part is checked against your deny, ask and allow rules, so an allowed git status can't carry an rm along with it. In yolo, bash runs in sandbox-exec or bubblewrap, and each command gets its own empty /tmp and /run.

    < 1 ms

    sandbox cost per command on Linux, down from about 90 ms

    A Linux host with a busy /tmp.

  6. Sandbox escape matrix

    The sandbox has a test suite that tries to escape it46

    21 escape attempts run under three network policies through every way venom runs bash: write outside the workspace, read ~/.ssh, remount /, reach the Docker socket or the ssh agent, resolve DNS around the proxy. On Linux the whole matrix runs again as root.

  7. Spotlighting

    File contents and web pages can't give orders25

    Everything venom reads from outside, from files to command output to web pages, is fenced as untrusted, and an advisory detector flags text that tries to steer the agent.

    9/18 → 18/18

    prompt-injection trials resisted

    Eval safety slice, k=3.

  8. Content-defined chunking

    A re-run test suite costs only what changed47

    When a command's output mostly repeats one the model still has in view, such as a test run after a fix, venom sends a back-reference and a diff instead. FastCDC chunking finds the overlap, and the note is used only when it rebuilds the output exactly.

    82.5%

    fewer characters sent on near-duplicate output, with 0 false positives

    scripts/cdc-bench.ts on recorded outputs.

  9. Handoff by citation

    Subagents get pointers, not pastes35,33

    A parent hands a subagent the files and line ranges that matter, and the child reads them itself. Subagents run concurrently under a scheduler, each with its own context and tools.

    17.6k → 2.9k

    prompt tokens to hand over a 42 KB document

    A context handoff by file citation instead of pasting.

  10. Paired bootstrap eval gate

    Harness changes have to measure no worse48,49,22

    Harness changes run against a committed baseline of 42 tasks in 8 categories. A seeded, paired bootstrap decides, and a safety task that used to pass every trial may not fail one. Ideas that measured worse stay off: an effort router that cost 7.6% more a trial, and a compaction pass that cut pass@1.

    97.6% / 95.2%

    pass@1 and pass^3 on the baseline, 126 trials for $1.23

    Self-authored tasks, gpt-5.4-mini. Not a public benchmark.

The router

  1. Rendezvous hashing with bounded loads

    A session stays where its prompt cache is warm50

    Each session ranks the daemons by a 64-bit rendezvous hash. It goes to the first one under its load cap, the bound from Mirrokni, Thorup and Zadimoghaddam with a floor for idle slots. A daemon that leaves moves only its own sessions, and a full one spills along each session's own order.

    61-64% → 100%

    of requests on the session's warm daemon at 10% load; 64-68% → 81-86% at 50%, with no added queueing

    Simulated: 10 daemons × 4 slots, Zipf-popular sessions, 50k requests.

  2. Power of two choices

    No stampede onto the daemon that looked idle51

    Load reports can be 10 s old, and picking the least loaded daemon from a stale snapshot sends every request to the same one. The router samples two daemons and keeps the session's owner only if it costs no more than both.

    400 → 21

    requests on the busiest of 20 daemons in a 400-request burst (a fair share is 20)

    Simulated burst between two heartbeats.

  3. Cost in milliseconds of TTFT

    Moving a long session has to pay for its re-prefill52

    Every candidate gets a score in expected milliseconds to the first token: network, queue, error rate and breaker state. The owner's bonus is the cold prefill a move would cost, from each daemon's own running average of prefill time per token, so a 50k-token session doesn't leave a warm GPU to save a second of queueing.

  4. Radix tree of served prompts

    A new session goes to the backend that already holds its prefix20

    On a route with several self-hosted backends, the orchestrator keeps a bounded radix tree of the prompts each one served, in 512-byte blocks. A new session lands on the backend with the longest match, unless that backend is busier than the rest.

    75.5% → 79.7%

    prefix-cache hit rate, for +0.62 µs and one allocation per request

    Simulated on 3 fake vLLM targets with tight caches.

  5. Hedged requests

    Short requests don't wait on a slow daemon53

    A short non-streamed request, an embedding or a systemone call goes to a second daemon once the first passes its own p95 time to first token. The first answer wins, the other is cancelled, and you are billed once. Hedges stay under 5% of a key group's traffic.

  6. Retry budgets and backpressure

    Retries that can't turn an outage into a storm54,55

    Same-upstream retries draw on a budget of 20% of first attempts, and a retry the OpenAI SDK is already making isn't retried again underneath it. A connection reset counts as backpressure, not a dead upstream, so a cold start no longer trips the breaker.

    30 → 13

    upstream calls for 10 requests to a failing upstream; breaker trips on a cold start went from 4 of 14 to 0

    Orchestrator tests and load tests on a shared M2 Pro.

  7. Streaming without buffering

    The first token reaches you when the model writes it56,57

    Bytes go to the client as the upstream writes them, and every stream tells proxies such as nginx not to hold events back. Request bodies are rewritten by scanning and splicing the JSON in place, not by decoding and re-encoding it.

    5.05 s → 0.01 s

    to the first event behind nginx with gzip; a 525 KB body rewritten in 285 µs instead of 1.60 ms

    make smoke against the prod image, and make bench on an M2 Pro.

  8. Weighted priority classes

    Interactive requests get most of a busy backend's slots58

    When a backend is at its concurrency limit, waiting requests are served by class: interactive, then merge checks, then background work, by smooth weighted round robin at 8:3:1, first come first served within a class.

  9. Hybrid logical clocks

    A request's trace is in the right order even when clocks drift59

    Every step of a job, on the cloud and on each daemon, carries a hybrid logical clock stamp, so a failover or a resumed session reads in causal order on the request's timeline.

    6,202 → 0

    of 46,373 causal pairs out of order, wall clock against the stamps

    Simulated: 600 jobs, daemon clocks off by up to ±5 s.

  10. Space-Saving and HyperLogLog++

    Usage questions answered in milliseconds60

    Top spenders, top models and distinct sessions come from mergeable sketches kept in hourly and daily rollups, not from scanning the request log. Money totals stay exact.

    1,629 ms → 155 ms

    at 1M rows, distinct sessions within 0.28% and every top-10 entry found

    Postgres, 31 days of rows, scripts/bench-usage-sketches.ts.

  11. CUSUM over audits

    Public nodes that swap in a cheaper model get flagged61

    Venom replays a sample of a public node's requests on a reference and scores how far the answers drift. A cumulative sum over those scores alarms on small, steady drift that no single audit would fail, and suspicion raises the audit rate.

    13 audits

    median to flag Q6_K served in place of Q8_0; the honest node never alarmed in 3,000

    Qwen3-4B on llama.cpp, an RTX 5070 Ti, bootstrapped audit sequences.

More on the venom CLI · More on the router.

Trade-offs

When something else may suit you better

  • You want Anthropic's own agent, tuned for Claude models and supported by Anthropic, and you already have a Claude plan or Console account.5
  • You want a managed product with no endpoint to choose or run. venom needs a model behind an OpenAI-compatible URL, local or hosted.4
  • You are on native Windows and want an OS sandbox. venom's Windows build is in beta without one; Claude Code's sandbox doesn't support native Windows either, and WSL works for both.9,10

Try it on your own repository

The CLI and the daemon are free and MIT licensed. Venom Cloud is free for small teams.

Sources

  1. 1.The venom CLI, the orchestrator and the cloud are MIT licensed; the CLI needs no account. cli/package.json:7, docs/release.md:144.
  2. 2.Claude Code's license file: "All rights reserved. Use is subject to Anthropic's Commercial Terms of Service." github.com/anthropics/claude-code/blob/main/LICENSE.md. Checked 2026-10-18.
  3. 3.Qwen Code README: originally based on Google Gemini CLI v0.8.2, developed independently since v0.1; Apache-2.0; supports OpenAI, Anthropic, Gemini and Qwen APIs and local models (Ollama, vLLM). github.com/QwenLM/qwen-code. Checked 2026-10-18.
  4. 4.Venom README: the CLI turns off telemetry, update checks and downloads, and the orchestrator calls only its configured upstreams. README.md:32-33.
  5. 5.Claude Code setup docs: requires a Pro, Max, Team, Enterprise or Console account, or a third-party provider (Amazon Bedrock, Google Cloud, Microsoft Foundry); "internet connection required"; native installs update in the background, package-manager installs don't by default. code.claude.com/docs/en/setup. Checked 2026-10-18.
  6. 6.Claude Code LLM gateway docs: Anthropic "doesn't support routing Claude Code to non-Claude models through any gateway." code.claude.com/docs/en/llm-gateway. Checked 2026-10-18.
  7. 7.Claude Code data-usage docs: metrics are on by default with the Claude API (they never include code, prompts or file paths), off by default on Bedrock, Google Cloud and Foundry; DISABLE_TELEMETRY or CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC turns them off. code.claude.com/docs/en/data-usage. Checked 2026-10-18.
  8. 8.Qwen Code settings docs: privacy.usageStatisticsEnabled defaults to true; general.enableAutoUpdate (update checks and installs on startup) defaults to true; approval modes plan, default, auto-edit, auto and yolo. qwenlm.github.io/qwen-code-docs/en/users/configuration/settings. Checked 2026-10-18.
  9. 9.venom's permission modes, compound-command parsing and the OS sandbox (sandbox-exec on macOS, bubblewrap on Linux) with a per-host network allowlist. docs/getting-started.md, CHANGELOG.md "per-host network allowlist".
  10. 10.Claude Code permission modes (default, acceptEdits, plan, auto, dontAsk, bypassPermissions) and sandboxing (Seatbelt on macOS, bubblewrap on Linux and WSL2, off by default). code.claude.com/docs/en/permission-modes, /sandboxing. Checked 2026-10-18.
  11. 11.Qwen Code docs: subagents, an opt-in sandbox (macOS Seatbelt, Docker or Podman, Linux bwrap), MCP, and extensions that also install Gemini CLI and Claude Code marketplace plugins. qwenlm.github.io/qwen-code-docs (sub-agents, sandbox, mcp, extension). Checked 2026-10-18.
  12. 12.venom's subagents: explorer, worker, verifier and your own, each with its own context and tools; workers can run in git worktrees. docs/workspaces.md, packages/harness/AGENTS.md.
  13. 13.Claude Code subagents, plugins, hooks, skills, MCP, and automatic prompt caching. code.claude.com/docs/en/sub-agents, /plugins, /mcp, /costs. Checked 2026-10-18.
  14. 14.venom loads pi packages and Claude Code plugins, and reads CLAUDE.md, .claude skills, commands, agents, settings, hooks and .mcp.json. docs/plugins.md, cli/AGENTS.md.
  15. 15.The fixed system prompt and tool list are held under a token budget a test enforces (2,906 tokens measured, budget 2,910), and the prefix is tested append-only across turns and identical across sessions. packages/harness/test/prompt-budget.test.ts "by team and org memory", CHANGELOG.md "pins the prompt prefix as append-only".
  16. 16.The orchestrator: model aliases over vLLM, SGLang, llama.cpp, Ollama and cloud providers, fallback before the first byte, circuit breakers, session-sticky routing for KV-cache reuse, per-key limits and budgets, usage and keys in SQLite. README.md:87.
  17. 17.Ollama docs: "supports a subset of the OpenAI API"; binds 127.0.0.1:11434 by default; the local API "does not require authentication"; hosted cloud models are also offered. docs.ollama.com (openai-compatibility, faq, api/authentication). Checked 2026-10-18.
  18. 18.vLLM docs: an OpenAI-compatible server (Completions, Chat and more) and Automatic Prefix Caching, which reuses the KV cache of queries that share a prefix. docs.vllm.ai (openai_compatible_server, automatic_prefix_caching). Checked 2026-10-18.
  19. 19.The vLLM project's separate routers: production-stack (session-ID routing) and vllm-project/router (cache-aware load balancing, circuit breakers, retries). github.com/vllm-project/production-stack, github.com/vllm-project/router. Checked 2026-10-18.
  20. 20.Longest-prefix routing for new sessions, simulated on 3 fake vLLM targets with tight caches and 8 prompt families. A simulation, not a production measurement. CHANGELOG.md "unless that target is busy against the others (SGLang's balance".
  21. 21.Venom Cloud's routing: direct (a 307 redirect with a single-use ticket) or relayed through the cloud; daemons only dial out. docs/cloud-mesh.md:36-44.
  22. 22.Venom eval baseline, 2026-10-09: 42 self-authored tasks in 8 categories, k=3 (126 trials), venom-default routed to gpt-5.4-mini through a local orchestrator. Not a public benchmark and not a head-to-head with other agents. evals/results/2026-10-09T08-33-43Z-23be.json, evals/AGENTS.md "`apply_patch` and no `edit`. Results: pass@1".
  23. 23.Bundled CLI vs. unbundled, Apple M2 Pro, Node 24.9, n=15 interleaved runs, medians. cli/AGENTS.md "**Measured, the bundle**".
  24. 24.Eval longsession slice at a 16k-token context window, k=2, before and after the compaction work. CHANGELOG.md "At a 16k window, longsession pass@1 went from".
  25. 25.Eval safety slice (prompt injection), k=3, before and after spotlighting and the injection guidance. CHANGELOG.md "safety slice went from".
  26. 26.Linux sandbox: each command gets its own empty /tmp and /run; per-command cost on a host with a busy /tmp. CHANGELOG.md "per-command cost from about".
  27. 27.Code index on the Kubernetes repository (26,918 files), worker and overlay updates, on a machine loaded by other runs. packages/harness/AGENTS.md "| | kubernetes, first | kubernetes, now |".
  28. 28.make bench, Apple M2 Pro, a local fake upstream, a ~5 KB chat request with a tool; the added time includes the extra loopback hop. Re-runs on a loaded machine measured +59 to +63 µs. services/go-orchestrator/AGENTS.md "chat body with a tool; non-stream: one JSON response;".
  29. 29.Load test against a fake vLLM upstream (300 ms TTFT, 100 tokens/s) on a shared M2 Pro; numbers are at 1,000 concurrent streams. services/go-orchestrator/AGENTS.md "tokens/s per stream,", services/go-orchestrator/AGENTS.md "**Streaming levels**".
  30. 30.OpenAI prompt_cache_key pooled across sessions, measured live: eight new sessions with the same 5k-token prefix. CHANGELOG.md "live, eight new sessions with the same".
  31. 31.llama.cpp b11459 with the host-RAM prompt cache, interleaved sessions, measured through the orchestrator. CHANGELOG.md "pinning each session to a slot with `id_slot`".
  32. 32.Venom Cloud's route() wall-clock time on Node 24 for 10 eligible daemons (65 µs p50 for 100). docs/cloud-mesh.md "Only the `route()` timings are wall-clock measurements", docs/cloud-mesh.md "`route()` itself takes".
  33. 33.Subagent scheduler benchmark with scripted agents (no model): 16 agents at concurrency cap 16. CHANGELOG.md "Runtime with the R2 scheduler".
  34. 34.apply_patch for GPT-5 models, 27 eval tasks: pass@1 held at 100%. CHANGELOG.md "`apply_patch` (OpenAI's V4A format) for GPT-5 family models".
  35. 35.Context handoff to a subagent by file citation instead of pasting a 42 KB doc. CHANGELOG.md "doc handed over this way cost a child".
  36. 36.A compaction summary re-sends the session's prefix byte for byte, so it is a cache read (a faux-provider measurement). CHANGELOG.md "so its prefix is a cache read (faux measurement".
  37. 37.Images resized to the size each provider uses (scripts/image-bench.ts); a 2880×1800 screenshot. CHANGELOG.md "images go to the model at the size its provider actually uses".
  38. 38.Tool-output noise filters on 14 recorded real outputs (scripts/noise-bench.ts); failures and errors stay verbatim. CHANGELOG.md "Harness: tool-output noise filters".
  39. 39.Merkle change detection for /undo checkpoints on a 30k-file workspace (scripts/checkpoint-bench.ts). CHANGELOG.md "nested repositories and other doubtful cases fall back to a full capture".
  40. 40.Trigram index for grep on a 29k-file synthetic repository, nine patterns (scripts/trigram-bench.ts). CHANGELOG.md "hands `rg` the files that cannot match".
  41. 41.Single-pass Aho-Corasick scanning for the injection detector and the secret scrubs, against the previous code as an oracle (node scripts/scan-bench.ts). CHANGELOG.md "`detectInjection`".
  42. 42.The fixed prompt (system prompt and tool schemas, chars/4) before and after the prompt diet, measured on main. packages/harness/AGENTS.md "**The prompt diet**".
  43. 43.Lazy MCP tools (test/lazy-tools.test.ts): a fake MCP server of 60 GitHub-sized tools, the fixed prompt with the server declared and with it deferred. packages/harness/AGENTS.md "Measured (`test/lazy-tools.test.ts`".
  44. 44.apply_patch matches a hunk exactly, then ignoring trailing whitespace, then ignoring indentation, then folding Unicode quotes, dashes and spaces; a patch applies whole or not at all. packages/harness/src/patch/.
  45. 45.edit against apply_patch on gpt-5.4-mini, 27 eval tasks: failed edit calls per arm. evals/results/compare-edit-vs-apply-patch.md "Per trial, over the", evals/results/compare-edit-vs-apply-patch.md "| baseline, edit, k=3 |".
  46. 46.The sandbox escape matrix (test/sandbox-matrix.test.ts): 21 probes under three network policies, through every bash entry point, live on macOS and on Linux, the Linux run once as a user and once as root. packages/harness/AGENTS.md:8837-8876.
  47. 47.Near-duplicate command output found by content-defined chunking (FastCDC) and a bounded line diff (scripts/cdc-bench.ts): characters sent on the near-duplicate cases, and false positives. CHANGELOG.md "`scripts/cdc-bench.ts`:".
  48. 48.The eval gate: a paired, seeded bootstrap of per-task pass rates; it fails when the 95% interval lies below the tolerance, or when any safety task that passed every trial before fails one. evals/AGENTS.md:18-20, evals/src/score.ts, evals/src/compare.ts.
  49. 49.Changes measured and not shipped: an effort router (+7.6% cost a trial, one hard trial lost) and a compaction relevance pass (pass@1 97.2% → 91.7%, off by default). evals/AGENTS.md "### Effort router (tried, not shipped)", docs/adr/0016-compaction-relevance-pass.md "Whole run (longsession, memory,".
  50. 50.Bounded-load session owner, simulated (npm run sim:bounded): 10 daemons × 4 slots, Poisson arrivals, 1,000 Zipf-popular sessions (s = 0.8 and 1.1), 50k requests. A simulation, not a production measurement. docs/cloud-mesh.md "(`bun run sim:bounded`, `sim/bounded-load.ts`). The setup".
  51. 51.Power-of-two-choices pick, simulated: a burst of 400 sessionless requests between two heartbeats on 20 daemons (asserted in sim/chaos.test.ts). docs/cloud-mesh.md "sessionless requests between two heartbeats put all".
  52. 52.Venom Cloud's score: expected TTFT in ms (network, queue, error penalties) less an affinity bonus worth the prefill a move would cost, up to 20 s, from each daemon's own EWMA of prefill time per uncached token. docs/cloud-mesh.md:795-831.
  53. 53.Venom Cloud's hedging: short non-streamed requests, embeddings and systemone only; sent to the runner-up after the chosen daemon's p95 TTFT; under 5% of a key group's requests a minute; the loser is cancelled and the request billed once. Tested in test/db/hedge.test.ts. docs/cloud-mesh.md:856-895.
  54. 54.Retry budget test: 10 requests to an upstream that always answers with a 503 error, max_attempts 3 (server.TestRetryBudgetCapsRetries). services/go-orchestrator/AGENTS.md "`server.TestRetryBudgetCapsRetries` (".
  55. 55.Load test: a fresh orchestrator hit by 1,000 streams at once, 14 interleaved trials, before and after connection resets were treated as backpressure. services/go-orchestrator/AGENTS.md "**Failure behaviour under load**".
  56. 56.make smoke STAGES=prod: nginx 1.30.5 with gzip in front of the prod image, a ~5 s stream; time to the first event with and without the orchestrator's X-Accel-Buffering header. services/go-orchestrator/AGENTS.md "`make smoke STAGES=prod` checks it end to end".
  57. 57.Request rewrite on a ~525 KB body (a long agent session): a JSON scan and splice against a decode and re-encode, make bench, Apple M2 Pro. services/go-orchestrator/AGENTS.md "Request rewriting on a".
  58. 58.The orchestrator's priority classes: interactive, blocking-merge and background requests share an upstream's slots by smooth weighted round robin, 8:3:1. services/go-orchestrator/AGENTS.md:1668-1671.
  59. 59.Hybrid logical clocks, simulated (sim/hlc-skew.ts): 600 jobs, 30% failovers, 20% session resumes, three function instances, daemon clocks off by up to ±5 s. docs/cloud-mesh.md "restore before the old daemon's last event".
  60. 60.Usage sketches (Space-Saving, HyperLogLog++) against exact GROUP BYs, 1M request rows over 31 days in Postgres (scripts/bench-usage-sketches.ts). services/nodejs-cloud/AGENTS.md "- **Measured** (`node scripts/bench-usage-sketches.ts`".
  61. 61.Public-mesh audits of Qwen3-4B on llama.cpp, Q8_0 registered, on an RTX 5070 Ti; CUSUM with k = 4, h = 10,000, median over 400 bootstrapped sequences of the measured audits. docs/public-mesh-hardening.md "### Real quantisations (llama.cpp, Qwen3-4B)".