Pond

Benchmarking the local stack — ./cast bench

./cast bench answers, with numbers, what the local stack costs: how long each piece takes to stand up, what it’s hogging at steady state (memory, CPU, process count), what a run costs end-to-end (dispatch latency through the real pool), and where the disk went. Every invocation renders tables and persists a JSON report under .pond-local/bench/ so you can compare over time.

./cast bench                    # non-disruptive: resources + dispatch latency + disk
./cast bench --standup          # ALSO stop + re-time the `cast pool` bring-up (warm)
./cast bench --cold --yes       # bring-up from scratch (drops DB volume + sandbox image!)
./cast bench --runs 5 --docker-runs 2   # more dispatch-path samples

What it measures

Stand-up phases (--standup / --cold). The bring-up sequence in cast.py stamps a timestamp at every →/✓ transition; bench tears the stack down, re-runs cast pool, and attributes wall-clock to each phase — postgres ready, uv sync, migrations, control-plane boot (uvicorn import + health), MinIO, sandbox-image check/build, orchestrator pairing + registration, worker enrollment + registration. Warm keeps the DB volume and the sandbox image (the everyday ./cast pool loop); cold drops both first — destructive: a cold run wipes registered credentials/harnesses/models, so it asks first (--yes to skip).

Steady-state resources. Three samples, one second apart:

  • host components (control plane, orchestrator, worker) — RSS/CPU/process count summed over each one’s whole process subtree, so agent children count toward the worker;
  • a separate agents (under worker) row — the non-Python children of the worker (e.g. a live codex-app-server) and their subtrees, so long-lived agents are visible on their own;
  • containers (postgres, MinIO) via docker stats.

Dispatch-path latency. Bench registers a tiny echo harness (cast-bench-echo — no model, no credential; safe to delete) and submits real runs through POST /v1/runs, so the sample covers the same path a real run takes: submit → implicit fetch → dispatch row → orchestrator poll → worker claim → sandbox launch → execution → result ingest → run done. Reported per sample:

columnmeaningsource
total (s)submit → run terminal, as the client sees itclient clock
dispatch→claimdispatch row created → worker claimedserver (dispatch_jobs)
claim→doneclaim → job finishedserver (dispatch_jobs)
overhead*claim→done minus the probe’s fixed 2s sleep — sandbox spin-up + teardown + reportingderived

--runs N samples profile: none (bare subprocess — the floor: pure dispatch machinery); --docker-runs N samples untrusted-code-read (container spin-up on the sandbox image — the confined increment over that floor). Bench skips profile-none samples when POND_REQUIRE_SANDBOX=true (they’d be rejected) and confined samples when no sandbox.*-capable worker is registered. During a confined sample it also grabs docker stats from the pond-job-* container, so the report shows the per-job sandbox footprint.

Disk. .pond-local/ and .pond-state/ sizes, every .pond-local/*.log individually (a chatty control plane log grows multi-GB — this is where you catch it), image sizes (sandbox image, postgres, MinIO), pg_database_size, and the bundle-bucket bytes in MinIO.

Reading the numbers

  • Warm stand-up is the number that matters day-to-day: it’s what ./cast pool costs after the first run. The first-ever run is dominated by the sandbox-image build; that shows up in --cold (with Docker’s layer cache still in play — prune the cache for a true from-nothing build).
  • The overhead* column is the honest per-job tax of the machinery. For profile: none expect it near zero; the confined delta over that floor is the price of the container boundary.
  • agents (under worker) only appears while an agent is live (e.g. an attachable codex-app-server between attaches). The echo probe is too short-lived to register — benchmark a real harness by watching this row during a real run.
  • Reports are plain JSON (kind: pond-stackbench, version: 1) — diff two files from .pond-local/bench/ to see a regression.

Caveats

  • --standup/--cold restart the pool: in-flight runs on the local worker get reclaimed/failed, and cast pool rewrites POND_REQUIRE_SANDBOX=true (its standard posture). If you had flipped it off for profile: none work, re-flip and ./cast up afterwards.
  • CPU% for host processes is ps’s recent-usage figure — treat it as a steady-state indicator, not an integral.
  • Client-clock and server-clock segments are reported separately (never mixed), so total ≠ sum of the server-side segments; the difference is fetch, queue time before the dispatch row, and result ingest.