What’s measured
A 4-node synchronous webhook flow that holds the HTTP connection open until the flow returns:Webhook trigger
Catches the request on a
/sync URL and holds the connection until the flow finishes.AP_REUSE_SANDBOX=true) — the engine process is reused between jobs.
Results
686 req/s
Peak warm throughput — 16 apps, 160 workers.
~4.5 req/s
Per worker, held flat from 40 to 160 workers — throughput scales linearly with the fleet.
| Apps · Workers | Ratio | Warm req/s | Warm req/s per worker |
|---|---|---|---|
| 4 app · 40 workers | 1:10 | 185.3 | 4.6 |
| 8 app · 80 workers | 1:10 | 409.5 | 5.1 |
| 12 app · 120 workers | 1:10 | 553.0 | 4.6 |
| 16 app · 160 workers | 1:10 | 686.3 | 4.3 |
What each tier ran — and what it was actually doing
Only the app and worker counts scale (1:10). Postgres and Redis are a single fixed-size pod each — the same for every row below. CPU is the average across three warm load tests; the singletons’ figures are the whole pod, app/worker are per pod.| Apps · Workers | Warm req/s | Postgres used / cap | Redis used / cap | App used / cap (per pod) | Worker used / cap (per pod) |
|---|---|---|---|---|---|
| 4 · 40 | 185 | 522m / 3000m | 134m / 2000m | 782m / 1000m | 102m / 500m |
| 8 · 80 | 410 | 640m / 3000m | 150m / 2000m | 518m / 1000m | 72m / 500m |
| 12 · 120 | 553 | 546m / 3000m | 132m / 2000m | 311m / 1000m | 50m / 500m |
| 16 · 160 | 686 | 396m / 3000m | 169m / 2000m | 205m / 1000m | 37m / 500m |
max_connections=100 would cap the fleet at ~10 apps, which is the artifact behind the earlier “120 cliff”.)
How throughput scales
- Warm scales linearly with the fleet. Per-worker throughput stays flat at ~4.5 req/s from 40 to 160 workers, so total throughput tracks the worker count (185 → 410 → 553 → 686; 3.7× for 4× the fleet). The shared Postgres and Redis singletons are not the wall — they sit near-idle at every fleet size (Postgres under 0.6 of a core, Redis under 0.2, both far below their caps), and raising their resources several-fold does not move the curve. The ceiling is the concurrency-1 worker model: each worker is busy for the whole per-flow time — engine run plus the end-of-run run-log persistence it finishes before taking the next job — so fleet throughput is
workers ÷ per-flow-time, which is linear in the fleet. (The synchronous response reaches the client sooner than that — it is sent at the response step, before the worker wraps up the log write — so client-perceived latency is lower than the worker-busy time that sets throughput.) Per-flow time carries run-to-run variance (the object-store log-write tail), which is why a single run’s curve looks bumpy; the invariant that the per-worker rate holds constant is what shows the scaling is linear.
Why Production Setup recommends 1:10. Apps at 1 vCPU are cheap relative to the worker fleet, and 1:10 is the warm-headroom margin that keeps the app tier from becoming the wall during bursts. See Production Setup.
Latency anatomy
Where the worker’s milliseconds go — warm at peak (16 app · 160 w):| Layer | Warm |
|---|---|
| Provision (flow bundle + piece + engine, mostly disk-cache hits) | ~10 ms |
| Sandbox boot (engine process reused) | ~5 ms |
| Flow run (4 steps: engine→app callbacks + end-of-run log persist) | ~203 ms |
| Worker-busy avg per job | ~218 ms |
workers ÷ worker-busy-time). The synchronous client sees less: the response is published at the flow’s response step, before the worker finishes persisting the run log, so client-perceived latency runs below the worker-busy figure.
Test environment
- Cluster: GKE
n2-standard-16× 10 nodes,europe-west1-b - Worker: 0.5 vCPU / 1 GB, concurrency 1,
SANDBOX_CODE_ONLY(Node fork +isolated-vm) - App: 1 vCPU / 1 GB
- Object store: same-region GCS bucket (
europe-west1) over the S3-interop endpoint, path-style SigV4 presigned URLs (AP_S3_USE_SIGNED_URLS=true) - Piece bundles: official tarballs served from the Activepieces CDN (
AP_USE_CDN_FOR_BUNDLES=true) - Postgres + Redis: in-cluster singletons, deliberately over-provisioned so they stay off the critical path — Postgres at 3 vCPU / 3 GB with
max_connections=2000(the default 100 would starve the app pools past ~10 apps), durability off, and its data dir on tmpfs; Redis at 2 vCPU / 2 GB withio-threads. Under load both stay near-idle (Postgres<0.6core, Redis<0.2), confirming the worker tier, not the singletons, is the ceiling. - Load:
hey, concurrency matched to worker count (40/80/120/160) so requests don’t queue behind the concurrency-1 workers — latency reflects real service time, not backlog
How to reproduce
benchmark/k8s-sandbox.yaml to the cluster, runs the load test against the app LoadBalancer, and reports warm throughput and the per-run breakdown from worker-pod logs. Set APP_REPLICAS and WORKER_REPLICAS (keeping the 1:10 ratio) to reproduce any row in the results table.
Benchmark your own installation
Load-test and diagnose your own deployment — no cluster scripts required — with the CLI. It publishes a synchronous flow (webhook trigger → data mapper → return response), fires load at its sync webhook endpoint with autocannon, and returns one self-contained diagnostic bundle you can hand to support.AP_WORKER_CONCURRENCY across connected workers) so requests don’t queue and you read real service time, not backlog. Comparing two deployments? Match concurrency to each one’s own slots — never a fixed number, which makes the smaller one queue.
| Option | Default | Description |
|---|---|---|
--url | http://localhost:3000 | Base URL of your instance |
--api-key | AP_API_KEY env var | Platform API key (Bearer) — required |
--project-id | Project to create the flow in — required | |
--concurrency | auto = execution slots | Concurrent connections |
--requests | 40 × concurrency | Total requests to fire |
--body | {"test":true} | JSON body sent to the webhook |
--json | Emit the full machine-readable bundle (share with support) |
Reference numbers
A real run against the recommended GKE deployment from the results table: 4 workers @ 0.5 vCPU / 1 GB, concurrency 1,SANDBOX_CODE_ONLY, AP_REUSE_SANDBOX=true, same-region GCS with signed URLs, warm, load = concurrency 4 (= slots) × 200 requests.
Run the CLI against your own deployment and compare tier by tier. A number several times larger localizes the problem: RUN ≫ 200 ms means a heavier flow or a CPU-starved worker; storage ≫ 240 ms means a mis-regioned or throttled object store; a large QUEUE with climbing queue depth means you drove more concurrency than you have slots.
| Section | What it measures | Where it’s measured |
|---|---|---|
| Version & health | App release, whether every connected worker matches it (a version-skewed worker is silently withheld jobs), and app/worker/DB CPU-RAM-disk health | GET /v1/health/system |
| Infra round-trip | Authoritative in-region DB / Redis / S3 write+read latency, the effective execution/storage config, the app tier (every app replica’s CPU / RAM / disk / event-loop — apps self-register into a diagnostics cache on their metrics tick), and the worker fleet with live per-worker CPU. The S3 round-trip is the same cost folded into every run’s end-of-run log backup — a slow object store shows up here and in RUN. Self-hosted only. | GET /v1/health/diagnostics |
| Config flags | The limits and throttles that silently cap throughput — a PROJECT_RATE_LIMITER_ENABLED or a low DEFAULT_CONCURRENT_JOBS_LIMIT re-queues jobs (latency, not errors); the memory/timeout/log ceilings that turn into MEMORY_LIMIT_EXCEEDED / TIMEOUT / LOG_SIZE_EXCEEDED statuses | GET /v1/flags |
| Setup | Execution slots (Σ AP_WORKER_CONCURRENCY) and a PASS/WARN check of every worker’s specs, sandbox mode, reuse, and concurrency against the recommended shape | GET /v1/worker-machines |
| Network | CLI→server round-trip, so cross-region distance is quantified and subtracted rather than blamed on the server | timed GET /v1/flags |
| Load + latency split | Throughput, run-status outcomes (server truth, not just HTTP codes), live queue depth during load, and the per-run latency split into QUEUE / PROVISION / BOOT / RUN — measured inside the worker, so it’s the same on any deployment regardless of where the CLI runs. The verdict says whether latency is queue-bound (too much concurrency for the slot count) or service-bound (real engine time) | FlowRun.timeline + GET /v1/worker-machines/queue-metrics |
| Storage | Fraction of runs whose logs were persisted — proves the worker→storage write path works end to end | FlowRun.logsFileId |
- Exits non-zero if any request fails — usable as a CI gate.
- The benchmark flow stays in your project — delete it when done.
If your hardware or sandbox mode differs from the recommended shape, the absolute numbers shift — but the shape holds: match concurrency to your own slot count so nothing queues, then read whether you are queue-bound or service-bound and which tier dominates.