What’s measured
A 4-node synchronous webhook flow that holds the HTTP connection open until the flow returns:1
Webhook trigger
Catches the request on a
/sync URL and holds the connection until the flow finishes.2
Math Helper
Adds
2 + 3.3
Code step
Runs
return inputs.sum + 1 inside an isolated-vm context.4
Webhook response
Returns the result, closing the held connection.
AP_REUSE_SANDBOX=true) — the engine process is reused between jobs.
Results
777 req/s
Peak warm throughput — 16 apps, 160 workers.
6.1 req/s
Best per-worker rate, at 80 workers. It falls to 4.9 by 160 — adding workers keeps adding throughput, but less of it each time.
What each tier ran — and what it was actually doing
Only the app and worker counts scale (1:10). Postgres and Redis are a single fixed-size pod each — the same for every row below. CPU is sampled every 3 s throughout the measured pass; the singletons’ figures are the whole pod, app/worker are per pod. Read the two right-hand columns differently from the two on the left. Only the worker has a CPU limit (requests == limits == 500m, a hard cap). Postgres, Redis and the app declare CPU requests — a scheduling reservation they may burst above on a node with spare cores — so their numbers show what each tier consumed, not how close it came to a ceiling.
Postgres is by far the fastest-growing tier. It climbs 529m → 2738m as the fleet quadruples — 5.2× the CPU for 4× the workers — and ends up the single largest CPU consumer in the deployment, above the entire 160-worker fleet combined (~14.6 cores across workers vs 2.7 in one database process, but spread over 160 pods versus one). Its cost per unit of work is near-constant (~2.5 millicores per req/s at every tier), so it grows with throughput, not with fleet size, and no configuration makes it stop growing. Redis behaves the same way (123m → 781m).
By contrast the workers — the only hard-capped tier — sit at ≤0.1 of their 0.5-core cap at every fleet size, and apps hold steady at ~0.52–0.61 of a core per pod because the 1:10 ratio adds app capacity in step with the load.
The singletons here are deliberately generous (3 vCPU request,
max_connections=2000, durability off, data dir on tmpfs). A managed Postgres at the 2 vCPU / 4 GB this documentation recommends, with fsync on and real disks, does less work per commit than this one and will feel the same load sooner. Size the database against your peak throughput, not against your worker count.How throughput scales
Adding workers always adds throughput, but the return per worker falls after ~80. Total throughput goes 213 → 484 → 641 → 777 (3.6× for 4× the fleet). Per worker that is 5.3 → 6.1 → 5.3 → 4.9: the rate peaks at 80 workers and then gives back about 20%. Two things are happening at once:- Below ~80 workers the fleet is still filling out. 40 workers cannot push enough concurrent work to use the tiers behind them, so the per-worker rate is not yet at its best. This is why the 40→80 step is super-linear (2.27× throughput for 2× the fleet).
- Past ~80 workers, per-flow time stretches. The measured
sandbox runphase goes 146 ms at 80 workers to 181 ms at 160, so each worker completes fewer flows per second even though it is nowhere near its CPU cap. Something shared is absorbing the extra concurrency, and database CPU is the tier visibly growing with it (see the caveat above — growth is measured, causation is not).
workers ÷ per-flow-time. That is linear in the fleet only while per-flow time holds constant, and past ~80 workers it does not. (The synchronous response reaches the client sooner than that — it is sent at the response step, before the worker wraps up the log write — so client-perceived latency is lower than the worker-busy time that sets throughput.)
What this means for sizing: worker count is the right lever up to roughly this scale, but do not read the worker fleet as the only dial — the shared tiers behind it grow with throughput and need to grow with you.
Why Production Setup recommends 1:10. Apps at 1 vCPU are cheap relative to the worker fleet, and 1:10 is the warm-headroom margin that keeps the app tier from becoming the wall during bursts — it holds here, with apps steady at ~0.6 of a core per pod at every tier. See Production Setup.
Latency anatomy
Where the worker’s milliseconds go — warm at peak (16 app · 160 w), averaged over all 64,000 measured runs:
Warm provisioning and boot are effectively free — the piece cache is on local disk and the engine process is reused, so essentially the entire worker-busy time is the flow run itself. That figure grows with the fleet (167 ms at 40 workers, 146 ms at 80, 167 ms at 120, 181 ms at 160), which is the per-flow stretch behind the falling per-worker rate above.
This is the time the worker is occupied per job — and at concurrency 1 it is what sets throughput (
workers ÷ worker-busy-time). The synchronous client sees less: the response is published at the flow’s response step, before the worker finishes persisting the run log, so client-perceived latency runs below the worker-busy figure. At peak the in-cluster client measured p50 180 ms, p95 288 ms, p99 652 ms.
Test environment
- Cluster: GKE
n2-standard-16× 10 nodes,europe-west1-b - Worker: 0.5 vCPU / 1 GB, concurrency 1,
SANDBOX_CODE_ONLY(Node fork +isolated-vm). Idle RSS ~145 Mi - App: 1 vCPU / 1 GB
- Object store: same-region GCS bucket (
europe-west1) over the S3-interop endpoint, path-style SigV4 presigned URLs (AP_S3_USE_SIGNED_URLS=true) - Piece bundles: official tarballs served from the Activepieces CDN (
AP_USE_CDN_FOR_BUNDLES=true) - Postgres + Redis: in-cluster singletons, deliberately generous — Postgres requesting 3 vCPU / 3 GB with
max_connections=2000(the default 100 would starve the app pools past ~10 apps), durability off, and its data dir on tmpfs; Redis requesting 2 vCPU / 2 GB withio-threads. Neither has a CPU limit, so both may burst above their request; Postgres consumed 2738m at the top tier - Load:
heyrun inside the cluster, against the app Service, concurrency matched to worker count (40/80/120/160) so requests don’t queue behind the concurrency-1 workers — latency reflects real service time, not backlog. 400 requests per worker per tier (16k/32k/48k/64k), preceded by an unmeasured warmup pass
How to reproduce
benchmark/k8s-sandbox.yaml to the cluster, runs the load test from a pod inside the cluster, and reports warm throughput and the per-run breakdown from worker-pod logs. Set APP_REPLICAS and WORKER_REPLICAS (keeping the 1:10 ratio) to reproduce any row in the results table.
Benchmark your own installation
Load-test and diagnose your own deployment — no cluster scripts required — with the CLI. It publishes a synchronous flow (webhook trigger → data mapper → return response), fires load at its sync webhook endpoint with autocannon, and returns one self-contained diagnostic bundle you can hand to support.AP_WORKER_CONCURRENCY across connected workers) so requests don’t queue and you read real service time, not backlog. Comparing two deployments? Match concurrency to each one’s own slots — never a fixed number, which makes the smaller one queue.
Reference numbers
A real run against a small deployment of the recommended shape — 4 workers @ 0.5 vCPU / 1 GB, concurrency 1,SANDBOX_CODE_ONLY, AP_REUSE_SANDBOX=true, same-region GCS with signed URLs, warm, load = concurrency 4 (= slots) × 200 requests. This is a separate, deliberately tiny deployment used to show what the CLI’s output looks like; it is not a row in the results table above, and its cross-region client latency is not comparable to those numbers.
Run the CLI against your own deployment and compare tier by tier. A number several times larger localizes the problem: RUN ≫ 200 ms means a heavier flow or a CPU-starved worker; storage ≫ 240 ms means a mis-regioned or throttled object store; a large QUEUE with climbing queue depth means you drove more concurrency than you have slots.
Here the read is unambiguous: workers pegged at ~100% of their 0.5-core limit and RUN ≈ 200 ms dominate, while QUEUE (~100 ms at concurrency = slots) and the infra round-trips are small — the deployment is service-bound on worker CPU, so the lever is more/bigger workers, not a code change.
- Exits non-zero if any request fails — usable as a CI gate.
- The throwaway project (and its benchmark flow) is deleted automatically when the run finishes.
If your hardware or sandbox mode differs from the recommended shape, the absolute numbers shift — but the shape holds: match concurrency to your own slot count so nothing queues, then read whether you are queue-bound or service-bound and which tier dominates.