Infrastructure Upgrade — Mar 16–17, 2026

​

Summary

​

Impact

• Most cloud customers experienced failed or delayed flow executions over a ~12–24 hour period, with the first ~6 hours being the most severe. All affected executions were replayed from the failed step once service was restored.
• One enterprise customer with a dedicated worker had flows fail because npm was blocked after a trust level misconfiguration during pre-incident migration. All affected executions were replayed from the failed step once resolved.

​

Timeline

1. Mar 16, 8:13 PM — Shared workers begin failing after the architecture refactor deployment is applied. Team immediately begins investigating.
2. Mar 16, 8:50 PM — Brief recovery observed.
3. Mar 16, 8:52 PM — Errors resurface. Pattern of brief recoveries followed by recurring errors points to a resource issue rather than a code bug.
4. Mar 17, 3:26 AM — Root cause identified: persistent volumes (PVCs) filled up with no shell access to worker pods to fix in-place. Decision made to revert to the previous deployment method (Kamal) with multi-tenant workers. This was not a simple rollback — reverting to Kamal required significant code changes to re-adapt the worker architecture back to the previous multi-tenant deployment model.
5. Mar 17, ~8:00 AM — Kamal revert deployed, flow execution restored for cloud customers.
6. Mar 17, 8:13 AM — Attention turned to enterprise dedicated workers. Discovered one customer’s dedicated worker had trustedEnvironment incorrectly set to false after the namespace migration, blocking npm libraries in their sandbox.
7. Mar 17, 11:30 AM — Trust level configuration fixed, test coverage added for this code path. All failed executions identified across cloud and enterprise customers and replayed from the exact failed step. No customer data or automation results were permanently lost.

​

Root Causes

​

Issue 1: Persistent volume provisioning on Kubernetes

​

Issue 2: Enterprise dedicated worker trust level misconfiguration

​

What Went Well

1. Enterprise worker isolation was completed ahead of rollout. Dedicated workers were moved to their own namespaces 1–2 days before the shared worker migration, which limited the blast radius and prevented the PVC issue from affecting enterprise customers directly.
2. Execution replay from failed step. The platform’s ability to replay failed executions from the exact step that failed meant no customer data was permanently lost, despite the extended outage.
3. Smoke tests for trusted environments already existed in worker v2. The new worker architecture already included smoke tests that validate sandbox npm access for trusted environments, which helped catch Issue 2 quickly once it was discovered.
4. Gradual dedicated worker migration was already built into worker v2. Per-customer validation when migrating dedicated workers was already implemented, reducing future risk.
5. Quick identification of the revert path. Once the PVC root cause was identified, the team made a clear decision to revert to Kamal rather than continue debugging the Kubernetes deployment, which accelerated recovery.
6. Team stayed engaged through the night (8 PM – 11:30 AM UTC) with continuous investigation and response.

​

What Went Wrong

1. No rollback plan for the Kubernetes deployment. The migration to Kubernetes was deployed without a documented or tested rollback strategy. When PVCs filled up, there was no fast path back, delaying recovery by several hours.
2. PVC sizing was not validated under production load. Persistent volumes were provisioned without load-testing against real production traffic patterns, causing them to fill up unexpectedly fast.
3. Trust level configuration had no test coverage. The code path that set trustedEnvironment for dedicated workers was not covered by tests, allowing a misconfiguration to ship undetected.
4. No canary deployment strategy. The new architecture was rolled out to all shared workers at once rather than incrementally, so there was no opportunity to catch issues on a small subset before full impact.
5. Issue 2 was discovered late. The enterprise worker misconfiguration was not found until ~12 hours after the initial incident began, because investigation was focused on the PVC issue affecting shared workers.

​

Action Items

​

Improvements Done

• Worker trust level and sandbox configuration tests — added dedicated tests covering the trustedEnvironment code path that caused Issue 2, ensuring configuration changes are validated automatically.
• Worker polling resilience tests — tests covering job execution lifecycle, resilience to invalid job data, null polls, unrecognized job types, and mixed valid/invalid sequences.
• Sandbox execution tests — tests for sandbox creation, startup, RPC communication, resource cleanup on timeout or memory issues, and process cleanup.
• Race condition tests for queue dispatcher — 12 tests covering orphaned job handling, double-loop spawn prevention during close, single dequeue concurrency control, and waiter timeout/retry behavior.
• Subflow resume race condition tests — 8 tests covering the race where the engine writes pause metadata to Redis before it’s persisted to DB, verifying Redis fallback when DB is stale.
• Rate limiter concurrency tests — tests for concurrent job slot allocation, idempotency with concurrent dispatch, and per-project isolation.
• End-to-end smoke tests in CI — GitHub Actions workflows that validate health checks and webhook flow execution on both AMD64 and ARM64.
• Benchmark tests in CI — load testing across 6 app/worker configurations measuring throughput, mean latency, P50, and P99.