Automated reports and alerts that tell us something is wrong before a customer does.
Enterprise OOM Report
Runs that run out of memory end with status MEMORY_LIMIT_EXCEEDED. They are not failed queue jobs, so queue alerts never see them. This report surfaces them so we can reach out with a fix (bigger dedicated workers, flow optimization) before the customer escalates.
- Runs: as a daily cron job on the DevOps machine, reading the read-only Postgres replica and storing one row per platform per day in ClickHouse.
- Who: enterprise Cloud platforms only, every platform with at least one OOM run the previous day.
- Where: one message in the internal Discord channel, one line per platform with OOM runs, flows affected and OOM rate. Nothing is sent on a quiet day.
- History: the Enterprise OOM Tracker dashboard in HyperDX shows trends and every past day’s report.
- If it stops: a Betterstack heartbeat alerts when the daily job has not run.
Nightly Benchmark
The Benchmark GitHub Actions workflow (.github/workflows/benchmark.yml) runs every night at 03:00 UTC, once per worker shape (shared and dedicated).
- Results: each scheduled run stores one row per shape in ClickHouse. The Nightly Benchmark dashboard in HyperDX charts throughput, latency, queue wait and worker resources over time.
- Regression alert:
benchmark/detect-regression.mjs compares the run with the median of the last 14 runs and alerts Betterstack when a metric is more than 5% worse on 2 runs in a row. It stays silent until a shape has 3 runs of history.
- Pipeline alert: on scheduled runs only, the job’s last step opens a
bench-pipeline-<shape> alert in Betterstack when anything in the job failed, and resolves it on the next green scheduled run.
Manual runs (workflow_dispatch) skip the ClickHouse writes and both Betterstack alert steps. A green manual run does not prove these steps work or resolve an existing pipeline alert. Check the next scheduled run after changing them or their secrets.
