Loud failures are manageable: a job crashes, an alert fires, someone fixes it. Silent failures are the ones that cost money. An automation that stops receiving input, processes everything as an exception, or drifts away from correct behavior can look healthy on a status page for weeks. Monitoring for these needs to watch outcomes, not just uptime.
The five alerts
- Volume drop — expected input stopped arriving. A mailbox rule changed, an API key expired, an upstream export broke. Alert when throughput falls below its normal range for the time of day or week.
- Volume spike — sudden surges can mean a loop, a duplicate feed or an upstream error. Compare with a rolling baseline.
- Exception rate shift — the share of items sent to humans rising or falling sharply. A jump suggests input changes; a sudden drop to near zero often means the checks stopped working.
- Outcome sampling — regularly pull a random sample of completed items and have a person verify them. This is the only monitor that catches confident wrong answers.
- Latency and age — how long items sit before processing, and the age of the oldest unprocessed item. A stuck queue looks like a quiet system.
A monitor that has never fired deserves suspicion. Test each alert on purpose: stop the input, send a bad batch, break a rule. If nobody is notified, you have a decoration, not a control.
Make alerts actionable
- Name an owner for each alert — a person or rota, not a shared channel nobody reads.
- Say what to check first — a short runbook link in the alert itself.
- Set thresholds from real history — a few weeks of data, adjusted after false alarms. Alerts that cry wolf get muted.
- Include a heartbeat — a regular 'I ran and processed N items' message, so silence itself is an alarm.
Close the loop
Review incidents monthly, and turn each into a new check or a changed threshold. Over time the monitoring set reflects how your automation actually fails, which is far more valuable than a generic dashboard.