Anomaly detection is easy to build and hard to make useful. If it fires constantly, teams ignore it; if it's too loose, it misses real incidents.
Design choices
- Account for seasonality — weekday, month-end and holiday patterns are not anomalies.
- Alert on impact, not just deviation: tie signals to a business consequence.
- Group related alerts into one incident.
- Give each alert an owner and a first step.
- Review alerts regularly and retire or tune the noisy ones.
Track what share of alerts led to action. A low share means retune before adding more.