Hands-On Microservices with Kubernetes

Fine-tuning noisy alerts

Noisy alerts are a problem. If there are too many alerts – especially low-priority ones – then there are two major problems:

• It is distracting to all the people that get notified (especially to the poor engineer being woke up in the middle of the night).
• It might lead to people ignoring alerts.

You don't want to miss an important alert because of a lot of noisy low-priority alerts. Fine-tuning your alerts is an art and an ongoing process.

I recommend reading and adopting My Philosophy on Alerting (https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit) by Rob Ewaschuk (ex-Google site reliability engineer).