Hands-On Infrastructure Monitoring with Prometheus

Alertmanager clustering

The overview of the notification pipeline does not touch on the high availability component of Alertmanager. The way high availability is achieved is by relying on gossip (based on the HashiCorp memberlist, https://github.com/hashicorp/memberlist), instead of using a consensus-based protocol; this means there's no real reason for choosing an odd number of instances in a cluster. Using gossip, the cluster shares the notification log (nflog) between all Alertmanager instances, which in turn will be aware of the collective state of the cluster regarding notifications. In the case of a network partition, there will be notifications being sent from each side of the partition, since logically it's better to receive more ...