Hands-On Infrastructure Monitoring with Prometheus

inhibit_rules

In this section, we will add rules to inhibit alerts. The way this works is by matching source alerts and muting target ones by using matchers on both. The only requirement is that the labels on the target and source match in terms of both label name and value, for example:

inhibit_rules:  - source_match:      job: 'icmp'    target_match_re:      alertname: (AlertmanagerDown|NodeExporterDown)    equal: ['base_instance']

In this example, we can read the following: if there's an alert with the job label set to icmp, mute all other alerts with an alertname that matches AlertmanagerDown or NodeExporterDown when base_instance is the same across all matched alerts. In other words, if the instance that is running Alertmanager and Node Exporter is down, ...