Hands-On Infrastructure Monitoring with Prometheus

Alternatives to sharding

As mentioned at the beginning of this chapter, a single Prometheus instance will take you a long way if configured and utilized properly. Striving to avoid high cardinality metrics should be a top concern, mitigate the need to start sharding. Something that helps to protect Prometheus instances from scraping targets that produce unreasonable amounts of metrics involves defining the maximum sample ingestion per scrape job. For this, you just need to add sample_limit to the scrape job; if the number of samples after metric_relabel_configs goes over the configured limit, the scrape will be dropped entirely. An example configuration is as follows:

scrape_configs:  - job_name: 'massive_job'    sample_limit: 5000 static_configs: ...