Hands-On Infrastructure Monitoring with Prometheus

The query of death

Lastly, care should be taken when crafting queries with overly broad selectors and memory-intensive aggregations. While Prometheus has default checks and bounds implemented to avoid unlimited memory usage (as discussed in Chapter 5, Running a Prometheus Server), it is still possible to run up against a memory limit that is not large enough—either a container limit or even the actual system RAM, which would make the OS unceremoniously terminate the Prometheus server. Compounding the problem, pinpointing which queries are using the most resources is hard, especially in environments where you have little control over what queries are sent to the server; there is no slow query log functionality built in as making it work would ...