Hands-On Infrastructure Monitoring with Prometheus

PromQL approach

The container_cpu_usage_seconds_total metric gives us the amount of CPU seconds spent running each container, and comes from the cadvisor exporter. The node_exporter version can be found in the node_exporter_build_info metric. To make things a bit harder, since the container metrics come from cadvisor, the container and pod registered in those metrics are the cadvisor ones and not from the target pods; however, we can find the original container names and pod names in the container_label_io_kubernetes_container_name and container_label_io_kubernetes_pod_name labels, respectively.

The first thing we need to do is get the average number of CPU seconds per second each pod is using on a rolling window of one minute:

sum by (container_label_io_kubernetes_ ...