Hands-On Microservices with Kubernetes

Warnings versus alerts

Let's consider our out of disk space situation again. This is a case were the state gets worse over time. The disk space is gradually reduced as more and more data is logged to the log files. If you've got nothing in place, you will discover that you've ran out of disk space when an application starts to issue strange errors, often downstream from the actual failure, and you'll have to trace it back to the source. I've been there and done that; it's no fun. A better approach is to check the disk space regularly and raise an alert when it exceeds a certain threshold (for example, 95%). But why wait until the situation becomes critical? In such gradually worsening situations, it is much better to detect the problem early ...