Consul: Up and Running

Chapter 7. Observability

Observability is the ability to understand the state of a running system as an outside observer. Good observability is crucial in software systems because problems that occur in production often can’t be reproduced in development environments.

When a system with good observability has an issue, engineers can quickly investigate and figure out what’s happening by looking at the observability data. For example, if a service seems to be responding slowly, they may be able to look at the metrics and see that the machine the service is running on is using 100% of its CPU. Or, if users see a blank page when loading a frontend service, engineers might be able to look at the service’s logs and see that it’s erroring because it can’t connect to its database.

In contrast, a system with poor observability is difficult to debug. If users are reporting errors that can’t be reproduced in a development environment, then the engineers are left guessing as to the cause.

An observable system exposes three key types of telemetry:¹

Metrics

Statistics about the service or the underlying infrastructure. For example, how many requests per second the service is receiving or the current CPU and memory usage. Metrics are often viewed in aggregate—for example, the average number of requests per second across all instances of a particular service.

Logs

Messages written by service developers that are output in response to certain events that occur in the service. For example, ...