Becoming SRE

Chapter 10. Learning from Failure

There are lots of SRE practices in this book, but this is the only one that is getting its own chapter. Learning from failure is at the nexus of an active SRE practice that seeks to lead us to the appropriate level of reliability we desire. To best explain, let’s look at the roads that meet at this crossroads.

First off, you have monitoring/observability—previously described as the most crucial things to get solidly in place as you start your reliability work. That data gives us clarity on the current state of our systems, the what is.

Second, you have work planning processes like SLIs/SLOs, which allow us to specify with a reasonable degree of clarity our intentions and objectives for what should be.

And finally, you have incidents/outages (with the accompanying response practices). They provide (whether we like it or not) data on how what is can diverge or has diverged from what should be.

The practices around learning from failure sit at this crossroads. They allow us to create and nurture the feedback loops that will help us iterate from where we are to where we want to be, using the information we have from how we have diverged. But that only happens if we are intentional, hence, this chapter.

Talking About Failure

Before we get to the specific activities associated with learning from failure, I believe it is important to discuss a few overarching ideas that will inform them. One idea that is very clearly articulated in the resilience engineering ...