Site Reliability Engineering

Chapter 30. Embedding an SRE to Recover from Operational Overload

Written by Randall Bosetti

Edited by Diane Bates

It’s standard policy for Google’s SRE teams to evenly split their time between projects and reactive ops work. In practice, this balance can be upset for months at a time by an increase in the daily ticket volume. A burdensome amount of ops work is especially dangerous because the SRE team might burn out or be unable to make progress on project work. When a team must allocate a disproportionate amount of time to resolving tickets at the cost of spending time improving the service, scalability and reliability suffer.

One way to relieve this burden is to temporarily transfer an SRE into the overloaded team. Once embedded in a team, the SRE focuses on improving the team’s practices instead of simply helping the team empty the ticket queue. The SRE observes the team’s daily routine and makes recommendations to improve their practices. This consultation gives the team a fresh perspective on its routines that team members can’t provide for themselves.

When you are using this approach, it isn’t necessary to transfer more than one engineer. Two SREs don’t necessarily produce better results and may actually cause problems if the team reacts defensively.

If you are starting your first SRE team, the approach outlined in this chapter will help you to avoid turning into an operation team solely focused on a ticket rotation. If you decide to embed yourself or one of your reports ...