Errata for Site Reliability Engineering

On https://landing.google.com/sre/book/chapters/part3.html#id-dA2uaIyFqF4, the link to the US Digital Service page is broken. The new links you should use are either https://obamawhitehouse.archives.gov/participate/united-states-digital-service or https://www.usds.gov/.

Note from the Author or Editor:
Replace https://www.whitehouse.gov/digital/united-states-digital-service with https://www.usds.gov.

On page https://landing.google.com/sre/book/chapters/practical-alerting.html: "The sheer number of components being analyze" "Analyze" should be "analyzed".

In the online version at https://landing.google.com/sre/book/chapters/introduction.html a space is missing between "monitoring" and "strategy" in the sentence […]monitoringstrategy should be constructed thoughtfully[…] and between "common" and "approach" in […]A classic and commonapproach to monitoring[…] and between "an" and "effective" in […]this type of email alerting is not aneffective solution[…] and between "an" and "email" in […]a human to read an emailand decide[…] and between "in" an "response" in […]action needs to be taken inresponse is fundamentally flawed.[…] and "a" and "human" in […]Monitoring should never require ahuman to interpret[…] and between "software" and "should" in […]Instead, softwareshould do the interpreting[…] and between "when" and "they" in […]notified only whenthey need to take action[…]

Note from the Author or Editor:
Fixed on site.

In the online version at https://landing.google.com/sre/book/chapters/introduction.html a space is missing between "both" and "development" in the sentence […]and an occurrence that bothdevelopment and SRE teams[…]

Note from the Author or Editor:
Fixed on site.

In the online version at https://landing.google.com/sre/book/chapters/introduction.html a space is missing between "is" and "useful" in the following sentence: […]in addition had a set of technical skills that isuseful to SRE[…]

Note from the Author or Editor:
Fixed on site.

In the acknowledgements list, "Sean Sechrest" is listed (a Google SRE). Sean's actual name is "Sean Sechrist".

Note from the Author or Editor:
Replaced Sechrest with Sechrist.

https://landing.google.com/sre/sre-book/chapters/addressing-cascading-failures/ There are quite a lot of whitespace characters missing in this chapter. Namely: 1 & 2. A vicious cycle can occur in this scenario:less CPU is available, resulting in slower requests, resulting in increased RAM usage, resulting in more GC, resulting in even lower availability of CPU.This is known colloquially as the “GC death spiral.” One after ":" and another one in "...of CPU.This...". 3. Serve lower-quality, cheaper-to-compute results to the user. Your strategy here will be service-specific.See Load Shedding and Graceful Degra After "be service-specific". 4. Servers should protect themselves from becoming overloaded and crashing.When overloaded at either the frontend or backend layers, fail early and cheaply. For details, see Load Shedding and Graceful Degradation. After "and crashing". 5. Note that because rate limiting often doesn’t take overall service health into account, it may not be able to stop a failure that has already begun.Simple rate-limiting implementations are also likely to leave capacity unused. Rate limiting can be implemented in a number of places: After "has already begun". 6 & 7. Good capacity planning can reduce the probability that a cascading failure will occur.Capacity planning should be coupled with performance testing to determine the load at which the service will fail.For instance, if every cluster’s breaking point is 5,000 QPS, the load is evenly spread across clusters,109 and the service’s peak load is 19,000 QPS, then approximately six clusters are needed to run the service at N + 2. One after "failure will occur" and the other one after "service will fail". 8. Most thread-per-request servers use a queue in front of a thread pool to handle requests.Requests come in, they sit on a queue, and then threads pick requests off the queue and perform the actual work (whatever actions are required by the server). Usually, if the queue is full, the server will reject new requests. After "handle requests". 9. More sophisticated approaches include identifying clients to be more selective about what work is dropped, or picking requests that are more important and prioritizing.Such strategies are more likely to be needed for shared services. After "are more important and prioritizing". 10 & 11. Suppose the code in the frontend that talks to the backend implements retries naively.It retries after encountering a failure and caps the number of backend RPCs per logical request to 10.Consider this code in the frontend, using gRPC in Go: After "implements retries naively" and "RPCs per logical request to 10". 12. Think about the service holistically and decide if you really need to perform retries at a given level. In particular, avoid amplifying retries by issuing retries at multiple levels: a single request at the highest layer may produce a number of attempts as large as the product of the number of attempts at each layer to the lowest layer. If the database can’t service requests because it’s overloaded, and the backend, frontend, and JavaScript layers all issue 3 retries (4 attempts), then a single user action may create 64 attempts (4^3) on the database.This behavior is undesirable when the database is returning those errors because it’s overloaded. After "on the database". 13. Suppose an RPC has a 10-second deadline, as set by the client.The server is very overloaded, and as a result, it takes 11 seconds to move from a queue to a thread pool. At this point, the client has already given up on the request. Under most circumstances, it would be unwise for the server to attempt to handle this request, because it would be doing work for which no credit will be granted—the client doesn’t care what work the server does after the deadline has passed, because it’s given up on the request already. After "as set by the client". 14. Suppose that the frontend from the preceding example consists of 10 servers, each with 100 worker threads. This means that the frontend has a total of 1,000 threads of capacity.During usual operation, the frontends perform 1,000 QPS and requests complete in 100 ms. This means that the frontends usually have 100 worker threads occupied out of the 1,000 configured worker threads (1,000 QPS * 0.1 seconds). After "1,000 threads of capacity". 15. Suppose an event causes 5% of the requests to never complete.This could be the result of the unavailability of some Bigtable row ranges, which renders the requests corresponding to that Bigtable keyspace unservable. As a result, 5% of the requests hit the deadline, while the remaining 95% of the requests take the usual 100 ms. After "5% of the requests to never complete". 16. Employ general cascading failure prevention techniques.In particular, servers should reject requests when they’re overloaded or enter degraded modes, and testing should be performed to see how the service behaves after events such as a large restart. After "failure prevention techniques". 17. Understand how large clients use your service.For example, you want to know if clients: After "clients use your service". 18. If servers are somehow wedged and not making progress, restarting them may help.Try restarting servers when: After "them may help".

Note from the Author or Editor:
Yes, this is likely an HTML issue; Google has opened a web bug to address

The first bullet point is missing a close parenthesis: "(e.g., our HTTP response code on the http_responses variable" I am reading this in the free online version at https://landing.google.com/sre/sre-book/chapters/practical-alerting/, so I am unable to give a page number.

If correct ordering of references is first by author and then by publication date, then: Dea13, Dea04,Dea07 Should be ordered: Dea04,Dea07, Dea13

"...SRE can be broken down into two main categories."

Note from the Author or Editor:
"As a whole, SRE can be broken down two main categories." should be "As a whole, SREs can be broken down into two main categories.".

"If 1% of your requests are 10x the average, " should be "If 1% of your requests are 50x the average, " as it's 5s (99-percentile) / 100ms (average).

Note from the Author or Editor:
Change "10x" to "50x".

"why each clusters took six or more weeks" "clusters" should be singular.

Note from the Author or Editor:
Replace "why each clusters took six or more weeks" with "why each cluster took six or more weeks".

SNMP abbreviation is decoded with an error. Book says: SNMP (Simple Networking Monitoring Protocol) but should be: SNMP (Simple *Network Management* Protocol) See referenced in book source for the prove: https://technet.microsoft.com/en-us/library/cc776379(v=ws.10).aspx or Wikipedia: https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol

Note from the Author or Editor:
Replace "Simple Networking Monitoring Protocol" with "Simple Network Management Protocol".

For the results of task:http_requests:rate10m rule, hostnames in the instance labels should be host0 through host4. .

Note from the Author or Editor:
Replace host2 .. host5 with host1 .. host4.

Missing ');' in second rule. 'jobwebserver' should be 'job=webserver' in third rule.

Should be a space in 'usable.Retain'.

In the first sentence of the paragraph, "a subset of servers is upgraded", but few sentences later, "the single modified server can be quickly reverted". Either "a subset of servers" or "single (modified) server", the number of the servers should be the same in those sentences, to revert canary-ing properly.

Note from the Author or Editor:
Replace "the single modified server" with "the modified servers", and in the next sentence, replace "the upgraded server" with "the upgraded servers".

Missing ')' somewhere in the following sentence: This component formulates a machine-readable request (a protocol buffer that can be understood by the Auxon Solver.

Note from the Author or Editor:
Replace 'request (a' with 'request: a'.

"This is the most important important exercise you should conduct in order to prevent server overload." should probably only have "important" once.

Note from the Author or Editor:
s/most important important exercise/most important exercise/

"Process 3 sends a conflicting Prepare messge" Should be "Process 3 sends a conflicting Prepare message"

Note from the Author or Editor:
Replace 'messge' with 'message' in figure 23-8.

"... run once a month should not be be skipped." have an extra "be". Should be "... run once a month should not be skipped".

Note from the Author or Editor:
Remove excess 'be'.

The sentence "In either case, ..." should not be on the second list item as "either" refers to both types of fires.

Note from the Author or Editor:
Move sentence "In either case, the team needs to build tools to control the burn." out of the list into its own paragraph.

The second sentence of Footnote 3 might be missing some words (e.g., add "for example" before "adding specific ..." and also add some description at the end of the sentence why it is bad).

Note from the Author or Editor:
Add "such as" before "adding specific monitoring/alerting".

Jai13 points to https://research.google.com/pubs/pub41761.html, but should point to https://research.google.com/pubs/pub41761.pdf.