Skip to Content
Site Reliability Engineering
book

Site Reliability Engineering

by Niall Richard Murphy, Betsy Beyer, Chris Jones, Jennifer Petoff
April 2016
Intermediate to advanced
552 pages
15h 44m
English
O'Reilly Media, Inc.
Audiobook available
Content preview from Site Reliability Engineering

Appendix F. Example Production Meeting Minutes

Date: 2015-10-23

Attendees: agoogler, clarac, docbrown, jennifer, martym

Announcements:

  • Major outage (#465), blew through error budget

Previous Action Item Review

  • Certify Goat Teleporter for use with cattle (bug 1011101)

    • Nonlinearities in mass acceleration now predictable, should be able to target accurately in a few days.

Outage Review

  • New Sonnet (outage 465)

    • 1.21B queries lost due to cascading failure after interaction between latent bug (leaked file descriptor on searches with no results) + not having new sonnet in corpus + unprecedented & unexpected traffic volume

    • File descriptor leak bug fixed (bug 5554825) and deployed to prod

    • Looking into using flux capacitor for load balancing (bug 5554823) and using load shedding (bug 5554826) to prevent recurrence

    • Annihilated availability error budget; pushes to prod frozen for 1 month unless docbrown can obtain exception on grounds that event was bizarre & unforeseeable (but consensus is that exception is unlikely)

Paging Events

  • AnnotationConsistencyTooEventual: paged 5 times this week, likely due to cross-regional replication delay between Bigtables.

    • Investigation still ongoing, see bug 4821600

    • No fix expected soon, will raise acceptable consistency threshold to reduce unactionable alerts

Nonpaging Events

  • None

Monitoring Changes and/or Silences

  • AnnotationConsistencyTooEventual, acceptable delay threshold raised from 60s to 180s, see bug 4821600; ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

AirBnbBlueOriginElectronic ArtsHomeDepotNasdaqRakutenTata Consultancy Services

QuotationMarkO’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.
Julian F.
Head of Cybersecurity
QuotationMarkI wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.
Addison B.
Field Engineer
QuotationMarkI’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.
Amir M.
Data Platform Tech Lead
QuotationMarkI'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.
Mark W.
Embedded Software Engineer

You might also like

Site Reliability Engineering Fundamentals

Site Reliability Engineering Fundamentals

Emil Stolarsky, Jaime Woo
Site Reliability Engineering: How Google Runs Production Systems

Site Reliability Engineering: How Google Runs Production Systems

Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy
Observability Engineering

Observability Engineering

Charity Majors, Liz Fong-Jones, George Miranda
AI Engineering

AI Engineering

Chip Huyen

Publisher Resources

ISBN: 9781491929117Errata Page