book

Seeking SRE

by David N. Blank-Edelman

September 2018

Intermediate to advanced

587 pages

17h 34m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

And So It Begins...Origin StoryVoicesForward in All Directions!1Acknowledgments
Interviewing 101Who Is InvolvedIndustry Versus UniversityBiasesThe FunnelSRE FunnelsPhone ScreensThe Onsite InterviewTake-Home QuestionsAdvice for Hiring ManagersFinal Thoughts on Interviewing SREsFurther Reading
Choose SRE for the Right ReasonsOrienting to a Data-Driven ApproachCommitment to SREMaking a Decision About SRE
The Virtuous Cycle to the Rescue: If You Don’t Measure It…Metrics Review: If a Metric Falls in the Forest…Surrogate MetricsRepair DebtVirtual Repair Debt: Exorcising the Ghost in the MachineReal-Time Dashboards: The Bread and Butter of SRELearnings: TL;DRFurther Reading
Build, Buy, or Adopt?Establish ImportanceIdentify StakeholdersMake a DecisionAcknowledge RealityThird Parties as First-Class CitizensWhen They’re Down, You’re DownRunning the Black Box Like a ServiceService-Level Indicators, Service-Level Objectives, and SLAsPlaybook: From Staging to ProductionClosing Thoughts
SREs to the Rescue! (and How They Failed)A Matter of Scale in Terms of HeadcountThe Embedded SREYou Build It, You Run ItThe Deployment PlatformClosing the Loop: Take Your Own PagerIntroducing Production EngineeringSome Implementation DetailsDevelopers’ Productivity and Health Versus the PagerResolving Cross-Team Reliability Issues by Using PostmortemsUniform Infrastructure and Tooling Versus Autonomy and InnovationGetting Buy-InConclusionFurther Reading
Tabula Rasa: 2006–2007PreludeKey LearningsBeta and Release: 2008–2009PreludeBringing Scalability and Reliability to the ForefrontKey LearningsThe Curse of Success: 2010PreludeA New Ownership ModelFormalizing Core ServicesBlessed Deployment Time SlotsOn-Call and AlertingSpawning Off Internal Office SupportAddressing the Remaining Top ConcernsCreating DetectivesKey LearningsPets and Cattle, and Agile: 2011PreludeForming Bad HabitsBreaking Those Bad HabitsKey LearningsA System That Didn’t Scale: 2012PreludeManual Work Hits a CliffKey LearningsIntroducing Ops-in-Squads: 2013–2015PreludeBuilding on TrustDriving the Paradigm ShiftKey LearningsAutonomy Versus Consistency: 2015–2017PreludeBenefitsTrade-OffsKey LearningsThe Future: Speed at Scale, Safely
BackgroundIntroducing SREDefining Current StateIdentifying and Educating StakeholdersPresenting the Business CaseImplementing the SRE TeamLessons LearnedSample Implementation RoadmapClosing ThoughtsFurther Reading

Clarifying TerminologyService-Level IndicatorSLAService-Level ObjectiveEstablishing SLAs for Internal ComponentsUnderstanding External DependenciesNontechnical SolutionsTracking Availability LevelDealing with Corner CasesConclusion
Toil, the Enemy of SREToil in the EnterpriseSilos, Queues, and TicketsSilos Get in the WayTicket-Driven Request Queues Are ExpensiveTake Action NowStart by Leaning on LeanGet Rid of as Many Handoffs as PossibleReplace Remaining Handoffs with Self-ServiceSelf-Service Is More Than a ButtonSelf-Service Helps SREs in Multiple WaysOperations as a ServiceError Budgets, Toil Limits, and Other Tools for Empowering HumansError BudgetsToil LimitsLeverage Existing Enthusiasm for DevOpsUnify Backlogs and Protect CapacityPsychological Safety and Human FactorsJoin the Movement
Pattern 1: Birth of Automated Testing at GooglePattern 2: Launch and Handoff Readiness Review at GooglePattern 3: Create a Shared Source Code RepositoryConclusionFurther Reading and Source Material
BackgroundMethodResultsReplies
The Problem with SystemsEconomic Pillars of ComplexityBeginning ChaosNavigating Complexity for SafetyChaos Goes BigFormalizationAdvanced PrinciplesFrequently Asked QuestionsConclusion
The Intersection of Reliability and PrivacyThe General Landscape of Privacy EngineeringPrivacy and SRE: Common ApproachesReducing ToilEfficient and Deliberate Problem SolvingRelationship ManagementEarly Intervention and Education Through EvangelismNuances, Differences, and Trade-OffsConclusionFurther Reading
Guiding Principles of the Database Reliability EngineerProtect the DataSelf-Service for ScaleDatabases Are Not SpecialA Culture of Database Reliability EngineeringRecoverabilityConsiderations for RecoveryAnatomy of a Recovery StrategyBuilding Block 1: DetectionBuilding Block 2: Diverse StorageBuilding Block 3: A Varied ToolboxBuilding Block 4: TestingChampioning Recovery ReliabilityContinuous Delivery: From Development to ProductionEducation and CollaborationCollaborationDeploymentMigrations and VersioningImpact AnalysisMigration PatternsChampioning CDMaking the Case for DBREFurther Reading
Replication Is Table StakesBackupsReplicationReal-World DurabilityIsolationProtectionTestingSafeguardsRecoveryVerificationThe Power of ZeroVerification CoverageWatching the WatchersAutomationWindow of VulnerabilityOperator FatigueReliabilityConclusion
Why Use Machine Learning for SRE?Why and How Should My Company Be Engaging in This?Some SRE Problems Machine Learning Can Help SolveThe Awakening of Applied AIWhat Is Machine Learning?What Do We Mean by Learning?From Chess to Go: How Deep Can We Dive?Why Now? What Changed for Us?What Are Neural Networks?Neurons and Neural NetworksHow and When Should We Apply Neural Networks?What Kinds of Data Can We Use?Practical Machine LearningPopular Libraries for Neural NetworksPractical Machine Learning ExamplesSuccess StoriesFurther ReadingMy GitHub RepositoryRecommended Books
Defining Quality: What Do Good Docs Look Like?Functional Requirements for SRE DocumentationIntegrating Docs into the Engineering WorkflowThe Google Experience: g3doc and EngPlayWhat We LearnedDoing Docs Better: Best PracticesCreate Templates for Each Documentation TypeBetter > Best: Set Realistic Standards for QualityRequire Docs as Part of Code ReviewRuthlessly Prune Your DocsRecognize and Reward DocumentationCommunicating the Value of DocumentationFurther Reading
Active LearningActive Learning Example: Wheel of MisfortuneActive Learning Example: Incident Manager (a Card Game)Active Learning Example: SRE ClassroomThe Costs of Failing to LearnLearning Habits of Effective SRE TeamsProduction MeetingsPostmortemsA Call to Action: Ditch the Boring Slides
Why Set Goals?AvailabilityTime QuantaTransactionsTransactions over Time QuantaOn Evaluating SLOsHistogramsWhere Percentiles Fall Down (and Histograms Step Up)Parting Thought: Looking at SLOs Upside DownFurther Reading
Where Did SRE Come From?Key Values for SREKeeping the Site UpEmpowering Teams to “Do the Right Thing”Approaching Operations as an Engineering ProblemAchieving Business Success Through Promises (Service Levels)Critical Enabling Functions of SREMonitoring, Metrics, and KPIsIncident Management and Emergency ResponseCapacity Planning and Demand ForecastingPerformance Analysis and OptimizationProvisioning, Change Management, and VelocityPhases of SRE ExecutionPhase 1: Firefighting/ReactivePhase 2: GatekeepersPhase 3: Advocates/PartnersPhase 4: CatalyticComplications of Differing PhasesFocus on the Details of SuccessFurther Reading
Antipattern 1: Site Reliability OperationsAntipattern 2: Humans Staring at ScreensAntipattern 3: Mob Incident ResponseAntipattern 4: Root Cause = Human ErrorAntipattern 5: Passing the PagerAntipattern 6: Magic Smoke Jumping!Antipattern 7: Alert Reliability EngineeringAntipattern 8: Hiring a Dog-Walker to Tend Your PetsAntipattern 9: Speed-Bump EngineeringAntipattern 10: Design ChokepointsAntipattern 11: Too Much Stick, Not Enough CarrotAntipattern 12: Postponing ProductionAntipattern 13: Optimizing Failure Avoidance Rather Than Recovery Time (MTTF > MTTR)Antipattern 14: Dependency HellAntipattern 15: Ungainly GovernanceAntipattern 16: Ill-Considered SLOh-OhsAntipattern 17: Tossing Your API Over the FirewallAntipattern 18: Fixing the Ops TeamSo, That’s It, Then?
Scalability, Reliability, and PerformanceFailure RecoverySimpler OperationsFaster Startup TimesKnown StateContinuous Integration/Continuous Deployment with ConfidenceSecurityMultiregion OperationsRelease EngineeringBuilding the Base ImageDeploying ApplicationsDisadvantagesConclusion
Scriptable Load Balancers: The New Kid on the BlockWhy Scriptable Load Balancers?Making the Difficult EasyShard-Aware RoutingHarnessing PotentialCase Study: IntermissionService-Level MiddlewareMiddleware to the RescueAPIs of Service-Level MiddlewareCase Study: WAF/Bot MitigationAvoiding DisasterGetting Clever with StateCase Study: Checkout QueueLooking to the Future and Further Reading
Ready to Get Rid of the Monolith?Current State of Microservice NetworkingService Mesh to the RescueThe Benefits of a Sidecar ProxyEventually Consistent Service DiscoveryObservability and AlarmingSidecar Performance ImplicationsThin Libraries and Context PropagationConfiguration Management (Control Plane Versus Data Plane)The Service Mesh in PracticeThe Origin and Development of Envoy at LyftOperating Envoy at LyftThe Future of the Service MeshFurther Reading
The Primary Indicator of a Successful TeamHow to Build Psychological Safety into Your Own TeamFurther Reading
IntroductionWhat Do SRE People Do?Why Should We Care About Practitioner Cognition?Critical Decisions Made Under Uncertainty and Time Pressure Cannot Be ScriptedHuman Performance in Modern Complex Systems: The Main ThemesObservations on SRE Cognitive Work Around IncidentsEvery Incident Could Have Been WorseSacrifice Decisions Take Place Under UncertaintyRepairs to Functional SystemsSpecial Knowledge About Complex SystemsManaging the Costs of CoordinationSREs Are Cognitive Agents Working in a Joint Cognitive SystemThe Calibration ProblemMental ModelsIncidents Trigger Individual RecalibrationIncidents Are Opportunities for Collective RecalibrationWhat Are the Implications of All This?Incidents Will ContinueIncidents Will Impose CostsIncident Patterns Will ChangeIncidents Point to Specific Calibration Problems and LocationsWhat Should Happen Next?Build a Corpus of CasesFocus on Making Automation a Team Player in SRE WorkAddress the Calibration ProblemWhat Can You Do?ConclusionReferences
Defining Mental DisordersMental Disorders Are Missing from the Diversity ConversationSanity Isn’t a Business RequirementThoughts and Prayers Aren’t ScalableFull-Stack InclusivityApplicationInterviewingCompensationBenefitsOnboardingWorking ConditionsJob DutiesTrainingPromotionLeavingInclusivity for Anyone Helps EveryoneMental Disorder Resources
The Rationale for On-CallFirst, Do No HarmParallels with SREDifferences with SREUnderlying Assumptions Driving On-Call for EngineersOn-Call Is Emergency Medicine Instead of Ward MedicineCounterargumentsThe Cost to Humans of Doing On-CallWe don’t need another heroActual SolutionsTrainingPrioritizationImproving On-the-Job PerformanceWe Need a Fundamental Change in ApproachStrong-Anti-On-CallWeak-Anti-On-CallA Union of the TwoConclusion
The Computer and Human Systems Cannot Be SeparatedDecoherence and Cascading FailureAlways in a State of Partial FailureNovelty Priority InversionNobody Anticipates the Overhead of CoordinationYour healthcare.gov Is Out ThereTo Get InvolvedFurther Reading
Before, During, AfterCreating the Perfect PlanPrinciples of OrganizingManaging Crisis: Responding When Things Break DownWriting Our Own History: Making Sense of What Went DownThe Long Tail: Turning Action into ChangeActivism and Change Within a CompanyConclusion

Content preview from Seeking SRE

Chapter 12. DevOps and SRE: Voices from the Community

As told to David N. Blank-Edelman

Background

From almost the very beginning of my time interacting with the SRE community, I’ve been curious about the relationship between DevOps and SRE. I’ve had the pleasure of talking about this with many smart people and heard many smart things. As far as I can tell, it’s not a settled question. Each person I have talked to added something to my understanding. When it came time to find a contributor on this topic, it seemed like the best thing I could do would be to invite as many voices into the discussion as possible.

Welcome to an experiment—an entirely crowdsourced chapter.

Method

At the end of February 2018, I put up a website with a page that asked the following:

In two paragraphs or less, what do you think is the relationship between DevOps and SRE? How are they similar? How are they different? Can both be implemented at every organization? Can the two exist in the same org at the same time? And so on…

I put a call out via Twitter and LinkedIn to my professional social network for contributions (and to ask them to pass a pointer to this web page to their network, as well, which many kindly did). In the end, Google Analytics reports 1,165 people from 34 separate countries came to the contribution page.

Results

It was thrilling to see the sorts of thoughtful answers I received to these questions. I present to you now, only slightly copyedited, a sample of the responses¹ I received ...