book

Becoming a Rockstar SRE

Name: Becoming a Rockstar SRE
ISBN: 9781803239224

by Jeremy Proffitt, Rod Anami L. Anami

April 2023

Intermediate to advanced

420 pages

10h 56m

English

Packt Publishing

Read now

Unlock full access

Becoming a Rockstar SRE
ContributorsAbout the authorsAbout the reviewers
Preface
Who is this book forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your ThoughtsDownload a free PDF copy of this book
Part 1 - Understanding the Basics of Who, What, and Why
Chapter 1: SRE Job Role – Activities and Responsibilities
Making this journey personalSRE driving forcesSRE skillsSRE traitsUnderstanding the mindset and hobbies of an SRESRE affinity gameSRE guiding principlesSRE hobbiesDevOps engineers versus SRE versus othersDevOps and site reliability engineersSoftware and site reliability engineersDescribing an SRE’s main responsibilitiesAn overview of the daily activities of an SREPeople that inspireJeremy’s recognition – Paul Tyma, former CTO, LendingTreeRod’s recognition – Ingo Averdunk, Distinguished Engineer, IBM, and Gene Brown, Distinguished Engineer, KyndrylSummaryFurther reading
Chapter 2: Fundamental Numbers – Reliability Statistics
SLA commitment – a conversation, not a numberInternal partner SLAsExternal partner SLAsThe cost of more 9s in an SLAA final word on SLAsDefining and leveraging SLOs and SLIsSLOsSLOs and timeTracking outage frequency with the MTBFMeasuring the downtime with the MTTRUnderstanding the customer and revenue impactTransparency in outagesThe rockstar SRE’s SLASummary
Chapter 3: Imperfect Habits – Duct Tape Architecture and Spaghetti Code
The business of software development – let’s start with the dollarsDefining the “value” of software to a businessThe value of protecting businessThe value of growing a businessThe value of saving labor costsThe A/B testing mindset – the art of change in customer interactionA/B testing in customer flowsAnalyzing the results of A/B testingLeveraging A/B testing to satisfy quarterly numbersDedication to the craft of development – and why some are just here for a jobA quick guide to communicating with your colleaguesReviewing the merge request – it’s about training, oversight, and reliabilityAvoiding the typical rubber stamp mentalityA word on production deploymentsWhy businesses want us to outright ignore best practicesThe truth about the ownership of a developer’s timeUnderstanding the flaws in how we estimate development costFast, good, cheap – pick oneWhy is observability the answer to reliability issues?The cost of highly available architectureMixing good and bad – tricks to wrapping bad code and making it resilientAlerting that fires actionsAdding additional logging to monitor potential issuesUsing try catch to encapsulate exceptionsRetries to the rescue…or notSummary
Part 2 - Implementing Observability for Site Reliability Engineering
Chapter 4: Essential Observability – Metrics, Events, Logs, and Traces (MELT)
Technical requirementsAccomplishing systems monitoring and telemetryMonitoring targets for infrastructureMonitoring types and toolsMonitoring golden signalsMonitoring dataUnderstanding APMGetting to know topology self-discovery, the blast radius, predictability, and correlationAlerting – the art of doing it quietlyThe user perspective notification trigger principleEvent-to-incident mapping principleMixing everything into observabilityOutages versus downtimeObservability architectureObservability effectivenessIn practice – applying what you have learnedLab architectureLab contentsLab instructionsSummaryFurther reading
Chapter 5: Resolution Path – Master Troubleshooting
Properly defining the problem – and what to ask and not askSource of informationThe knowledge base of the reporterNaming conventionsFalse urgencyExecutive summaryBreaking down and testing systemsBreaking down hardware versus the operating systemBreaking down a web APIUnderstanding the stepsThe problems with this method of troubleshootingPrevious and common events – checking for the simple problemsPrior Root Cause Analysis (RCA) documentsTimeline analysisComparisonThe best approachEffective research both online and among peersThe art of the Google searchSkimming the content quickly and refining itNever forget your internal resourcesBreaking down source code efficientlyCode you’ve never seenWhen that failsLogging plus codeIn practice – applying what you’ve learnedSummary
Chapter 6: Operational Framework – Managing Infrastructure and Systems
Technical requirementsApproaching systems administration as a disciplineDesignInstallationConfigurationApp deploymentManagementUpgradeUninstallationUnderstanding IT service managementITILDevOpsSeeing systems administration as multiple layers and multiple towersAutomating systems provisioning and managementInfrastructure as CodeImmutable infrastructureIn practice – applying what you’ve learnedLab architectureLab contentsLab instructionsSummaryFurther readings

Chapter 7: Data Consumed – Observability Data Science
Technical requirementsMaking data-driven decisionsDefining the question and optionsDetermining which data to useIdentifying which data is already availableCollecting the missing dataAnalyzing all datasets togetherPresenting the decision as a recordDocumenting the lessons learned in the processSolving problems through a scientific approachFormulationHypothesisPredictionExperimentAnalysisUnderstanding the most common statistical methodsPercentagesMean, average, and standard deviationQuantiles and percentilesHistogramsUsing other mathematical models in observabilityVisualizing histograms with GrafanaIn practice – applying what you’ve learnedLab architectureLab contentsLab instructionsSummaryFurther reading
Part 3 - Applying Architecture for Reliability
Chapter 8: Reliable Architecture – Systems Strategy and Design
Technical requirementsDesigning for reliabilityArchitectural aspectsReliability equationsDesign patternsModern applicationsSplitting and balancing the workloadSplittingBalancingFailing over – almost as goodScaling up and out – horizontal versus verticalHorizontalVerticalAutoscalingIn practice – applying what you’ve learnedLab architectureLab contentsLab instructionsSummaryFurther reading
Chapter 9: Valued Automation – Toil Discovery and Elimination
Technical requirementsEliminating toilToil redefinedWhy toil is badHandling toil the right wayTreating automation as a software problemDocumentAlgorithmCodeAutomating the (in)famous CI/CD pipelineContinuous integrationContinuous deliveryProduction releasesIn practice – applying what you’ve learnedLab architectureLab contentsLab instructionsSummaryFurther reading
Chapter 10: Exposing Pipelines – GitOps and Testing Essentials
A basic pipeline – building automation to deploy infrastructure as code architecture and codePipelines in chronological orderPipeline templatesErrors or breaks in pipelinesUsing containers in pipelinesPipeline artifactsPipeline troubleshooting tipsAutomating compliance and security in pipelinesLibrary ageApplication security testingDynamic Application Security Testing (DAST)Static Application Security Testing (SAST)Secrets scanningAutomated linting for code quality and standardsCompiling with linting feedbackValidating functionality during deployment with automated testingWhy is testing so important to reliability?Test dataThe types of testingWhen to test a pipelineTesting observabilityAutomated rollbacksThe reduction of developer toil through automated processesWhat is the impact of addressing toil?In practice – applying what you’ve learnedPreparing AWS for the labCreating your repositoryAdding secrets to your repositoryDownloading and committing the lab filesUnderstanding the pipelineAdding more stepsTesting but not deployingLab final thoughtsSummary
Chapter 11: Worker Bees – Orchestrations of Serverless, Containers, and Kubernetes
Technical requirementsThe multiple definitions of serverlessServerless FrameworkServerless computingServerless functionsMonitoring serverless functionsErrorsContainers and why we love themIsolationImmutabilityPromotabilityTaggingRollbacksSecuritySignableMonitoring containersKubernetes and other ways to orchestrate containersHealth checksCrashing and force-closing containersHTTP-based load balancingServer load balancingContainers as a Service (CaaS)Simple container orchestrationKubernetesDeployment techniques and workersTraditional replacement deploymentRolling deploymentA/B or blue/green deploymentCanary deploymentAutomation and rolling back failed deploymentsRollback metricsWhen to roll backHow to roll backIn practice – applying what you’ve learnedLeveraging Gitpod – a containerized workspaceThe emulation source codeRunning the emulationSummary
Chapter 12: Final Exam – Tests and Capacity Planning
Technical requirementsUnderstanding types of testingDevelopment testsBuild testsDelivery testsDeployment testsProduction testsAdopting TDDUnit testing the hard wayUnit testing with a frameworkUsing test automation frameworksStaying ahead with capacity planningLoad test dataThe capacity curveThe demand curveIn practice – applying what you’ve learnedLab architectureLab contentsLab instructionsSummaryFurther reading
Part 4 - Mastering the Outage Moments
Chapter 13: First Thing – Runbooks and Low Noise Outage Notifications
Technical requirementsWhat makes a good runbook – the basicsRunbooks as living documentsUnderstanding the runbook audience knowledge levelRunbook audience permissionsWhat do you put into a runbook anyway?Beyond the runbook – code and commentsQuickly understanding source codeSearching source code for your needle in a haystackCommenting for understandingWhat’s in a good dashboard?Types of dashboardsNOC-style red and greenDisplaying trendsAggregates and breakdownsWhat dashboards are notThe basics of priority levelsResponse effortEngineer retentionIncident response systems and priorityIncident response systems and phone-based alertsWhat is a priority one event?Defining priority based on...The priority level of observability failuresForcing the priority – the rockstar way!Adjusting alertsLogs and alertingPausing alertsIn practice – applying what you’ve learnedDefining priority levelsCustom hat pricing API runbookAlertingSummary
Chapter 14: Rapid Response – Outage Management Techniques
Where to meet – an effective strategy for communicating good informationOnline collaborationIn-person collaborationThe historical data found in outage responsesParticipantsFollow-up workLeveraging the people involved in the responseTasksParticipants and personalitiesBreak strategy and stress managementThe opportunity to respond at the right timeTrainingRunbook and contact list revisionsTeam buildingExecutive messaging bugs in the earOpportunities to call out during the RCAMessaging customers and leadershipCustomer versus leadership messagingCadenceEmail groupsStatus sitesOver-messagingNotes, notes, notes...In practice – applying what you’ve learnedOutage and alarmNotification and responseTroubleshootingThe conclusionSummary
Chapter 15: Postmortem Candor – Long-Term Resolution
The content of the postmortem in executive summary styleExecutive summary styleOverviewImpactTimelineDetailed technical descriptionResponseResolutionFuture actionsDecisions are not blameBusiness is businessResource and time constraintsMonitoringThe cost of more reliability as a business decisionActive:ActiveManual failoverCost of time to identifyThe cost of time to move a loadHidden development costsTraining and skill sets – they matterIdentifying gapsTraining and certification targetsCreating future action plansImmediate follow-upWho to involveTimelines and priorityAssigning ownershipTracking the workIn-practice – an example of a postmortemWriting the overviewRounding out the postmortemCustom Hat Company postmortemImpactTimelineTechnical details and responseResolutionFuture actionsSummary
Part 5 - Looking into Future Trends and Preparing for SRE Interviews
Chapter 16: Chaos Injector – Advanced Systems Stability
Technical requirementsComprehending the wheel-of-misfortune gameAll ends are new beginningsLessons to be learnedRole-playing scenariosA little bit of gamificationUnderstanding chaos engineering for reliabilityPrinciples of chaos engineeringChaos system architectureChaos experimentsIn practice – employing the wheel-of-misfortune gameLab architectureLab contentsLab instructionsIn practice – injecting chaos into systemsLab architectureLab contentsLab instructionsSummaryFurther reading
Chapter 17: Interview Advice – Hiring and Being Hired
What we’re looking for in a candidateAre you qualified?Entry-level SRE jobProblem-solvingThe ability to accept feedback and directionA broad knowledge base and skill setResearch and learning skill setThe ability to say “No”Culture fitThe X factorPassionExperiencePersonal responsibilityCommon interview questions and answersTechnical questionsNon-technical questionsInsightfully odd questionsWhat should you look for in a career?Define a good bossDotted line reportingMoralsResearching the companyBusiness modelProfitability for the next decadeStructureLarge versus smallPublic versus privateOnline reviewsAre you over-or under-certified?Certifications that matterHow many are too many certifications?RelevancyTips for landing the job with a great salaryInterview tipsSalary negotiationsSummary
Appendix A – The Site Reliability Engineer Manifesto
The manifestoHow to adopt itHow to contribute to it
Appendix B – The 12-Factor App Questionnaire
The questionnaireFactor I – Code baseFactor II – DependenciesFactor III – Config (configuration)Factor IV – Backing (backend) servicesFactor V – Build, release, runFactor VI – ProcessesFactor VII – Port bindingFactor VIII – ConcurrencyFactor IX – DisposabilityFactor X – Development/production (dev/prod) parityFactor XI – LogsFactor XII – Admin processesHow to adopt this questionnaireHow to contribute to this questionnaire
Index
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your ThoughtsDownload a free PDF copy of this book

Overview

Becoming a Rockstar SRE offers a deep dive into the essential practices and strategies of site reliability engineering. You'll learn how to balance reliability, efficiency, and innovation in engineering resilient systems. With real-world examples and actionable tips, this book prepares you to excel in an SRE career.

What this Book will help me do

Master the key principles of site reliability engineering, including reliability metrics and automation.
Design and deploy highly resilient architectures using cutting-edge technologies such as Kubernetes and serverless systems.
Implement effective observability strategies to quickly identify, troubleshoot, and resolve production issues.
Reduce operational toil by employing automation in CI/CD pipelines and applying best practices in maintenance.
Develop critical skills for interacting with stakeholders and navigating outages with professionalism.

Author(s)

Jeremy Proffitt and Rod Anami L. Anami bring a wealth of experience in the field of IT and site reliability engineering. Jeremy has years of experience in system design and automation, while Rod excels in cloud infrastructure and observability practices. Together, they ensure readers gain practical insights.

Who is it for?

This book is ideal for IT professionals aiming to transition into the site reliability engineering domain, such as developers and sysadmins. It is also suited for executives looking to foster reliable systems within their organizations. Readers should have basic knowledge of APIs, web architecture, and cloud services.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781803239224

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills