book

Chaos Engineering

by Casey Rosenthal, Nora Jones

April 2020

Intermediate to advanced

305 pages

8h 45m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Conventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Management Principles as CodeChaos Monkey Is BornGoing BigFormalizing the DisciplineCommunity Is BornFast Evolution
Contemplating ComplexityEncountering ComplexityExample 1: Mismatch Between Business Logic and Application LogicExample 2: Customer-Induced Retry StormExample 3: Holiday Code FreezeConfronting ComplexityAccidental ComplexityEssential ComplexityEmbracing Complexity
Dynamic Safety ModelEconomicsWorkloadSafetyEconomic Pillars of ComplexityStateRelationshipsEnvironmentReversibilityEconomic Pillars of Complexity Applied to SoftwareThe Systemic Perspective
What Chaos Engineering IsExperimentation Versus TestingVerification Versus ValidationWhat Chaos Engineering Is NotBreaking StuffAntifragilityAdvanced PrinciplesBuild a Hypothesis Around Steady-State BehaviorVary Real-World EventsRun Experiments in ProductionAutomate Experiments to Run ContinuouslyMinimize Blast RadiusThe Future of “The Principles”
Retrofitting ChaosDesign Patterns Common in Older SystemsDesign Patterns Common in Newer SystemsGetting to Basic Fault ToleranceDisasterpiece TheaterGoalsAnti-GoalsThe ProcessPreparationThe ExerciseDebriefingHow the Process Has EvolvedGetting Management Buy-InResultsAvoid Cache InconsistencyTry, Try Again (for Safety)Impossibility ResultConclusion
Life of a DiRT TestThe Rules of EngagementWhat to TestHow to TestGathering ResultsScope of Tests at GoogleConclusion
Why Is Everything So Complicated?An Example of Unexpected ComplicationsA Simple System Is the Tip of the IcebergCategories of Experiment OutcomesKnown Events/Unexpected ConsequencesUnknown Events/Unexpected ConsequencesPrioritization of FailuresExplore DependenciesDegree of VariationVarying FailuresCombining Variation and PrioritizationExpanding Variation to DependenciesDeploying Experiments at ScaleConclusion

Learning from DisasterGranularly Targeting ExperimentsExperimenting at Scale, SafelyIn Practice: LinkedOutFailure ModesUsing LiX to Target ExperimentsBrowser Extension for Rapid ExperimentationAutomated ExperimentationConclusion
A Capital One Case StudyBlind Resiliency TestingTransition to Chaos EngineeringChaos Experiments in CI/CDThings to Watch Out for While Designing the ExperimentToolingTeam StructureEvangelismConclusion
Chaos Engineering and ResilienceSteps of the Chaos Engineering CycleDesigning the ExperimentTool Support for Chaos Experiment DesignEffectively Partnering InternallyUnderstand Operating ProceduresDiscuss ScopeHypothesizeConclusion
Humans in the SystemPutting the “Socio” in Sociotechnical SystemsOrganizations Are a System of SystemsEngineering Adaptive CapacitySpotting Weak SignalsFailure and Success, Two Sides of the Same CoinPutting the Principles into PracticeBuild a HypothesisVary Real-World EventsMinimize the Blast RadiusCase Study 1: Gaming Your Game DaysCommunication: The Network Latency of Any OrganizationCase Study 2: Connecting the DotsLeadership Is an Emergent Property of the SystemCase Study 3: Changing a Basic AssumptionSafely Organizing the ChaosAll You Need Is Altitude and a DirectionClose the LoopsIf You’re Not Failing, You’re Not Learning
The Why, How, and When of ExperimentsThe WhyThe HowThe WhenFunctional Allocation, or Humans-Are-Better-At/Machines-Are-Better-AtThe Substitution MythConclusion
Choosing ExperimentsRandom SearchThe Age of the ExpertsObservability: The OpportunityObservability for Intuition EngineeringConclusion
Ephemeral Nature of Incident ReductionKirkpatrick ModelLevel 1: ReactionLevel 2: LearningLevel 3: TransferLevel 4: ResultsAlternative ROI ExampleCollateral ROIConclusion
Collaborative MindsetsOpen Science; Open SourceOpen Chaos ExperimentsExperiment Findings, Shareable ResultsConclusion
AdoptionWho Bought into Chaos EngineeringHow Much of the Organization Participates in Chaos EngineeringPrerequisitesObstacles to AdoptionSophisticationPutting It All Together
Where CV Comes FromTypes of CV SystemsCV in the Wild: ChAPChAP: Selecting ExperimentsChAP: Running ExperimentsThe Advanced Principles in ChAPChAP as Continuous VerificationCV Coming Soon to a System Near YouPerformance TestingData ArtifactsCorrectness
The Rise of Cyber-Physical SystemsFunctional Safety Meets Chaos EngineeringFMEA and Chaos EngineeringSoftware in Cyber-Physical SystemsChaos Engineering as a Step Beyond FMEAProbe EffectAddressing the Probe EffectConclusion
What Is Human and Organizational Performance (HOP)?Key Principles of HOPPrinciple 1: Error Is NormalPrinciple 2: Blame Fixes NothingPrinciple 3: Context Drives BehaviorPrinciple 4: Learning and Improving Is VitalPrinciple 5: Intentional Response MattersHOP Meets Chaos EngineeringChaos Engineering and HOP in PracticeConclusion
Why Do We Need Chaos Engineering?Robustness and StabilityA Real-World ExampleApplying Chaos EngineeringOur Way of Embracing ChaosFault InjectionFault Injection in ApplicationsFault Injection in CPU and MemoryFault Injection in the NetworkFault Injection in the FilesystemDetecting FailuresAutomating ChaosAutomated Experimentation Platform: SchrodingerSchrodinger WorkflowConclusion
A Modern Approach to SecurityHuman Factors and FailureRemove the Low-Hanging FruitFeedback LoopsSecurity Chaos Engineering and Current MethodsProblems with Red TeamingProblems with Purple TeamingBenefits of Security Chaos EngineeringSecurity Game DaysExample Security Chaos Engineering Tool: ChaoSlingrThe Story of ChaoSlingrConclusionContributors/Reviewers

Content preview from Chaos Engineering

Chapter 4. Slack’s Disasterpiece Theater

Richard Crowley

How do you get into Chaos Engineering if your team and tools weren’t born with it? It can seem like an overwhelming and insurmountable task to retrofit chaos into systems designed with the mindset that computers can and should last a long time. Complex systems born from this mindset tend to be less accommodating of extreme transience in the underlying computers than their cloud native successors. Such systems probably perform very well in optimal conditions but degrade quickly and sometimes catastrophically in case of failure.

You may be the proud owner of just such a system. It wasn’t designed to accommodate chaos, but whether you like it or not, chaos is coming as its scale increases and ongoing development asks it to do more, faster, and more reliably. There isn’t time for a rewrite—the system is under duress already. Applying new Chaos Engineering practices to old systems is liable to make the situation worse. You need a different strategy.

This chapter describes one strategy for safely and systematically testing complex systems that weren’t necessarily designed with Chaos Engineering in mind by introducing failures and network partitions in a thoughtful and controlled manner. This is a process, not automation, that helps your team understand how your software is vulnerable, motivates improvements, and validates that systems tolerate the faults you can anticipate. This process has been in active use at Slack since the ...