book

Learning Chaos Engineering

by Russ Miles

July 2019

Beginner

175 pages

3h 55m

English

O'Reilly Media, Inc.

Read now

Unlock full access

AudienceWhat This Book Is AboutWhat This Book Is Not AboutAbout the SamplesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Chaos Engineering DefinedChaos Engineering Addresses the Whole Sociotechnical SystemLocations of Dark DebtThe Process of Chaos EngineeringThe Practices of Chaos EngineeringSandbox/Staging or Production?Chaos Engineering and ObservabilityIs There a “Chaos Engineer”?Summary
Start with Experiments?Gathering HypothesesIncident AnalysisSketching Your SystemCapturing “What Could Possibly Go Wrong?”Introducing Likelihood and ImpactBuilding a Likelihood-Impact MapAdding What You Care AboutCreating Your Hypothesis BacklogSummary
What Is a Game Day?Planning Your Game DayPick a HypothesisPick a Style of Game DayDecide Who Participates and Who ObservesDecide WhereDecide When and For How LongDescribe Your Game Day ExperimentGet Approval!Running the Game DayConsider a “Safety Monitor”Summary
Installing Python 3Installing the Chaos Toolkit CLISummary
Setting Up the Sample Target SystemA Quick Tour of the Sample SystemExploring and Discovering Evidence of WeaknessesRunning Your ExperimentUnder the Skin of chaos runSteady-State Deviation Might Indicate “Opportunity for Improvement”Improving the SystemValidating the ImprovementSummary
The Target SystemThe Platform: A Three-Node Kubernetes ClusterThe Application: A Single Service, Replicated Three TimesThe People: Application Team and Cluster AdministratorsHunting for a WeaknessNaming Your ExperimentDefining Your Steady-State HypothesisInjecting Turbulent Conditions in an Experiment’s MethodUsing the Kubernetes Driver from Your MethodBeing a Good Citizen with RollbacksBringing It All Together and Running Your ExperimentOvercoming a Weakness: Applying a Disruption BudgetSummary
Sharing Experiment DefinitionsMoving Values into ConfigurationSpecifying Configuration Properties as Environment VariablesExternalizing SecretsScoping SecretsSpecifying a Contribution ModelCreating and Sharing Human-Readable Chaos Experiment ReportsCreating a Single-Experiment Execution ReportCreating and Sharing a Multiple Experiment Execution ReportSummary

Creating Your Own Custom Driver with No Custom CodeImplementing Probes and Actions with HTTP CallsImplementing Probes and Actions Through Process CallsCreating Your Own Custom Chaos Driver in PythonCreating a New Python Module for Your Chaos Toolkit Extension ProjectAdding the ProbeSummary
Experiment “Controls”Enabling ControlsEnabling a Control Inline in an ExperimentEnabling a Control GloballySummary
Adding Logging to Your Chaos ExperimentsCentralized Chaos Logging in ActionTracing Your Chaos ExperimentsIntroducing OpenTracingApplying the OpenTracing ControlSummary
Creating a New Chaos Toolkit Extension for Your ControlsAdding Your (Very) Simple Human Interaction ControlSkipping or Executing an Experiment’s ActivitySummary
What Is Continuous Chaos?Scheduling Continuous Chaos Using cronCreating a Script to Execute Your Chaos TestsAdding Your Chaos Tests Script to cronScheduling Continuous Chaos with JenkinsGrabbing a Copy of JenkinsAdding Your Chaos Tests to a Jenkins BuildScheduling Your Chaos Tests in Jenkins with Build TriggersSummary
The Default Chaos CommandsDiscovering What’s Possible with the chaos discover CommandAuthoring a New Experiment with the chaos init CommandChecking Your Experiment with the chaos validate CommandExtending the Chaos Commands with Plug-ins

Content preview from Learning Chaos Engineering

Chapter 2. Building a Hypothesis Backlog

The aim of chaos engineering is to help you build trust and confidence in how your whole sociotechnical system will perform during turbulent conditions. One way to get rolling might be to jump straight in and start introducing those turbulent conditions—break networks, introduce latency, and see what collapses¹

But wait! Before you unleash Chaos Monkey unchained in production, remember that you want to learn from your chaos, which means that you need much more than just to have caused chaos. You need to follow the scientific method.

How NOT to Do Chaos Engineering

Chaos Monkey unchained might sound fun, interesting, or plain frightening, depending on your experience. In this case, it happens to be a real case study of exactly how not to do chaos engineering…

I received a call from a client that started like this:

Client: “We’re considering doing chaos engineering and might need some of your help…”

Me: “OK, what can we help you with?”

Client: “We’ve just run Chaos Monkey…”

Me: “OK…”

Client: “… in production.”

Me: “Oh… so you’ve been doing this for a while?”

Client: “It broke everything.”

Me: “Ah.”

Client: “We knew it would…”

Me: “OK, I think you should stop—you’re not actually being chaos engineers…”

Chaos engineering is not just about breaking things, and if that’s what you aim for, then all that will result is pain for you and your users! This is exactly why we say that chaos engineering is a controlled, disciplined scientific approach. ...