Chaos Engineering

Table of contents

1. Preface
   1. Conventions Used in This Book
   2. O’Reilly Online Learning
   3. How to Contact Us
   4. Acknowledgments
2. Introduction: Birth of Chaos
   1. Management Principles as Code
   2. Chaos Monkey Is Born
   3. Going Big
   4. Formalizing the Discipline
   5. Community Is Born
   6. Fast Evolution
3. I. Setting the Stage
4. 1. Encountering Complex Systems
   1. Contemplating Complexity
   2. Encountering Complexity
      1. Example 1: Mismatch Between Business Logic and Application Logic
      2. Example 2: Customer-Induced Retry Storm
      3. Example 3: Holiday Code Freeze
   3. Confronting Complexity
      1. Accidental Complexity
      2. Essential Complexity
   4. Embracing Complexity
5. 2. Navigating Complex Systems
   1. Dynamic Safety Model
      1. Economics
      2. Workload
      3. Safety
   2. Economic Pillars of Complexity
      1. State
      2. Relationships
      3. Environment
      4. Reversibility
      5. Economic Pillars of Complexity Applied to Software
   3. The Systemic Perspective
6. 3. Overview of Principles
   1. What Chaos Engineering Is
      1. Experimentation Versus Testing
      2. Verification Versus Validation
   2. What Chaos Engineering Is Not
      1. Breaking Stuff
      2. Antifragility
   3. Advanced Principles
      1. Build a Hypothesis Around Steady-State Behavior
      2. Vary Real-World Events
      3. Run Experiments in Production
      4. Automate Experiments to Run Continuously
      5. Minimize Blast Radius
   4. The Future of “The Principles”
7. II. Principles in Action
8. 4. Slack’s Disasterpiece Theater
   1. Retrofitting Chaos
      1. Design Patterns Common in Older Systems
      2. Design Patterns Common in Newer Systems
      3. Getting to Basic Fault Tolerance
   2. Disasterpiece Theater
      1. Goals
      2. Anti-Goals
   3. The Process
      1. Preparation
      2. The Exercise
      3. Debriefing
   4. How the Process Has Evolved
   5. Getting Management Buy-In
   6. Results
      1. Avoid Cache Inconsistency
      2. Try, Try Again (for Safety)
      3. Impossibility Result
   7. Conclusion
9. 5. Google DiRT: Disaster Recovery Testing
   1. Life of a DiRT Test
      1. The Rules of Engagement
      2. What to Test
      3. How to Test
      4. Gathering Results
   2. Scope of Tests at Google
   3. Conclusion
10. 6. Microsoft Variation and Prioritization of Experiments
    1. Why Is Everything So Complicated?
       1. An Example of Unexpected Complications
       2. A Simple System Is the Tip of the Iceberg
    2. Categories of Experiment Outcomes
       1. Known Events/Unexpected Consequences
       2. Unknown Events/Unexpected Consequences
    3. Prioritization of Failures
       1. Explore Dependencies
    4. Degree of Variation
       1. Varying Failures
       2. Combining Variation and Prioritization
       3. Expanding Variation to Dependencies
    5. Deploying Experiments at Scale
    6. Conclusion
11. 7. LinkedIn Being Mindful of Members
    1. Learning from Disaster
    2. Granularly Targeting Experiments
    3. Experimenting at Scale, Safely
    4. In Practice: LinkedOut
       1. Failure Modes
       2. Using LiX to Target Experiments
       3. Browser Extension for Rapid Experimentation
       4. Automated Experimentation
    5. Conclusion
12. 8. Capital One Adoption and Evolution of Chaos Engineering
    1. A Capital One Case Study
       1. Blind Resiliency Testing
       2. Transition to Chaos Engineering
       3. Chaos Experiments in CI/CD
    2. Things to Watch Out for While Designing the Experiment
    3. Tooling
    4. Team Structure
    5. Evangelism
    6. Conclusion
13. III. Human Factors
14. 9. Creating Foresight
    1. Chaos Engineering and Resilience
    2. Steps of the Chaos Engineering Cycle
       1. Designing the Experiment
    3. Tool Support for Chaos Experiment Design
    4. Effectively Partnering Internally
       1. Understand Operating Procedures
       2. Discuss Scope
       3. Hypothesize
    5. Conclusion
15. 10. Humanistic Chaos
    1. Humans in the System
       1. Putting the “Socio” in Sociotechnical Systems
       2. Organizations Are a System of Systems
    2. Engineering Adaptive Capacity
       1. Spotting Weak Signals
       2. Failure and Success, Two Sides of the Same Coin
    3. Putting the Principles into Practice
       1. Build a Hypothesis
       2. Vary Real-World Events
       3. Minimize the Blast Radius
       4. Case Study 1: Gaming Your Game Days
       5. Communication: The Network Latency of Any Organization
       6. Case Study 2: Connecting the Dots
       7. Leadership Is an Emergent Property of the System
       8. Case Study 3: Changing a Basic Assumption
       9. Safely Organizing the Chaos
       10. All You Need Is Altitude and a Direction
       11. Close the Loops
       12. If You’re Not Failing, You’re Not Learning
16. 11. People in the Loop
    1. The Why, How, and When of Experiments
       1. The Why
       2. The How
       3. The When
       4. Functional Allocation, or Humans-Are-Better-At/Machines-Are-Better-At
       5. The Substitution Myth
    2. Conclusion
17. 12. The Experiment Selection Problem (and a Solution)
    1. Choosing Experiments
       1. Random Search
       2. The Age of the Experts
    2. Observability: The Opportunity
       1. Observability for Intuition Engineering
    3. Conclusion
18. IV. Business Factors
19. 13. ROI of Chaos Engineering
    1. Ephemeral Nature of Incident Reduction
    2. Kirkpatrick Model
       1. Level 1: Reaction
       2. Level 2: Learning
       3. Level 3: Transfer
       4. Level 4: Results
    3. Alternative ROI Example
    4. Collateral ROI
    5. Conclusion
20. 14. Open Minds, Open Science, and Open Chaos
    1. Collaborative Mindsets
    2. Open Science; Open Source
       1. Open Chaos Experiments
       2. Experiment Findings, Shareable Results
    3. Conclusion
21. 15. Chaos Maturity Model
    1. Adoption
       1. Who Bought into Chaos Engineering
       2. How Much of the Organization Participates in Chaos Engineering
       3. Prerequisites
       4. Obstacles to Adoption
       5. Sophistication
    2. Putting It All Together
22. V. Evolution
23. 16. Continuous Verification
    1. Where CV Comes From
    2. Types of CV Systems
    3. CV in the Wild: ChAP
       1. ChAP: Selecting Experiments
       2. ChAP: Running Experiments
       3. The Advanced Principles in ChAP
       4. ChAP as Continuous Verification
    4. CV Coming Soon to a System Near You
       1. Performance Testing
       2. Data Artifacts
       3. Correctness
24. 17. Let’s Get Cyber-Physical
    1. The Rise of Cyber-Physical Systems
    2. Functional Safety Meets Chaos Engineering
       1. FMEA and Chaos Engineering
    3. Software in Cyber-Physical Systems
    4. Chaos Engineering as a Step Beyond FMEA
    5. Probe Effect
       1. Addressing the Probe Effect
    6. Conclusion
25. 18. HOP Meets Chaos Engineering
    1. What Is Human and Organizational Performance (HOP)?
    2. Key Principles of HOP
       1. Principle 1: Error Is Normal
       2. Principle 2: Blame Fixes Nothing
       3. Principle 3: Context Drives Behavior
       4. Principle 4: Learning and Improving Is Vital
       5. Principle 5: Intentional Response Matters
    3. HOP Meets Chaos Engineering
       1. Chaos Engineering and HOP in Practice
    4. Conclusion
26. 19. Chaos Engineering on a Database
    1. Why Do We Need Chaos Engineering?
       1. Robustness and Stability
       2. A Real-World Example
    2. Applying Chaos Engineering
       1. Our Way of Embracing Chaos
       2. Fault Injection
       3. Fault Injection in Applications
       4. Fault Injection in CPU and Memory
       5. Fault Injection in the Network
       6. Fault Injection in the Filesystem
    3. Detecting Failures
    4. Automating Chaos
       1. Automated Experimentation Platform: Schrodinger
       2. Schrodinger Workflow
    5. Conclusion
27. 20. The Case for Security Chaos Engineering
    1. A Modern Approach to Security
       1. Human Factors and Failure
       2. Remove the Low-Hanging Fruit
       3. Feedback Loops
    2. Security Chaos Engineering and Current Methods
       1. Problems with Red Teaming
       2. Problems with Purple Teaming
       3. Benefits of Security Chaos Engineering
    3. Security Game Days
    4. Example Security Chaos Engineering Tool: ChaoSlingr
       1. The Story of ChaoSlingr
    5. Conclusion
    6. Contributors/Reviewers
28. 21. Conclusion
29. Index