book

Chaos Engineering

by Mikolaj Pawlikowski

March 2021

Intermediate to advanced

424 pages

12h 5m

English

Manning Publications

Read now

Unlock full access

forewordforewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 What is chaos engineering?1.2 Motivations for chaos engineering1.2.1 Estimating risk and cost, and setting SLIs, SLOs, and SLAs1.2.2 Testing a system as a whole1.2.3 Finding emergent properties1.3 Four steps to chaos engineering1.3.1 Ensure observability1.3.2 Define a steady state1.3.3 Form a hypothesis1.3.4 Run the experiment and prove (or refute) your hypothesis1.4 What chaos engineering is not1.5 A taste of chaos engineering1.5.1 FizzBuzz as a service1.5.2 A long, dark night1.5.3 Postmortem1.5.4 Chaos engineering in a nutshellSummary
2.1 Setup: Working with the code in this book2.2 Scenario2.3 Linux forensics 1012.3.1 Exit codes2.3.2 Killing processes2.3.3 Out-Of-Memory Killer2.4 The first chaos experiment2.4.1 Ensure observability2.4.2 Define a steady state2.4.3 Form a hypothesis2.4.4 Run the experiment2.5 Blast radius2.6 Digging deeper2.6.1 Saving the worldSummary

3.1 The app is slow3.2 The USE method3.3 Resources3.3.1 System overview3.3.2 Block I/O3.3.3 Networking3.3.4 RAM3.3.5 CPU3.3.6 OS3.4 Application3.4.1 cProfile3.4.2 BCC and Python3.5 Automation: Using time series3.5.1 Prometheus and Grafana3.6 Further readingSummary
4.1 We’re doing WordPress4.2 Weak links4.2.1 Experiment 1: Slow disks4.2.2 Experiment 2: Slow connection4.3 Testing in productionSummary
5.1 My (Dockerized) app is slow!5.1.1 Architecture5.2 A brief history of Docker5.2.1 Emulation, simulation, and virtualization5.2.2 Virtual machines and containers5.3 Linux containers and Docker5.4 Peeking under Docker’s hood5.4.1 Uprooting processes with chroot5.4.2 Implementing a simple container(-ish) part 1: Using chroot5.4.3 Experiment 1: Can one container prevent another one from writing to disk?5.4.4 Isolating processes with Linux namespaces5.4.5 Docker and namespaces5.5 Experiment 2: Killing processes in a different PID namespace5.5.1 Implementing a simple container(-ish) part 2: Namespaces5.5.2 Limiting resource use of a process with cgroups5.6 Experiment 3: Using all the CPU you can find!5.7 Experiment 4: Using too much RAM5.7.1 Implementing a simple container(-ish) part 3: Cgroups5.8 Docker and networking5.8.1 Capabilities and seccomp5.9 Docker demystified5.10 Fixing my (Dockerized) app that’s being slow5.10.1 Booting up Meower5.10.2 Why is the app slow?5.11 Experiment 5: Network slowness for containers with Pumba5.11.1 Pumba: Docker chaos engineering tool5.11.2 Chaos experiment implementation5.12 Other parts of the puzzle5.12.1 Docker daemon restarts5.12.2 Storage for image layers5.12.3 Advanced networking5.12.4 SecuritySummary
6.1 Scenario: Congratulations on your promotion!6.1.1 System X: If everyone is using it, but no one maintains it, is it abandonware?6.2 A brief refresher on syscalls6.2.1 Finding out about syscalls6.2.2 Using the standard C library and glibc6.3 How to observe a process’s syscalls6.3.1 strace and sleep6.3.2 strace and System X6.3.3 strace’s problem: Overhead6.3.4 BPF6.3.5 Other options6.4 Blocking syscalls for fun and profit part 1: strace6.4.1 Experiment 1: Breaking the close syscall6.4.2 Experiment 2: Breaking the write syscall6.5 Blocking syscalls for fun and profit part 2: Seccomp6.5.1 Seccomp the easy way with Docker6.5.2 Seccomp the hard way with libseccompSummary
7.1 Scenario7.1.1 Introducing FizzBuzzEnterpriseEdition7.1.2 Looking around FizzBuzzEnterpriseEdition7.2 Chaos engineering and Java7.2.1 Experiment idea7.2.2 Experiment plan7.2.3 Brief introduction to JVM bytecode7.2.4 Experiment implementation7.3 Existing tools7.3.1 Byteman7.3.2 Byte-Monkey7.3.3 Chaos Monkey for Spring Boot7.4 Further readingSummary
8.1 Scenario8.1.1 Implementation details: Before chaos8.2 Experiment 1: Redis latency8.2.1 Experiment 1 plan8.2.2 Experiment 1 steady state8.2.3 Experiment 1 implementation8.2.4 Experiment 1 execution8.2.5 Experiment 1 discussion8.3 Experiment 2: Failing requests8.3.1 Experiment 2 plan8.3.2 Experiment 2 implementation8.3.3 Experiment 2 execution8.4 Application vs. infrastructureSummary
9.1 Scenario9.1.1 Pgweb9.1.2 Pgweb implementation details9.2 Experiment 1: Adding latency9.2.1 Experiment 1 plan9.2.2 Experiment 1 steady state9.2.3 Experiment 1 implementation9.2.4 Experiment 1 run9.3 Experiment 2: Adding failure9.3.1 Experiment 2 implementation9.3.2 Experiment 2 run9.4 Other good-to-know topics9.4.1 Fetch API9.4.2 Throttling9.4.3 Tooling: Greasemonkey and TampermonkeySummary
10.1 Porting things onto Kubernetes10.1.1 High-Profile Project documentation10.1.2 What’s Goldpinger?10.2 What’s Kubernetes (in 7 minutes)?10.2.1 A very brief history of Kubernetes10.2.2 What can Kubernetes do for you?10.3 Setting up a Kubernetes cluster10.3.1 Using Minikube10.3.2 Starting a cluster10.4 Testing out software running on Kubernetes10.4.1 Running the ICANT Project10.4.2 Experiment 1: Kill 50% of pods10.4.3 Party trick: Kill pods in style10.4.4 Experiment 2: Introduce network slownessSummary
11.1 Automating chaos with PowerfulSeal11.1.1 What’s PowerfulSeal?11.1.2 PowerfulSeal installation11.1.3 Experiment 1b: Killing 50% of pods11.1.4 Experiment 2b: Introducing network slowness11.2 Ongoing testing and service-level objectives11.2.1 Experiment 3: Verifying pods are ready within (n) seconds of being created11.3 Cloud layer11.3.1 Cloud provider APIs, availability zones11.3.2 Experiment 4: Taking VMs downSummary
12.1 Anatomy of a Kubernetes cluster and how to break it12.1.1 Control plane12.1.2 Kubelet and pause container12.1.3 Kubernetes, Docker, and container runtimes12.1.4 Kubernetes networking12.2 Summary of key componentsSummary
13.1 Chaos engineering mindset13.1.1 Failure is not a maybe: It will happen13.1.2 Failing early vs. failing late13.2 Getting buy-in13.2.1 Management13.2.2 Team members13.2.3 Game days13.3 Teams as distributed systems13.3.1 Finding knowledge single points of failure: Staycation13.3.2 Misinformation and trust within the team: Liar, Liar13.3.3 Bottlenecks in the team: Life in the Slow Lane13.3.4 Testing your processes: Inside JobSummary13.4 Where to go from here?
A.1 PrerequisitesA.2 Installing the Linux toolsA.2.1 PumbaA.2.2 Python 3.7 with DTrace optionA.2.3 PgwebA.2.4 Pip dependenciesA.2.5 Example data to look at for pgwebA.3 Configuring WordPressA.4 Checking out the source code for this bookA.5 Installing Minikube (Kubernetes)A.5.1 LinuxA.5.2 macOSA.5.3 Windows
Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12
C.1 CloudC.2 Chaos engineering tools comparisonC.3 WindowsC.4 RuntimesC.5 Node.jsC.6 Architecture problemsC.7 The four steps to a chaos experimentC.8 You should have included <tool X>!C.9 Real-world failure examples!C.10 “Chaos engineering” is a terrible name!C.11 Wrap!
D.1 SRE (’ShRoomEee) burgerD.1.1 IngredientsD.1.2 Hidden dependenciesD.1.3 Making the pattyD.1.4 Assembling the finished productD.2 Chaos pizzaD.2.1 IngredientsD.2.2 Preparation

Overview

Auto engineers test the safety of a car by intentionally crashing it and carefully observing the results. Chaos engineering applies the same principles to software systems. In Chaos Engineering: Site reliability through controlled disruption, you’ll learn to run your applications and infrastructure through a series of tests that simulate real-life failures. You'll maximize the benefits of chaos engineering by learning to think like a chaos engineer, and how to design the proper experiments to ensure the reliability of your software. With examples that cover a whole spectrum of software, you'll be ready to run an intensive testing regime on anything from a simple WordPress site to a massive distributed system running on Kubernetes.

About the Technology
Can your network survive a devastating failure? Could an accident bring your day-to-day operations to a halt? Chaos engineering simulates infrastructure outages, component crashes, and other calamities to show how systems and staff respond. Testing systems in distress is the best way to ensure their future resilience, which is especially important for complex, large-scale applications with little room for downtime.

About the Book
Chaos Engineering teaches you to design and execute controlled experiments that uncover hidden problems. Learn to inject system-shaking failures that disrupt system calls, networking, APIs, and Kubernetes-based microservices infrastructures. To help you practice, the book includes a downloadable Linux VM image with a suite of preconfigured tools so you can experiment quickly—without risk.

What's Inside

Inject failure into processes, applications, and virtual machines
Test software running on Kubernetes
Work with both open source and legacy software
Simulate database connection latency
Test and improve your team’s failure response

About the Reader
Assumes Linux servers. Basic scripting skills required.

About the Author
Mikolaj Pawlikowski is a recognized authority on chaos engineering. He is the creator of the Kubernetes chaos engineering tool PowerfulSeal, and the networking visibility tool Goldpinger.

Quotes
The topics covered in this book are easy to follow and detailed. It provides a number of hands-on exercises to help the reader master chaos engineering.
- Kelum Prabath Senanayake, Echoworx

The book we needed to improve our system’s reliability and resilience.
- Hugo Cruz, People Driven Technology

An important topic if you want to find hidden problems in your large system. This book gives a really good foundation.
- Yuri Kushch, Amazon

One of the best books about in-depth infrastructure, troubleshooting complex systems, and chaos engineering that I’ve ever read.
- Lev Andelman, Terasky Cloud & Devops