book

Chaos Engineering

by Mikolaj Pawlikowski

March 2021

Intermediate to advanced

424 pages

12h 5m

English

Manning Publications

Read now

Unlock full access

forewordforewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 What is chaos engineering?1.2 Motivations for chaos engineering1.2.1 Estimating risk and cost, and setting SLIs, SLOs, and SLAs1.2.2 Testing a system as a whole1.2.3 Finding emergent properties1.3 Four steps to chaos engineering1.3.1 Ensure observability1.3.2 Define a steady state1.3.3 Form a hypothesis1.3.4 Run the experiment and prove (or refute) your hypothesis1.4 What chaos engineering is not1.5 A taste of chaos engineering1.5.1 FizzBuzz as a service1.5.2 A long, dark night1.5.3 Postmortem1.5.4 Chaos engineering in a nutshellSummary
2.1 Setup: Working with the code in this book2.2 Scenario2.3 Linux forensics 1012.3.1 Exit codes2.3.2 Killing processes2.3.3 Out-Of-Memory Killer2.4 The first chaos experiment2.4.1 Ensure observability2.4.2 Define a steady state2.4.3 Form a hypothesis2.4.4 Run the experiment2.5 Blast radius2.6 Digging deeper2.6.1 Saving the worldSummary

3.1 The app is slow3.2 The USE method3.3 Resources3.3.1 System overview3.3.2 Block I/O3.3.3 Networking3.3.4 RAM3.3.5 CPU3.3.6 OS3.4 Application3.4.1 cProfile3.4.2 BCC and Python3.5 Automation: Using time series3.5.1 Prometheus and Grafana3.6 Further readingSummary
4.1 We’re doing WordPress4.2 Weak links4.2.1 Experiment 1: Slow disks4.2.2 Experiment 2: Slow connection4.3 Testing in productionSummary
5.1 My (Dockerized) app is slow!5.1.1 Architecture5.2 A brief history of Docker5.2.1 Emulation, simulation, and virtualization5.2.2 Virtual machines and containers5.3 Linux containers and Docker5.4 Peeking under Docker’s hood5.4.1 Uprooting processes with chroot5.4.2 Implementing a simple container(-ish) part 1: Using chroot5.4.3 Experiment 1: Can one container prevent another one from writing to disk?5.4.4 Isolating processes with Linux namespaces5.4.5 Docker and namespaces5.5 Experiment 2: Killing processes in a different PID namespace5.5.1 Implementing a simple container(-ish) part 2: Namespaces5.5.2 Limiting resource use of a process with cgroups5.6 Experiment 3: Using all the CPU you can find!5.7 Experiment 4: Using too much RAM5.7.1 Implementing a simple container(-ish) part 3: Cgroups5.8 Docker and networking5.8.1 Capabilities and seccomp5.9 Docker demystified5.10 Fixing my (Dockerized) app that’s being slow5.10.1 Booting up Meower5.10.2 Why is the app slow?5.11 Experiment 5: Network slowness for containers with Pumba5.11.1 Pumba: Docker chaos engineering tool5.11.2 Chaos experiment implementation5.12 Other parts of the puzzle5.12.1 Docker daemon restarts5.12.2 Storage for image layers5.12.3 Advanced networking5.12.4 SecuritySummary
6.1 Scenario: Congratulations on your promotion!6.1.1 System X: If everyone is using it, but no one maintains it, is it abandonware?6.2 A brief refresher on syscalls6.2.1 Finding out about syscalls6.2.2 Using the standard C library and glibc6.3 How to observe a process’s syscalls6.3.1 strace and sleep6.3.2 strace and System X6.3.3 strace’s problem: Overhead6.3.4 BPF6.3.5 Other options6.4 Blocking syscalls for fun and profit part 1: strace6.4.1 Experiment 1: Breaking the close syscall6.4.2 Experiment 2: Breaking the write syscall6.5 Blocking syscalls for fun and profit part 2: Seccomp6.5.1 Seccomp the easy way with Docker6.5.2 Seccomp the hard way with libseccompSummary
7.1 Scenario7.1.1 Introducing FizzBuzzEnterpriseEdition7.1.2 Looking around FizzBuzzEnterpriseEdition7.2 Chaos engineering and Java7.2.1 Experiment idea7.2.2 Experiment plan7.2.3 Brief introduction to JVM bytecode7.2.4 Experiment implementation7.3 Existing tools7.3.1 Byteman7.3.2 Byte-Monkey7.3.3 Chaos Monkey for Spring Boot7.4 Further readingSummary
8.1 Scenario8.1.1 Implementation details: Before chaos8.2 Experiment 1: Redis latency8.2.1 Experiment 1 plan8.2.2 Experiment 1 steady state8.2.3 Experiment 1 implementation8.2.4 Experiment 1 execution8.2.5 Experiment 1 discussion8.3 Experiment 2: Failing requests8.3.1 Experiment 2 plan8.3.2 Experiment 2 implementation8.3.3 Experiment 2 execution8.4 Application vs. infrastructureSummary
9.1 Scenario9.1.1 Pgweb9.1.2 Pgweb implementation details9.2 Experiment 1: Adding latency9.2.1 Experiment 1 plan9.2.2 Experiment 1 steady state9.2.3 Experiment 1 implementation9.2.4 Experiment 1 run9.3 Experiment 2: Adding failure9.3.1 Experiment 2 implementation9.3.2 Experiment 2 run9.4 Other good-to-know topics9.4.1 Fetch API9.4.2 Throttling9.4.3 Tooling: Greasemonkey and TampermonkeySummary
10.1 Porting things onto Kubernetes10.1.1 High-Profile Project documentation10.1.2 What’s Goldpinger?10.2 What’s Kubernetes (in 7 minutes)?10.2.1 A very brief history of Kubernetes10.2.2 What can Kubernetes do for you?10.3 Setting up a Kubernetes cluster10.3.1 Using Minikube10.3.2 Starting a cluster10.4 Testing out software running on Kubernetes10.4.1 Running the ICANT Project10.4.2 Experiment 1: Kill 50% of pods10.4.3 Party trick: Kill pods in style10.4.4 Experiment 2: Introduce network slownessSummary
11.1 Automating chaos with PowerfulSeal11.1.1 What’s PowerfulSeal?11.1.2 PowerfulSeal installation11.1.3 Experiment 1b: Killing 50% of pods11.1.4 Experiment 2b: Introducing network slowness11.2 Ongoing testing and service-level objectives11.2.1 Experiment 3: Verifying pods are ready within (n) seconds of being created11.3 Cloud layer11.3.1 Cloud provider APIs, availability zones11.3.2 Experiment 4: Taking VMs downSummary
12.1 Anatomy of a Kubernetes cluster and how to break it12.1.1 Control plane12.1.2 Kubelet and pause container12.1.3 Kubernetes, Docker, and container runtimes12.1.4 Kubernetes networking12.2 Summary of key componentsSummary
13.1 Chaos engineering mindset13.1.1 Failure is not a maybe: It will happen13.1.2 Failing early vs. failing late13.2 Getting buy-in13.2.1 Management13.2.2 Team members13.2.3 Game days13.3 Teams as distributed systems13.3.1 Finding knowledge single points of failure: Staycation13.3.2 Misinformation and trust within the team: Liar, Liar13.3.3 Bottlenecks in the team: Life in the Slow Lane13.3.4 Testing your processes: Inside JobSummary13.4 Where to go from here?
A.1 PrerequisitesA.2 Installing the Linux toolsA.2.1 PumbaA.2.2 Python 3.7 with DTrace optionA.2.3 PgwebA.2.4 Pip dependenciesA.2.5 Example data to look at for pgwebA.3 Configuring WordPressA.4 Checking out the source code for this bookA.5 Installing Minikube (Kubernetes)A.5.1 LinuxA.5.2 macOSA.5.3 Windows
Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12
C.1 CloudC.2 Chaos engineering tools comparisonC.3 WindowsC.4 RuntimesC.5 Node.jsC.6 Architecture problemsC.7 The four steps to a chaos experimentC.8 You should have included <tool X>!C.9 Real-world failure examples!C.10 “Chaos engineering” is a terrible name!C.11 Wrap!
D.1 SRE (’ShRoomEee) burgerD.1.1 IngredientsD.1.2 Hidden dependenciesD.1.3 Making the pattyD.1.4 Assembling the finished productD.2 Chaos pizzaD.2.1 IngredientsD.2.2 Preparation

Content preview from Chaos Engineering

6 Who you gonna call? Syscall-busters!

This chapter covers

Observing syscalls of a running process by using strace and BPF
Working with black-box software
Designing chaos experiments at the syscall level
Blocking syscalls by using strace and seccomp

It’s time to take a deep dive—all the way to the OS—to learn how to do chaos engineering at the syscall level. I want to show you that even in a simple system, like a single process running on a host, you can create plenty of value by applying chaos engineering and learning just how resilient that system is to failure. And, oh, it’s good fun too!

This chapter starts with a brief refresher on syscalls. You’ll then see how to do the following: