book

Chaos Engineering

Name: Chaos Engineering
Author: Mikolaj Pawlikowski
ISBN: 9781617297755

by Mikolaj Pawlikowski

March 2021

Intermediate to advanced

424 pages

12h 5m

English

Manning Publications

Read now

Unlock full access

inside front cover
Chaos Engineering
Copyright
dedication
brief contents
contents
front matter
forewordforewordprefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: a roadmapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1 Into the world of chaos engineering
1.1 What is chaos engineering?1.2 Motivations for chaos engineering1.2.1 Estimating risk and cost, and setting SLIs, SLOs, and SLAs1.2.2 Testing a system as a whole1.2.3 Finding emergent properties1.3 Four steps to chaos engineering1.3.1 Ensure observability1.3.2 Define a steady state1.3.3 Form a hypothesis1.3.4 Run the experiment and prove (or refute) your hypothesis1.4 What chaos engineering is not1.5 A taste of chaos engineering1.5.1 FizzBuzz as a service1.5.2 A long, dark night1.5.3 Postmortem1.5.4 Chaos engineering in a nutshellSummary
Part 1. Chaos engineering fundamentals
2 First cup of chaos and blast radius
2.1 Setup: Working with the code in this book2.2 Scenario2.3 Linux forensics 1012.3.1 Exit codes2.3.2 Killing processes2.3.3 Out-Of-Memory Killer2.4 The first chaos experiment2.4.1 Ensure observability2.4.2 Define a steady state2.4.3 Form a hypothesis2.4.4 Run the experiment2.5 Blast radius2.6 Digging deeper2.6.1 Saving the worldSummary

3 Observability
3.1 The app is slow3.2 The USE method3.3 Resources3.3.1 System overview3.3.2 Block I/O3.3.3 Networking3.3.4 RAM3.3.5 CPU3.3.6 OS3.4 Application3.4.1 cProfile3.4.2 BCC and Python3.5 Automation: Using time series3.5.1 Prometheus and Grafana3.6 Further readingSummary
4 Database trouble and testing in production
4.1 We’re doing WordPress4.2 Weak links4.2.1 Experiment 1: Slow disks4.2.2 Experiment 2: Slow connection4.3 Testing in productionSummary
Part 2. Chaos engineering in action
5 Poking Docker
5.1 My (Dockerized) app is slow!5.1.1 Architecture5.2 A brief history of Docker5.2.1 Emulation, simulation, and virtualization5.2.2 Virtual machines and containers5.3 Linux containers and Docker5.4 Peeking under Docker’s hood5.4.1 Uprooting processes with chroot5.4.2 Implementing a simple container(-ish) part 1: Using chroot5.4.3 Experiment 1: Can one container prevent another one from writing to disk?5.4.4 Isolating processes with Linux namespaces5.4.5 Docker and namespaces5.5 Experiment 2: Killing processes in a different PID namespace5.5.1 Implementing a simple container(-ish) part 2: Namespaces5.5.2 Limiting resource use of a process with cgroups5.6 Experiment 3: Using all the CPU you can find!5.7 Experiment 4: Using too much RAM5.7.1 Implementing a simple container(-ish) part 3: Cgroups5.8 Docker and networking5.8.1 Capabilities and seccomp5.9 Docker demystified5.10 Fixing my (Dockerized) app that’s being slow5.10.1 Booting up Meower5.10.2 Why is the app slow?5.11 Experiment 5: Network slowness for containers with Pumba5.11.1 Pumba: Docker chaos engineering tool5.11.2 Chaos experiment implementation5.12 Other parts of the puzzle5.12.1 Docker daemon restarts5.12.2 Storage for image layers5.12.3 Advanced networking5.12.4 SecuritySummary
6 Who you gonna call? Syscall-busters!
6.1 Scenario: Congratulations on your promotion!6.1.1 System X: If everyone is using it, but no one maintains it, is it abandonware?6.2 A brief refresher on syscalls6.2.1 Finding out about syscalls6.2.2 Using the standard C library and glibc6.3 How to observe a process’s syscalls6.3.1 strace and sleep6.3.2 strace and System X6.3.3 strace’s problem: Overhead6.3.4 BPF6.3.5 Other options6.4 Blocking syscalls for fun and profit part 1: strace6.4.1 Experiment 1: Breaking the close syscall6.4.2 Experiment 2: Breaking the write syscall6.5 Blocking syscalls for fun and profit part 2: Seccomp6.5.1 Seccomp the easy way with Docker6.5.2 Seccomp the hard way with libseccompSummary
7 Injecting failure into the JVM
7.1 Scenario7.1.1 Introducing FizzBuzzEnterpriseEdition7.1.2 Looking around FizzBuzzEnterpriseEdition7.2 Chaos engineering and Java7.2.1 Experiment idea7.2.2 Experiment plan7.2.3 Brief introduction to JVM bytecode7.2.4 Experiment implementation7.3 Existing tools7.3.1 Byteman7.3.2 Byte-Monkey7.3.3 Chaos Monkey for Spring Boot7.4 Further readingSummary
8 Application-level fault injection
8.1 Scenario8.1.1 Implementation details: Before chaos8.2 Experiment 1: Redis latency8.2.1 Experiment 1 plan8.2.2 Experiment 1 steady state8.2.3 Experiment 1 implementation8.2.4 Experiment 1 execution8.2.5 Experiment 1 discussion8.3 Experiment 2: Failing requests8.3.1 Experiment 2 plan8.3.2 Experiment 2 implementation8.3.3 Experiment 2 execution8.4 Application vs. infrastructureSummary
9 There’s a monkey in my browser!
9.1 Scenario9.1.1 Pgweb9.1.2 Pgweb implementation details9.2 Experiment 1: Adding latency9.2.1 Experiment 1 plan9.2.2 Experiment 1 steady state9.2.3 Experiment 1 implementation9.2.4 Experiment 1 run9.3 Experiment 2: Adding failure9.3.1 Experiment 2 implementation9.3.2 Experiment 2 run9.4 Other good-to-know topics9.4.1 Fetch API9.4.2 Throttling9.4.3 Tooling: Greasemonkey and TampermonkeySummary
Part 3. Chaos engineering in Kubernetes
10 Chaos in Kubernetes
10.1 Porting things onto Kubernetes10.1.1 High-Profile Project documentation10.1.2 What’s Goldpinger?10.2 What’s Kubernetes (in 7 minutes)?10.2.1 A very brief history of Kubernetes10.2.2 What can Kubernetes do for you?10.3 Setting up a Kubernetes cluster10.3.1 Using Minikube10.3.2 Starting a cluster10.4 Testing out software running on Kubernetes10.4.1 Running the ICANT Project10.4.2 Experiment 1: Kill 50% of pods10.4.3 Party trick: Kill pods in style10.4.4 Experiment 2: Introduce network slownessSummary
11 Automating Kubernetes experiments
11.1 Automating chaos with PowerfulSeal11.1.1 What’s PowerfulSeal?11.1.2 PowerfulSeal installation11.1.3 Experiment 1b: Killing 50% of pods11.1.4 Experiment 2b: Introducing network slowness11.2 Ongoing testing and service-level objectives11.2.1 Experiment 3: Verifying pods are ready within (n) seconds of being created11.3 Cloud layer11.3.1 Cloud provider APIs, availability zones11.3.2 Experiment 4: Taking VMs downSummary
12 Under the hood of Kubernetes
12.1 Anatomy of a Kubernetes cluster and how to break it12.1.1 Control plane12.1.2 Kubelet and pause container12.1.3 Kubernetes, Docker, and container runtimes12.1.4 Kubernetes networking12.2 Summary of key componentsSummary
13 Chaos engineering (for) people
13.1 Chaos engineering mindset13.1.1 Failure is not a maybe: It will happen13.1.2 Failing early vs. failing late13.2 Getting buy-in13.2.1 Management13.2.2 Team members13.2.3 Game days13.3 Teams as distributed systems13.3.1 Finding knowledge single points of failure: Staycation13.3.2 Misinformation and trust within the team: Liar, Liar13.3.3 Bottlenecks in the team: Life in the Slow Lane13.3.4 Testing your processes: Inside JobSummary13.4 Where to go from here?
Appendix A. Installing chaos engineering tools
A.1 PrerequisitesA.2 Installing the Linux toolsA.2.1 PumbaA.2.2 Python 3.7 with DTrace optionA.2.3 PgwebA.2.4 Pip dependenciesA.2.5 Example data to look at for pgwebA.3 Configuring WordPressA.4 Checking out the source code for this bookA.5 Installing Minikube (Kubernetes)A.5.1 LinuxA.5.2 macOSA.5.3 Windows
Appendix B. Answers to the pop quizzes
Chapter 2Chapter 3Chapter 4Chapter 5Chapter 6Chapter 7Chapter 8Chapter 9Chapter 10Chapter 11Chapter 12
Appendix C. Director’s cut (aka the bloopers)
C.1 CloudC.2 Chaos engineering tools comparisonC.3 WindowsC.4 RuntimesC.5 Node.jsC.6 Architecture problemsC.7 The four steps to a chaos experimentC.8 You should have included <tool X>!C.9 Real-world failure examples!C.10 “Chaos engineering” is a terrible name!C.11 Wrap!
Appendix D. Chaos-engineering recipes
D.1 SRE (’ShRoomEee) burgerD.1.1 IngredientsD.1.2 Hidden dependenciesD.1.3 Making the pattyD.1.4 Assembling the finished productD.2 Chaos pizzaD.2.1 IngredientsD.2.2 Preparation
index
inside back cover

Content preview from Chaos Engineering

Appendix D. Chaos-engineering recipes

Writing books makes you hungry. Well, at the very least, it makes me hungry. If my manuscript was in a paper form, it would smell of all of the following things.

I’m not a doctor, dietician or even a cook. What you find below is a result of letting a software engineer loose in the kitchen. Oh, and THE RECIPES ARE PROVIDED “AS IS,” WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617297755Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Chaos Engineering

by Mikolaj Pawlikowski

Appendix D. Chaos-engineering recipes

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.