book

Security Chaos Engineering

by Kelly Shortridge, Aaron Rinehart

March 2023

Intermediate to advanced

428 pages

14h 36m

English

O'Reilly Media, Inc.

Audiobook available

Read now

Unlock full access

Preface
Who Should Read This Book?Scope of This BookOutline of This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
1. Resilience in Software and Systems
What Is a Complex System?Variety Defines Complex SystemsComplex Systems Are AdaptiveThe Holistic Nature of Complex SystemsWhat Is Failure?Acute and Chronic Stressors in Complex SystemsSurprises in Complex SystemsWhat Is Resilience?Critical FunctionalitySafety Boundaries (Thresholds)Interactions Across Space-TimeFeedback Loops and Learning CultureFlexibility and Openness to ChangeResilience Is a VerbResilience: Myth Versus RealityMyth: Robustness = ResilienceMyth: We Can and Should Prevent FailureMyth: The Security of Each Component Adds Up to ResilienceMyth: Creating a “Security Culture” Fixes Human ErrorChapter Takeaways
2. Systems-Oriented Security
Mental Models of System BehaviorHow Attackers Exploit Our Mental ModelsRefining Our Mental ModelsResilience Stress TestingThe E&E Resilience Assessment ApproachEvaluation: Tier 1 AssessmentMapping Flows to Critical FunctionalityDocument Assumptions About Safety BoundariesMaking Attacker Math Work for YouStarting the Feedback Flywheel with Decision TreesMoving Toward Tier 2: ExperimentationExperimentation: Tier 2 AssessmentThe Value of Experimental EvidenceSustaining Resilience AssessmentsFail-Safe Versus Safe-to-FailUncertainty Versus AmbiguityFail-Safe Neglects the Systems PerspectiveThe Fragmented World of Fail-SafeSCE Versus Security TheaterWhat Is Security Theater?How Does SCE Differ from Security Theater?How to RAVE Your Way to ResilienceRepeatability: Handling ComplexityAccessibility: Making Security Easier for EngineersVariability: Supporting EvolutionChapter Takeaways
3. Architecting and Designing
The Effort Investment PortfolioAllocating Your Effort Investment PortfolioInvesting Effort Based on Local ContextThe Four Failure Modes Resulting from System DesignThe Two Key Axes of Resilient Design: Coupling and ComplexityDesigning to Preserve PossibilitiesCoupling in Complex SystemsThe Tight Coupling Trade-OffThe Dangers of Tight Coupling: Taming the ForestInvesting in Loose Coupling in Software SystemsChaos Experiments Expose CouplingComplexity in Complex SystemsUnderstanding Complexity: Essential and AccidentalComplexity and Mental ModelsIntroducing Linearity into Our SystemsDesigning for Interactivity: Identity and Access ManagementNavigating Flawed Mental ModelsChapter Takeaways
4. Building and Delivering
Mental Models When Developing SoftwareWho Owns Application Security (and Resilience)?Lessons We Can Learn from Database Administration Going DevOpsDecisions on Critical Functionality Before BuildingDefining System Goals and Guidelines on “What to Throw Out the Airlock”Code Reviews and Mental Models“Boring” Technology Is Resilient TechnologyStandardization of Raw MaterialsDeveloping and Delivering to Expand Safety BoundariesAnticipating Scale and SLOsAutomating Security Checks via CI/CDStandardization of Patterns and ToolsDependency Analysis and Prioritizing VulnerabilitiesObserve System Interactions Across Space-Time (or Make More Linear)Configuration as CodeFault Injection During DevelopmentIntegration Tests, Load Tests, and Test TheaterBeware Premature and Improper AbstractionsFostering Feedback Loops and Learning During Build and DeliverTest AutomationDocumenting Why and WhenDistributed Tracing and LoggingRefining How Humans Interact with Build and Delivery PracticesFlexibility and Willingness to ChangeIteration to Mimic EvolutionModularity: Humanity’s Ancient Tool for ResilienceFeature Flags and Dark LaunchesPreserving Possibilities for Refactoring: TypingThe Strangler Fig PatternChapter Takeaways
5. Operating and Observing
What Does Operating and Observing Involve?Operational Goals in SCEThe Overlap of SRE and SecurityMeasuring Operational SuccessCrafting Success Metrics like AttackersThe DORA MetricsSLOs, SLAs, and Principled Performance AnalyticsEmbracing Confidence-Based SecurityObservability for Resilience and SecurityThresholding to Uncover Safety BoundariesAttack ObservabilityScalable Is SaferNavigating ScalabilityAutomating Away ToilChapter Takeaways
6. Responding and Recovering
Responding to Surprises in Complex SystemsIncident Response and the Effort Investment PortfolioAction Bias in Incident ResponsePracticing Response ActivitiesRecovering from SurprisesBlameless CultureBlaming Human ErrorHindsight Bias and Outcome BiasThe Just-World HypothesisNeutral Practitioner QuestionsChapter Takeaways
7. Platform Resilience Engineering
Production Pressures and How They Influence System BehaviorWhat Is Platform Engineering?Defining a VisionDefining a User ProblemLocal Context Is CriticalUser Personas, Stories, and JourneysUnderstanding How Humans Make Trade-Offs Under PressureDesigning a SolutionThe Ice Cream Cone Hierarchy of Security SolutionsSystem Design and Redesign to Eliminate HazardsSubstitute Less Hazardous Methods or MaterialsIncorporate Safety Devices and GuardsProvide Warning and Awareness SystemsApply Administrative Controls Including Guidelines and TrainingTwo Paths: The Control Strategy or the Resilience StrategyExperimentation and Feedback Loops for Solution DesignImplementing a SolutionFostering ConsensusPlanning for MigrationSuccess MetricsChapter Takeaways
8. Security Chaos Experiments
Lessons Learned from Early AdoptersLesson #1. Start in Nonproduction Environments; You Can Still Learn a LotLesson #2. Use Past Incidents as a Source of ExperimentsLesson #3. Publish and Evangelize Experimental FindingsSetting Experiments Up for SuccessDesigning a HypothesisDesigning an ExperimentExperiment Design SpecificationsConducting ExperimentsCollecting EvidenceAnalyzing and Documenting EvidenceCapturing Knowledge for Feedback LoopsDocument Experiment Release NotesAutomating ExperimentsEasing into Chaos: Game DaysExample Security Chaos ExperimentsSecurity Chaos Experiments for Production InfrastructureSecurity Chaos Experiments for Build PipelinesSecurity Chaos Experiments in Cloud Native EnvironmentsSecurity Chaos Experiments in Windows EnvironmentsChapter Takeaways
9. Security Chaos Engineering in the Wild
Experience Report: The Existence of Order Through Chaos (UnitedHealth Group)The Story of ChaoSlingrStep-by-Step Example: PortSlingrExperience Report: A Quest for Stronger Reliability (Verizon)The Bigger They Are…All Hands on Deck Means No Hands on the HelmAssert Your HypothesisReliability ExperimentsCost ExperimentsPerformance ExperimentsRisk ExperimentsMore Traditionally Known ExperimentsChanging the Paradigm to ContinuousLessons LearnedExperience Report: Security Monitoring (OpenDoor)Experience Report: Applied Security (Cardinal Health)Building the SCE CultureThe Mission of Applied SecurityThe Method: Continuous Verification and Validation (CVV)The CVV Process Includes Four StepsExperience Report: Balancing Reliability and Security via SCE (Accenture Global)Our Roadmap to SCE Enterprise CapabilityOur Process for AdoptionExperience Report: Cyber Chaos Engineering (Capital One)What Does All This Have to Do with SCE?What Is Secure Today May Not Be Secure TomorrowHow We StartedHow We Did This in Ye Olden DaysThings I’ve Learned Along the WayA Reduction of GuessworkDriving ValueConclusionChapter Takeaways

Index
About the Authors

Content preview from Security Chaos Engineering

About the Authors

Kelly Shortridge is a senior principal engineer at Fastly in the office of the CTO. Shortridge is best known for their work on resilience in complex software systems, the application of behavioral economics to cybersecurity, and bringing security out of the dark ages. Shortridge has been a successful enterprise product leader as well as a startup founder (with an exit to CrowdStrike) and investment banker. Shortridge frequently advises Fortune 500s, investors, startups, and federal agencies and has spoken at major technology conferences internationally, including Black Hat USA, O’Reilly Velocity Conference, and SREcon. Shortridge’s research has been featured in ACM, IEEE, and USENIX, spanning behavioral science in cybersecurity, deception strategies, and the ROI of software resilience. They also serve on ACM Queue’s magazine editorial board.

Aaron Rinehart is a senior distinguished engineer, SRE and Chaos Engineering at Capital One. He has spent his career solving complex engineering problems and transforming cybersecurity practices across a diverse set of industries: healthcare, insurance, government, aerospace, technology, higher education, and the military. Aaron has been expanding the possibilities of chaos engineering in its application to other safety-critical portions of the IT domain, most notably in cybersecurity. He began shaping the application of chaos engineering within cybersecurity during his tenure as the Chief Security Architect at a Fortune ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098113810Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Security Chaos Engineering

by Kelly Shortridge, Aaron Rinehart

About the Authors

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.