book

Security Chaos Engineering

by Kelly Shortridge, Aaron Rinehart

March 2023

Intermediate to advanced

428 pages

14h 36m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?Scope of This BookOutline of This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
What Is a Complex System?Variety Defines Complex SystemsComplex Systems Are AdaptiveThe Holistic Nature of Complex SystemsWhat Is Failure?Acute and Chronic Stressors in Complex SystemsSurprises in Complex SystemsWhat Is Resilience?Critical FunctionalitySafety Boundaries (Thresholds)Interactions Across Space-TimeFeedback Loops and Learning CultureFlexibility and Openness to ChangeResilience Is a VerbResilience: Myth Versus RealityMyth: Robustness = ResilienceMyth: We Can and Should Prevent FailureMyth: The Security of Each Component Adds Up to ResilienceMyth: Creating a “Security Culture” Fixes Human ErrorChapter Takeaways
Mental Models of System BehaviorHow Attackers Exploit Our Mental ModelsRefining Our Mental ModelsResilience Stress TestingThe E&E Resilience Assessment ApproachEvaluation: Tier 1 AssessmentMapping Flows to Critical FunctionalityDocument Assumptions About Safety BoundariesMaking Attacker Math Work for YouStarting the Feedback Flywheel with Decision TreesMoving Toward Tier 2: ExperimentationExperimentation: Tier 2 AssessmentThe Value of Experimental EvidenceSustaining Resilience AssessmentsFail-Safe Versus Safe-to-FailUncertainty Versus AmbiguityFail-Safe Neglects the Systems PerspectiveThe Fragmented World of Fail-SafeSCE Versus Security TheaterWhat Is Security Theater?How Does SCE Differ from Security Theater?How to RAVE Your Way to ResilienceRepeatability: Handling ComplexityAccessibility: Making Security Easier for EngineersVariability: Supporting EvolutionChapter Takeaways
The Effort Investment PortfolioAllocating Your Effort Investment PortfolioInvesting Effort Based on Local ContextThe Four Failure Modes Resulting from System DesignThe Two Key Axes of Resilient Design: Coupling and ComplexityDesigning to Preserve PossibilitiesCoupling in Complex SystemsThe Tight Coupling Trade-OffThe Dangers of Tight Coupling: Taming the ForestInvesting in Loose Coupling in Software SystemsChaos Experiments Expose CouplingComplexity in Complex SystemsUnderstanding Complexity: Essential and AccidentalComplexity and Mental ModelsIntroducing Linearity into Our SystemsDesigning for Interactivity: Identity and Access ManagementNavigating Flawed Mental ModelsChapter Takeaways
Mental Models When Developing SoftwareWho Owns Application Security (and Resilience)?Lessons We Can Learn from Database Administration Going DevOpsDecisions on Critical Functionality Before BuildingDefining System Goals and Guidelines on “What to Throw Out the Airlock”Code Reviews and Mental Models“Boring” Technology Is Resilient TechnologyStandardization of Raw MaterialsDeveloping and Delivering to Expand Safety BoundariesAnticipating Scale and SLOsAutomating Security Checks via CI/CDStandardization of Patterns and ToolsDependency Analysis and Prioritizing VulnerabilitiesObserve System Interactions Across Space-Time (or Make More Linear)Configuration as CodeFault Injection During DevelopmentIntegration Tests, Load Tests, and Test TheaterBeware Premature and Improper AbstractionsFostering Feedback Loops and Learning During Build and DeliverTest AutomationDocumenting Why and WhenDistributed Tracing and LoggingRefining How Humans Interact with Build and Delivery PracticesFlexibility and Willingness to ChangeIteration to Mimic EvolutionModularity: Humanity’s Ancient Tool for ResilienceFeature Flags and Dark LaunchesPreserving Possibilities for Refactoring: TypingThe Strangler Fig PatternChapter Takeaways
What Does Operating and Observing Involve?Operational Goals in SCEThe Overlap of SRE and SecurityMeasuring Operational SuccessCrafting Success Metrics like AttackersThe DORA MetricsSLOs, SLAs, and Principled Performance AnalyticsEmbracing Confidence-Based SecurityObservability for Resilience and SecurityThresholding to Uncover Safety BoundariesAttack ObservabilityScalable Is SaferNavigating ScalabilityAutomating Away ToilChapter Takeaways
Responding to Surprises in Complex SystemsIncident Response and the Effort Investment PortfolioAction Bias in Incident ResponsePracticing Response ActivitiesRecovering from SurprisesBlameless CultureBlaming Human ErrorHindsight Bias and Outcome BiasThe Just-World HypothesisNeutral Practitioner QuestionsChapter Takeaways
Production Pressures and How They Influence System BehaviorWhat Is Platform Engineering?Defining a VisionDefining a User ProblemLocal Context Is CriticalUser Personas, Stories, and JourneysUnderstanding How Humans Make Trade-Offs Under PressureDesigning a SolutionThe Ice Cream Cone Hierarchy of Security SolutionsSystem Design and Redesign to Eliminate HazardsSubstitute Less Hazardous Methods or MaterialsIncorporate Safety Devices and GuardsProvide Warning and Awareness SystemsApply Administrative Controls Including Guidelines and TrainingTwo Paths: The Control Strategy or the Resilience StrategyExperimentation and Feedback Loops for Solution DesignImplementing a SolutionFostering ConsensusPlanning for MigrationSuccess MetricsChapter Takeaways
Lessons Learned from Early AdoptersLesson #1. Start in Nonproduction Environments; You Can Still Learn a LotLesson #2. Use Past Incidents as a Source of ExperimentsLesson #3. Publish and Evangelize Experimental FindingsSetting Experiments Up for SuccessDesigning a HypothesisDesigning an ExperimentExperiment Design SpecificationsConducting ExperimentsCollecting EvidenceAnalyzing and Documenting EvidenceCapturing Knowledge for Feedback LoopsDocument Experiment Release NotesAutomating ExperimentsEasing into Chaos: Game DaysExample Security Chaos ExperimentsSecurity Chaos Experiments for Production InfrastructureSecurity Chaos Experiments for Build PipelinesSecurity Chaos Experiments in Cloud Native EnvironmentsSecurity Chaos Experiments in Windows EnvironmentsChapter Takeaways
Experience Report: The Existence of Order Through Chaos (UnitedHealth Group)The Story of ChaoSlingrStep-by-Step Example: PortSlingrExperience Report: A Quest for Stronger Reliability (Verizon)The Bigger They Are…All Hands on Deck Means No Hands on the HelmAssert Your HypothesisReliability ExperimentsCost ExperimentsPerformance ExperimentsRisk ExperimentsMore Traditionally Known ExperimentsChanging the Paradigm to ContinuousLessons LearnedExperience Report: Security Monitoring (OpenDoor)Experience Report: Applied Security (Cardinal Health)Building the SCE CultureThe Mission of Applied SecurityThe Method: Continuous Verification and Validation (CVV)The CVV Process Includes Four StepsExperience Report: Balancing Reliability and Security via SCE (Accenture Global)Our Roadmap to SCE Enterprise CapabilityOur Process for AdoptionExperience Report: Cyber Chaos Engineering (Capital One)What Does All This Have to Do with SCE?What Is Secure Today May Not Be Secure TomorrowHow We StartedHow We Did This in Ye Olden DaysThings I’ve Learned Along the WayA Reduction of GuessworkDriving ValueConclusionChapter Takeaways

Overview

Cybersecurity is broken. Year after year, attackers remain unchallenged and undeterred, while engineering teams feel pressure to design, build, and operate "secure" systems. Failure can't be prevented, mental models of systems are incomplete, and our digital world constantly evolves. How can we verify that our systems behave the way we expect? What can we do to improve our systems' resilience?

In this comprehensive guide, authors Kelly Shortridge and Aaron Rinehart help you navigate the challenges of sustaining resilience in complex software systems by using the principles and practices of security chaos engineering. By preparing for adverse events, you can ensure they don't disrupt your ability to innovate, move quickly, and achieve your engineering and business goals.

Learn how to design a modern security program
Make informed decisions at each phase of software delivery to nurture resilience and adaptive capacity
Understand the complex systems dynamics upon which resilience outcomes depend
Navigate technical and organizational trade-offsthat distort decision making in systems
Explore chaos experimentation to verify critical assumptions about software quality and security
Learn how major enterprises leverage security chaos engineering