Building Secure and Reliable Systems

Building Secure and Reliable Systems

by Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield
Released March 2020
Publisher(s): O'Reilly Media, Inc.
ISBN: 9781492083122

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Buy on Amazon Buy on ebooks.com
Start your free trial

Book description

Can a system be considered truly reliable if it isn't fundamentally secure? Or can it be considered secure if it's unreliable? Security is crucial to the design and operation of scalable systems in production, as it plays an important part in product quality, performance, and availability. In this book, experts from Google share best practices to help your organization design scalable and reliable systems that are fundamentally secure.

Two previous O’Reilly books from Google—Site Reliability Engineering and The Site Reliability Workbook—demonstrated how and why a commitment to the entire service lifecycle enables organizations to successfully build, deploy, monitor, and maintain software systems. In this latest guide, the authors offer insights into system design, implementation, and maintenance from practitioners who specialize in security and reliability. They also discuss how building and adopting their recommended best practices requires a culture that’s supportive of such change.

You’ll learn about secure and reliable systems through:

  • Design strategies
  • Recommendations for coding, testing, and debugging practices
  • Strategies to prepare for, respond to, and recover from incidents
  • Cultural best practices that help teams across your organization collaborate effectively

Publisher resources

View/Submit Errata

Table of contents

  1. Foreword by Royal Hansen
  2. Foreword by Michael Wildpaner
  3. Preface
    1. Why We Wrote This Book
    2. Who This Book Is For
    3. A Note About Culture
    4. How to Read This Book
    5. Conventions Used in This Book
    6. O’Reilly Online Learning
    7. How to Contact Us
    8. Acknowledgments
  4. I. Introductory Material
  5. 1. The Intersection of Security and Reliability
    1. On Passwords and Power Drills
    2. Reliability Versus Security: Design Considerations
    3. Confidentiality, Integrity, Availability
      1. Confidentiality
      2. Integrity
      3. Availability
    4. Reliability and Security: Commonalities
      1. Invisibility
      2. Assessment
      3. Simplicity
      4. Evolution
      5. Resilience
      6. From Design to Production
      7. Investigating Systems and Logging
      8. Crisis Response
      9. Recovery
    5. Conclusion
  6. 2. Understanding Adversaries
    1. Attacker Motivations
    2. Attacker Profiles
      1. Hobbyists
      2. Vulnerability Researchers
      3. Governments and Law Enforcement
      4. Activists
      5. Criminal Actors
      6. Automation and Artificial Intelligence
      7. Insiders
    3. Attacker Methods
      1. Threat Intelligence
      2. Cyber Kill Chains™
      3. Tactics, Techniques, and Procedures
    4. Risk Assessment Considerations
    5. Conclusion
  7. II. Designing Systems
  8. 3. Case Study: Safe Proxies
    1. Safe Proxies in Production Environments
    2. Google Tool Proxy
    3. Conclusion
  9. 4. Design Tradeoffs
    1. Design Objectives and Requirements
      1. Feature Requirements
      2. Nonfunctional Requirements
      3. Features Versus Emergent Properties
      4. Example: Google Design Document
    2. Balancing Requirements
      1. Example: Payment Processing
    3. Managing Tensions and Aligning Goals
      1. Example: Microservices and the Google Web Application Framework
      2. Aligning Emergent-Property Requirements
    4. Initial Velocity Versus Sustained Velocity
    5. Conclusion
  10. 5. Design for Least Privilege
    1. Concepts and Terminology
      1. Least Privilege
      2. Zero Trust Networking
      3. Zero Touch
    2. Classifying Access Based on Risk
    3. Best Practices
      1. Small Functional APIs
      2. Breakglass
      3. Auditing
      4. Testing and Least Privilege
      5. Diagnosing Access Denials
      6. Graceful Failure and Breakglass Mechanisms
    4. Worked Example: Configuration Distribution
      1. POSIX API via OpenSSH
      2. Software Update API
      3. Custom OpenSSH ForceCommand
      4. Custom HTTP Receiver (Sidecar)
      5. Custom HTTP Receiver (In-Process)
      6. Tradeoffs
    5. A Policy Framework for Authentication and Authorization Decisions
      1. Using Advanced Authorization Controls
      2. Investing in a Widely Used Authorization Framework
      3. Avoiding Potential Pitfalls
    6. Advanced Controls
      1. Multi-Party Authorization (MPA)
      2. Three-Factor Authorization (3FA)
      3. Business Justifications
      4. Temporary Access
      5. Proxies
    7. Tradeoffs and Tensions
      1. Increased Security Complexity
      2. Impact on Collaboration and Company Culture
      3. Quality Data and Systems That Impact Security
      4. Impact on User Productivity
      5. Impact on Developer Complexity
    8. Conclusion
  11. 6. Design for Understandability
    1. Why Is Understandability Important?
      1. System Invariants
      2. Analyzing Invariants
      3. Mental Models
    2. Designing Understandable Systems
      1. Complexity Versus Understandability
      2. Breaking Down Complexity
      3. Centralized Responsibility for Security and Reliability Requirements
    3. System Architecture
      1. Understandable Interface Specifications
      2. Understandable Identities, Authentication, and Access Control
      3. Security Boundaries
    4. Software Design
      1. Using Application Frameworks for Service-Wide Requirements
      2. Understanding Complex Data Flows
      3. Considering API Usability
    5. Conclusion
  12. 7. Design for a Changing Landscape
    1. Types of Security Changes
    2. Designing Your Change
    3. Architecture Decisions to Make Changes Easier
      1. Keep Dependencies Up to Date and Rebuild Frequently
      2. Release Frequently Using Automated Testing
      3. Use Containers
      4. Use Microservices
    4. Different Changes: Different Speeds, Different Timelines
      1. Short-Term Change: Zero-Day Vulnerability
      2. Medium-Term Change: Improvement to Security Posture
      3. Long-Term Change: External Demand
    5. Complications: When Plans Change
    6. Example: Growing Scope—Heartbleed
    7. Conclusion
  13. 8. Design for Resilience
    1. Design Principles for Resilience
    2. Defense in Depth
      1. The Trojan Horse
      2. Google App Engine Analysis
    3. Controlling Degradation
      1. Differentiate Costs of Failures
      2. Deploy Response Mechanisms
      3. Automate Responsibly
    4. Controlling the Blast Radius
      1. Role Separation
      2. Location Separation
      3. Time Separation
    5. Failure Domains and Redundancies
      1. Failure Domains
      2. Component Types
      3. Controlling Redundancies
    6. Continuous Validation
      1. Validation Focus Areas
      2. Validation in Practice
    7. Practical Advice: Where to Begin
    8. Conclusion
  14. 9. Design for Recovery
    1. What Are We Recovering From?
      1. Random Errors
      2. Accidental Errors
      3. Software Errors
      4. Malicious Actions
    2. Design Principles for Recovery
      1. Design to Go as Quickly as Possible (Guarded by Policy)
      2. Limit Your Dependencies on External Notions of Time
      3. Rollbacks Represent a Tradeoff Between Security and Reliability
      4. Use an Explicit Revocation Mechanism
      5. Know Your Intended State, Down to the Bytes
      6. Design for Testing and Continuous Validation
    3. Emergency Access
      1. Access Controls
      2. Communications
      3. Responder Habits
    4. Unexpected Benefits
    5. Conclusion
  15. 10. Mitigating Denial-of-Service Attacks
    1. Strategies for Attack and Defense
      1. Attacker’s Strategy
      2. Defender’s Strategy
    2. Designing for Defense
      1. Defendable Architecture
      2. Defendable Services
    3. Mitigating Attacks
      1. Monitoring and Alerting
      2. Graceful Degradation
      3. A DoS Mitigation System
      4. Strategic Response
    4. Dealing with Self-Inflicted Attacks
      1. User Behavior
      2. Client Retry Behavior
    5. Conclusion
  16. III. Implementing Systems
  17. 11. Case Study: Designing, Implementing, and Maintaining a Publicly Trusted CA
    1. Background on Publicly Trusted Certificate Authorities
    2. Why Did We Need a Publicly Trusted CA?
    3. The Build or Buy Decision
    4. Design, Implementation, and Maintenance Considerations
      1. Programming Language Choice
      2. Complexity Versus Understandability
      3. Securing Third-Party and Open Source Components
      4. Testing
      5. Resiliency for the CA Key Material
      6. Data Validation
    5. Conclusion
  18. 12. Writing Code
    1. Frameworks to Enforce Security and Reliability
      1. Benefits of Using Frameworks
      2. Example: Framework for RPC Backends
    2. Common Security Vulnerabilities
      1. SQL Injection Vulnerabilities: TrustedSqlString
      2. Preventing XSS: SafeHtml
    3. Lessons for Evaluating and Building Frameworks
      1. Simple, Safe, Reliable Libraries for Common Tasks
      2. Rollout Strategy
    4. Simplicity Leads to Secure and Reliable Code
      1. Avoid Multilevel Nesting
      2. Eliminate YAGNI Smells
      3. Repay Technical Debt
      4. Refactoring
    5. Security and Reliability by Default
      1. Choose the Right Tools
      2. Use Strong Types
      3. Sanitize Your Code
    6. Conclusion
  19. 13. Testing Code
    1. Unit Testing
      1. Writing Effective Unit Tests
      2. When to Write Unit Tests
      3. How Unit Testing Affects Code
    2. Integration Testing
      1. Writing Effective Integration Tests
    3. Dynamic Program Analysis
    4. Fuzz Testing
      1. How Fuzz Engines Work
      2. Writing Effective Fuzz Drivers
      3. An Example Fuzzer
      4. Continuous Fuzzing
    5. Static Program Analysis
      1. Automated Code Inspection Tools
      2. Integration of Static Analysis in the Developer Workflow
      3. Abstract Interpretation
      4. Formal Methods
    6. Conclusion
  20. 14. Deploying Code
    1. Concepts and Terminology
    2. Threat Model
    3. Best Practices
      1. Require Code Reviews
      2. Rely on Automation
      3. Verify Artifacts, Not Just People
      4. Treat Configuration as Code
    4. Securing Against the Threat Model
    5. Advanced Mitigation Strategies
      1. Binary Provenance
      2. Provenance-Based Deployment Policies
      3. Verifiable Builds
      4. Deployment Choke Points
      5. Post-Deployment Verification
    6. Practical Advice
      1. Take It One Step at a Time
      2. Provide Actionable Error Messages
      3. Ensure Unambiguous Provenance
      4. Create Unambiguous Policies
      5. Include a Deployment Breakglass
    7. Securing Against the Threat Model, Revisited
    8. Conclusion
  21. 15. Investigating Systems
    1. From Debugging to Investigation
      1. Example: Temporary Files
      2. Debugging Techniques
      3. What to Do When You’re Stuck
      4. Collaborative Debugging: A Way to Teach
      5. How Security Investigations and Debugging Differ
    2. Collect Appropriate and Useful Logs
      1. Design Your Logging to Be Immutable
      2. Take Privacy into Consideration
      3. Determine Which Security Logs to Retain
      4. Budget for Logging
    3. Robust, Secure Debugging Access
      1. Reliability
      2. Security
    4. Conclusion
  22. IV. Maintaining Systems
  23. 16. Disaster Planning
    1. Defining “Disaster”
    2. Dynamic Disaster Response Strategies
    3. Disaster Risk Analysis
    4. Setting Up an Incident Response Team
      1. Identify Team Members and Roles
      2. Establish a Team Charter
      3. Establish Severity and Priority Models
      4. Define Operating Parameters for Engaging the IR Team
      5. Develop Response Plans
      6. Create Detailed Playbooks
      7. Ensure Access and Update Mechanisms Are in Place
    5. Prestaging Systems and People Before an Incident
      1. Configuring Systems
      2. Training
      3. Processes and Procedures
    6. Testing Systems and Response Plans
      1. Auditing Automated Systems
      2. Conducting Nonintrusive Tabletops
      3. Testing Response in Production Environments
      4. Red Team Testing
      5. Evaluating Responses
    7. Google Examples
      1. Test with Global Impact
      2. DiRT Exercise Testing Emergency Access
      3. Industry-Wide Vulnerabilities
    8. Conclusion
  24. 17. Crisis Management
    1. Is It a Crisis or Not?
      1. Triaging the Incident
      2. Compromises Versus Bugs
    2. Taking Command of Your Incident
      1. The First Step: Don’t Panic!
      2. Beginning Your Response
      3. Establishing Your Incident Team
      4. Operational Security
      5. Trading Good OpSec for the Greater Good
      6. The Investigative Process
    3. Keeping Control of the Incident
      1. Parallelizing the Incident
      2. Handovers
      3. Morale
    4. Communications
      1. Misunderstandings
      2. Hedging
      3. Meetings
      4. Keeping the Right People Informed with the Right Levels of Detail
    5. Putting It All Together
      1. Triage
      2. Declaring an Incident
      3. Communications and Operational Security
      4. Beginning the Incident
      5. Handover
      6. Handing Back the Incident
      7. Preparing Communications and Remediation
      8. Closure
    6. Conclusion
  25. 18. Recovery and Aftermath
    1. Recovery Logistics
    2. Recovery Timeline
    3. Planning the Recovery
      1. Scoping the Recovery
      2. Recovery Considerations
      3. Recovery Checklists
    4. Initiating the Recovery
      1. Isolating Assets (Quarantine)
      2. System Rebuilds and Software Upgrades
      3. Data Sanitization
      4. Recovery Data
      5. Credential and Secret Rotation
    5. After the Recovery
      1. Postmortems
    6. Examples
      1. Compromised Cloud Instances
      2. Large-Scale Phishing Attack
      3. Targeted Attack Requiring Complex Recovery
    7. Conclusion
  26. V. Organization and Culture
  27. 19. Case Study: Chrome Security Team
    1. Background and Team Evolution
    2. Security Is a Team Responsibility
    3. Help Users Safely Navigate the Web
    4. Speed Matters
    5. Design for Defense in Depth
    6. Be Transparent and Engage the Community
    7. Conclusion
  28. 20. Understanding Roles and Responsibilities
    1. Who Is Responsible for Security and Reliability?
      1. The Roles of Specialists
      2. Understanding Security Expertise
      3. Certifications and Academia
    2. Integrating Security into the Organization
      1. Embedding Security Specialists and Security Teams
      2. Example: Embedding Security at Google
      3. Special Teams: Blue and Red Teams
      4. External Researchers
    3. Conclusion
  29. 21. Building a Culture of Security and Reliability
    1. Defining a Healthy Security and Reliability Culture
      1. Culture of Security and Reliability by Default
      2. Culture of Review
      3. Culture of Awareness
      4. Culture of Yes
      5. Culture of Inevitably
      6. Culture of Sustainability
    2. Changing Culture Through Good Practice
      1. Align Project Goals and Participant Incentives
      2. Reduce Fear with Risk-Reduction Mechanisms
      3. Make Safety Nets the Norm
      4. Increase Productivity and Usability
      5. Overcommunicate and Be Transparent
      6. Build Empathy
    3. Convincing Leadership
      1. Understand the Decision-Making Process
      2. Build a Case for Change
      3. Pick Your Battles
      4. Escalations and Problem Resolution
    4. Conclusion
  30. Conclusion
  31. A. A Disaster Risk Assessment Matrix
  32. Index

Product information

  • Title: Building Secure and Reliable Systems
  • Author(s): Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield
  • Release date: March 2020
  • Publisher(s): O'Reilly Media, Inc.
  • ISBN: 9781492083122