book

Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2

Name: Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2
ISBN: 9780133478549

by Thomas A. Limoncelli, Strata R. Chalup, Christina J. Hogan

September 2014

Intermediate to advanced

560 pages

15h 52m

English

Addison-Wesley Professional

Read now

Unlock full access

About This eBook
Title Page
Copyright Page
Contents at a Glance
Contents
Preface
About This BookAcknowledgmentsPart I Design: Building ItPart II Operations: Running ItPart III Appendices
About the Authors
Introduction
Business ObjectivesIdeal System ArchitectureIdeal Release ProcessIdeal Operations
Part I: Design: Building It
Chapter 1. Designing in a Distributed World
1.1 Visibility at Scale1.2 The Importance of Simplicity1.3 Composition1.3.1 Load Balancer with Multiple Backend Replicas1.3.2 Server with Multiple Backends1.3.3 Server Tree1.4 Distributed State1.5 The CAP Principle1.5.1 Consistency1.5.2 Availability1.5.3 Partition Tolerance1.6 Loosely Coupled Systems1.7 Speed1.8 SummaryExercises

Chapter 2. Designing for Operations
2.1 Operational Requirements2.1.1 Configuration2.1.2 Startup and Shutdown2.1.3 Queue Draining2.1.4 Software Upgrades2.1.5 Backups and Restores2.1.6 Redundancy2.1.7 Replicated Databases2.1.8 Hot Swaps2.1.9 Toggles for Individual Features2.1.10 Graceful Degradation2.1.11 Access Controls and Rate Limits2.1.12 Data Import Controls2.1.13 Monitoring2.1.14 Auditing2.1.15 Debug Instrumentation2.1.16 Exception Collection2.1.17 Documentation for Operations2.2 Implementing Design for Operations2.2.1 Build Features in from the Beginning2.2.2 Request Features as They Are Identified2.2.3 Write the Features Yourself2.2.4 Work with a Third-Party Vendor2.3 Improving the Model2.4 SummaryExercises
Chapter 3. Selecting a Service Platform
3.1 Level of Service Abstraction3.1.1 Infrastructure as a Service3.1.2 Platform as a Service3.1.3 Software as a Service3.2 Type of Machine3.2.1 Physical Machines3.2.2 Virtual Machines3.2.3 Containers3.3 Level of Resource Sharing3.3.1 Compliance3.3.2 Privacy3.3.3 Cost3.3.4 Control3.4 Colocation3.5 Selection Strategies3.6 SummaryExercises
Chapter 4. Application Architectures
4.1 Single-Machine Web Server4.2 Three-Tier Web Service4.2.1 Load Balancer Types4.2.2 Load Balancing Methods4.2.3 Load Balancing with Shared State4.2.4 User Identity4.2.5 Scaling4.3 Four-Tier Web Service4.3.1 Frontends4.3.2 Application Servers4.3.3 Configuration Options4.4 Reverse Proxy Service4.5 Cloud-Scale Service4.5.1 Global Load Balancer4.5.2 Global Load Balancing Methods4.5.3 Global Load Balancing with User-Specific Data4.5.4 Internal Backbone4.6 Message Bus Architectures4.6.1 Message Bus Designs4.6.2 Message Bus Reliability4.6.3 Example 1: Link-Shortening Site4.6.4 Example 2: Employee Human Resources Data Updates4.7 Service-Oriented Architecture4.7.1 Flexibility4.7.2 Support4.7.3 Best Practices4.8 SummaryExercises
Chapter 5. Design Patterns for Scaling
5.1 General Strategy5.1.1 Identify Bottlenecks5.1.2 Reengineer Components5.1.3 Measure Results5.1.4 Be Proactive5.2 Scaling Up5.3 The AKF Scaling Cube5.3.1 x: Horizontal Duplication5.3.2 y: Functional or Service Splits5.3.3 z: Lookup-Oriented Split5.3.4 Combinations5.4 Caching5.4.1 Cache Effectiveness5.4.2 Cache Placement5.4.3 Cache Persistence5.4.4 Cache Replacement Algorithms5.4.5 Cache Entry Invalidation5.4.6 Cache Size5.5 Data Sharding5.6 Threading5.7 Queueing5.7.1 Benefits5.7.2 Variations5.8 Content Delivery Networks5.9 SummaryExercises
Chapter 6. Design Patterns for Resiliency
6.1 Software Resiliency Beats Hardware Reliability6.2 Everything Malfunctions Eventually6.2.1 MTBF in Distributed Systems6.2.2 The Traditional Approach6.2.3 The Distributed Computing Approach6.3 Resiliency through Spare Capacity6.3.1 How Much Spare Capacity6.3.2 Load Sharing versus Hot Spares6.4 Failure Domains6.5 Software Failures6.5.1 Software Crashes6.5.2 Software Hangs6.5.3 Query of Death6.6 Physical Failures6.6.1 Parts and Components6.6.2 Machines6.6.3 Load Balancers6.6.4 Racks6.6.5 Datacenters6.7 Overload Failures6.7.1 Traffic Surges6.7.2 DoS and DDoS Attacks6.7.3 Scraping Attacks6.8 Human Error6.9 SummaryExercises
Part II Operations: Running It
Chapter 7. Operations in a Distributed World
7.1 Distributed Systems Operations7.1.1 SRE versus Traditional Enterprise IT7.1.2 Change versus Stability7.1.3 Defining SRE7.1.4 Operations at Scale7.2 Service Life Cycle7.2.1 Service Launches7.2.2 Service Decommissioning7.3 Organizing Strategy for Operational Teams7.3.1 Team Member Day Types7.3.2 Other Strategies7.4 Virtual Office7.4.1 Communication Mechanisms7.4.2 Communication Policies7.5 SummaryExercises
Chapter 8. DevOps Culture
8.1 What Is DevOps?8.1.1 The Traditional Approach8.1.2 The DevOps Approach8.2 The Three Ways of DevOps8.2.1 The First Way: Workflow8.2.2 The Second Way: Improve Feedback8.2.3 The Third Way: Continual Experimentation and Learning8.2.4 Small Batches Are Better8.2.5 Adopting the Strategies8.3 History of DevOps8.3.1 Evolution8.3.2 Site Reliability Engineering8.4 DevOps Values and Principles8.4.1 Relationships8.4.2 Integration8.4.3 Automation8.4.4 Continuous Improvement8.4.5 Common Nontechnical DevOps Practices8.4.6 Common Technical DevOps Practices8.4.7 Release Engineering DevOps Practices8.5 Converting to DevOps8.5.1 Getting Started8.5.2 DevOps at the Business Level8.6 Agile and Continuous Delivery8.6.1 What Is Agile?8.6.2 What Is Continuous Delivery?8.7 SummaryExercises
Chapter 9. Service Delivery: The Build Phase
9.1 Service Delivery Strategies9.1.1 Pattern: Modern DevOps Methodology9.1.2 Anti-pattern: Waterfall Methodology9.2 The Virtuous Cycle of Quality9.3 Build-Phase Steps9.3.1 Develop9.3.2 Commit9.3.3 Build9.3.4 Package9.3.5 Register9.4 Build Console9.5 Continuous Integration9.6 Packages as Handoff Interface9.7 SummaryExercises
Chapter 10. Service Delivery: The Deployment Phase
10.1 Deployment-Phase Steps10.1.1 Promotion10.1.2 Installation10.1.3 Configuration10.2 Testing and Approval10.2.1 Testing10.2.2 Approval10.3 Operations Console10.4 Infrastructure Automation Strategies10.4.1 Preparing Physical Machines10.4.2 Preparing Virtual Machines10.4.3 Installing OS and Services10.5 Continuous Delivery10.6 Infrastructure as Code10.7 Other Platform Services10.8 SummaryExercises
Chapter 11. Upgrading Live Services
11.1 Taking the Service Down for Upgrading11.2 Rolling Upgrades11.3 Canary11.4 Phased Roll-outs11.5 Proportional Shedding11.6 Blue-Green Deployment11.7 Toggling Features11.8 Live Schema Changes11.9 Live Code Changes11.10 Continuous Deployment11.11 Dealing with Failed Code Pushes11.12 Release Atomicity11.13 SummaryExercises
Chapter 12. Automation
12.1 Approaches to Automation12.1.1 The Left-Over Principle12.1.2 The Compensatory Principle12.1.3 The Complementarity Principle12.1.4 Automation for System Administration12.1.5 Lessons Learned12.2 Tool Building versus Automation12.2.1 Example: Auto Manufacturing12.2.2 Example: Machine Configuration12.2.3 Example: Account Creation12.2.4 Tools Are Good, But Automation Is Better12.3 Goals of Automation12.4 Creating Automation12.4.1 Making Time to Automate12.4.2 Reducing Toil12.4.3 Determining What to Automate First12.5 How to Automate12.6 Language Tools12.6.1 Shell Scripting Languages12.6.2 Scripting Languages12.6.3 Compiled Languages12.6.4 Configuration Management Languages12.7 Software Engineering Tools and Techniques12.7.1 Issue Tracking Systems12.7.2 Version Control Systems12.7.3 Software Packaging12.7.4 Style Guides12.7.5 Test-Driven Development12.7.6 Code Reviews12.7.7 Writing Just Enough Code12.8 Multitenant Systems12.9 SummaryExercises
Chapter 13. Design Documents
13.1 Design Documents Overview13.1.1 Documenting Changes and Rationale13.1.2 Documentation as a Repository of Past Decisions13.2 Design Document Anatomy13.3 Template13.4 Document Archive13.5 Review Workflows13.5.1 Reviewers and Approvers13.5.2 Achieving Sign-off13.6 Adopting Design Documents13.7 SummaryExercises
Chapter 14. Oncall
14.1 Designing Oncall14.1.1 Start with the SLA14.1.2 Oncall Roster14.1.3 Onduty14.1.4 Oncall Schedule Design14.1.5 The Oncall Calendar14.1.6 Oncall Frequency14.1.7 Types of Notifications14.1.8 After-Hours Maintenance Coordination14.2 Being Oncall14.2.1 Pre-shift Responsibilities14.2.2 Regular Oncall Responsibilities14.2.3 Alert Responsibilities14.2.4 Observe, Orient, Decide, Act (OODA)14.2.5 Oncall Playbook14.2.6 Third-Party Escalation14.2.7 End-of-Shift Responsibilities14.3 Between Oncall Shifts14.3.1 Long-Term Fixes14.3.2 Postmortems14.4 Periodic Review of Alerts14.5 Being Paged Too Much14.6 SummaryExercises
Chapter 15. Disaster Preparedness
15.1 Mindset15.1.1 Antifragile Systems15.1.2 Reducing Risk15.2 Individual Training: Wheel of Misfortune15.3 Team Training: Fire Drills15.3.1 Service Testing15.3.2 Random Testing15.4 Training for Organizations: Game Day/DiRT15.4.1 Getting Started15.4.2 Increasing Scope15.4.3 Implementation and Logistics15.4.4 Experiencing a DiRT Test15.5 Incident Command System15.5.1 How It Works: Public Safety Arena15.5.2 How It Works: IT Operations Arena15.5.3 Incident Action Plan15.5.4 Best Practices15.5.5 ICS Example15.6 SummaryExercises
Chapter 16. Monitoring Fundamentals
16.1 Overview16.1.1 Uses of Monitoring16.1.2 Service Management16.2 Consumers of Monitoring Information16.3 What to Monitor16.4 Retention16.5 Meta-monitoring16.6 Logs16.6.1 Approach16.6.2 Timestamps16.7 SummaryExercises
Chapter 17. Monitoring Architecture and Practice
17.1 Sensing and Measurement17.1.1 Blackbox versus Whitebox Monitoring17.1.2 Direct versus Synthesized Measurements17.1.3 Rate versus Capability Monitoring17.1.4 Gauges versus Counters17.2 Collection17.2.1 Push versus Pull17.2.2 Protocol Selection17.2.3 Server Component versus Agent versus Poller17.2.4 Central versus Regional Collectors17.3 Analysis and Computation17.4 Alerting and Escalation Manager17.4.1 Alerting, Escalation, and Acknowledgments17.4.2 Silence versus Inhibit17.5 Visualization17.5.1 Percentiles17.5.2 Stack Ranking17.5.3 Histograms17.6 Storage17.7 Configuration17.8 SummaryExercises
Chapter 18. Capacity Planning
18.1 Standard Capacity Planning18.1.1 Current Usage18.1.2 Normal Growth18.1.3 Planned Growth18.1.4 Headroom18.1.5 Resiliency18.1.6 Timetable18.2 Advanced Capacity Planning18.2.1 Identifying Your Primary Resources18.2.2 Knowing Your Capacity Limits18.2.3 Identifying Your Core Drivers18.2.4 Measuring Engagement18.2.5 Analyzing the Data18.2.6 Monitoring the Key Indicators18.2.7 Delegating Capacity Planning18.3 Resource Regression18.4 Launching New Services18.5 Reduce Provisioning Time18.6 SummaryExercises
Chapter 19. Creating KPIs
19.1 What Is a KPI?19.2 Creating KPIs19.2.1 Step 1: Envision the Ideal19.2.2 Step 2: Quantify Distance to the Ideal19.2.3 Step 3: Imagine How Behavior Will Change19.2.4 Step 4: Revise and Select19.2.5 Step 5: Deploy the KPI19.3 Example KPI: Machine Allocation19.3.1 The First Pass19.3.2 The Second Pass19.3.3 Evaluating the KPI19.4 Case Study: Error Budget19.4.1 Conflicting Goals19.4.2 A Unified Goal19.4.3 Everyone Benefits19.5 SummaryExercises
Chapter 20. Operational Excellence
20.1 What Does Operational Excellence Look Like?20.2 How to Measure Greatness20.3 Assessment Methodology20.3.1 Operational Responsibilities20.3.2 Assessment Levels20.3.3 Assessment Questions and Look-For’s20.4 Service Assessments20.4.1 Identifying What to Assess20.4.2 Assessing Each Service20.4.3 Comparing Results across Services20.4.4 Acting on the Results20.4.5 Assessment and Project Planning Frequencies20.5 Organizational Assessments20.6 Levels of Improvement20.7 Getting Started20.8 SummaryExercises
Epilogue
Part III Appendices
Appendix A. Assessments
A.1 Regular Tasks (RT)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.2 Emergency Response (ER)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.3 Monitoring and Metrics (MM)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.4 Capacity Planning (CP)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.5 Change Management (CM)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.6 New Product Introduction and Removal (NPI/NPR)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.7 Service Deployment and Decommissioning (SDD)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.8 Performance and Efficiency (PE)Sample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.9 Service Delivery: The Build PhaseSample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.10 Service Delivery: The Deployment PhaseSample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.11 Toil ReductionSample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: OptimizingA.12 Disaster PreparednessSample Assessment QuestionsLevel 1: InitialLevel 2: RepeatableLevel 3: DefinedLevel 4: ManagedLevel 5: Optimizing
Appendix B. The Origins and Future of Distributed Computing and Clouds
B.1 The Pre-Web Era (1985–1994)Availability RequirementsTechnologyScalingHigh AvailabilityCostsB.2 The First Web Era: The Bubble (1995–2000)Availability RequirementsTechnologyScalingHigh AvailabilityN + 1 ConfigurationsN + 2 ConfigurationsCostsB.3 The Dot-Bomb Era (2000–2003)Availability RequirementsTechnologyHigh AvailabilityScalingData ScalingApplicabilityCostsB.4 The Second Web Era (2003–2010)Availability RequirementsTechnologyHigh AvailabilityScalingCostsB.5 The Cloud Computing Era (2010–present)Availability RequirementsCostsScaling and High AvailabilityTechnologyB.6 ConclusionExercises
Appendix C. Scaling Terminology and Concepts
C.1 Constant, Linear, and Exponential ScalingC.2 Big O NotationC.3 Limitations of Big O Notation
Appendix D. Templates and Examples
D.1 Design Document TemplateD.2 Design Document ExampleD.3 Sample Postmortem Template
Appendix E. Recommended Reading
DevOps:ITIL:Theory:Classic Google Papers:Classic Facebook Papers:Scalability:UNIX Internals:UNIX Systems Programming:Network Protocols:
Bibliography
Index

Content preview from Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2

Preface

About the Authors

Introduction

Part I Design: Building It

1 Designing in a Distributed World

1.1 Visibility at Scale

1.2 The Importance of Simplicity

1.3 Composition

1.3.1 Load Balancer with Multiple Backend Replicas

1.3.2 Server with Multiple Backends

1.3.3 Server Tree

1.4 Distributed State

1.5 The CAP Principle

1.5.1 Consistency

1.5.2 Availability

1.5.3 Partition Tolerance

1.6 Loosely Coupled Systems

1.7 Speed

1.8 Summary

Exercises

2 Designing for Operations

2.1 Operational Requirements

2.1.1 Configuration

2.1.2 Startup and Shutdown

2.1.3 Queue Draining

2.1.4 Software Upgrades

2.1.5 Backups and Restores

2.1.6 Redundancy

2.1.7 Replicated Databases

2.1.8 Hot Swaps

2.1.9 Toggles for Individual Features

2.1.10 Graceful Degradation ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9780133478549Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Practice of Cloud System Administration, The: DevOps and SRE Practices for Web Services, Volume 2

by Thomas A. Limoncelli, Strata R. Chalup, Christina J. Hogan

Contents

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.