book

Observability Engineering

by Charity Majors, Liz Fong-Jones, George Miranda

May 2022

Intermediate to advanced

318 pages

9h 15m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who This Book Is ForWhy We Wrote This BookWhat You Will LearnConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
The Mathematical Definition of ObservabilityApplying Observability to Software SystemsMischaracterizations About Observability for SoftwareWhy Observability Matters NowIs This Really the Best Way?Why Are Metrics and Monitoring Not Enough?Debugging with Metrics Versus ObservabilityThe Role of CardinalityThe Role of DimensionalityDebugging with ObservabilityObservability Is for Modern SystemsConclusion
How Monitoring Data Is Used for DebuggingTroubleshooting Behaviors When Using DashboardsThe Limitations of Troubleshooting by IntuitionTraditional Monitoring Is Fundamentally ReactiveHow Observability Enables Better DebuggingConclusion
An Introduction to ParseScaling at ParseThe Evolution Toward Modern SystemsThe Evolution Toward Modern PracticesShifting Practices at ParseConclusion
Cloud Native, DevOps, and SRE in a NutshellObservability: Debugging Then Versus NowObservability Empowers DevOps and SRE PracticesConclusion
Debugging with Structured EventsThe Limitations of Metrics as a Building BlockThe Limitations of Traditional Logs as a Building BlockUnstructured LogsStructured LogsProperties of Events That Are Useful in DebuggingConclusion
Distributed Tracing and Why It Matters NowThe Components of TracingInstrumenting a Trace the Hard WayAdding Custom Fields into Trace SpansStitching Events into TracesConclusion

A Brief Introduction to InstrumentationOpen Instrumentation StandardsInstrumentation Using Code-Based ExamplesStart with Automatic InstrumentationAdd Custom InstrumentationSend Instrumentation Data to a Backend SystemConclusion
Debugging from Known ConditionsDebugging from First PrinciplesUsing the Core Analysis LoopAutomating the Brute-Force Portion of the Core Analysis LoopThis Misleading Promise of AIOpsConclusion
Where Monitoring FitsWhere Observability FitsSystem Versus Software ConsiderationsAssessing Your Organizational NeedsExceptions: Infrastructure Monitoring That Can’t Be IgnoredReal-World ExamplesConclusion
Join a Community GroupStart with the Biggest Pain PointsBuy Instead of BuildFlesh Out Your Instrumentation IterativelyLook for Opportunities to Leverage Existing EffortsPrepare for the Hardest Last PushConclusion
Test-Driven DevelopmentObservability in the Development CycleDetermining Where to DebugDebugging in the Time of MicroservicesHow Instrumentation Drives ObservabilityShifting Observability LeftUsing Observability to Speed Up Software DeliveryConclusion
Traditional Monitoring Approaches Create Dangerous Alert FatigueThreshold Alerting Is for Known-Unknowns OnlyUser Experience Is a North StarWhat Is a Service-Level Objective?Reliable Alerting with SLOsChanging Culture Toward SLO-Based Alerts: A Case StudyConclusion
Alerting Before Your Error Budget Is EmptyFraming Time as a Sliding WindowForecasting to Create a Predictive Burn AlertThe Lookahead WindowThe Baseline WindowActing on SLO Burn AlertsUsing Observability Data for SLOs Versus Time-Series DataConclusion
Why Slack Needed ObservabilityInstrumentation: Shared Client Libraries and DimensionsCase Studies: Operationalizing the Supply ChainUnderstanding Context Through ToolingEmbedding Actionable AlertingUnderstanding What ChangedConclusion
How to Analyze the ROI of ObservabilityThe Real Costs of Building Your OwnThe Hidden Costs of Using “Free” SoftwareThe Benefits of Building Your OwnThe Risks of Building Your OwnThe Real Costs of Buying SoftwareThe Hidden Financial Costs of Commercial SoftwareThe Hidden Nonfinancial Costs of Commercial SoftwareThe Benefits of Buying Commercial SoftwareThe Risks of Buying Commercial SoftwareBuy Versus Build Is Not a Binary ChoiceConclusion
The Functional Requirements for ObservabilityTime-Series Databases Are Inadequate for ObservabilityOther Possible Data StoresData Storage StrategiesCase Study: The Implementation of Honeycomb’s RetrieverPartitioning Data by TimeStoring Data by Column Within SegmentsPerforming Query WorkloadsQuerying for TracesQuerying Data in Real TimeMaking It Affordable with TieringMaking It Fast with ParallelismDealing with High CardinalityScaling and Durability StrategiesNotes on Building Your Own Efficient Data StoreConclusion
Sampling to Refine Your Data CollectionUsing Different Approaches to SamplingConstant-Probability SamplingSampling on Recent Traffic VolumeSampling Based on Event Content (Keys)Combining per Key and Historical MethodsChoosing Dynamic Sampling OptionsWhen to Make a Sampling Decision for TracesTranslating Sampling Strategies into CodeThe Base CaseFixed-Rate SamplingRecording the Sample RateConsistent SamplingTarget Rate SamplingHaving More Than One Static Sample RateSampling by Key and Target RateSampling with Dynamic Rates on Arbitrarily Many KeysPutting It All Together: Head and Tail per Key Target Rate SamplingConclusion
Attributes of Telemetry PipelinesRoutingSecurity and ComplianceWorkload IsolationData BufferingCapacity ManagementData Filtering and AugmentationData TransformationEnsuring Data Quality and ConsistencyManaging a Telemetry Pipeline: AnatomyChallenges When Managing a Telemetry PipelinePerformanceCorrectnessAvailabilityReliabilityIsolationData FreshnessUse Case: Telemetry Management at SlackMetrics AggregationLogs and Trace EventsOpen Source AlternativesManaging a Telemetry Pipeline: Build Versus BuyConclusion
The Reactive Approach to Introducing ChangeThe Return on Investment of ObservabilityThe Proactive Approach to Introducing ChangeIntroducing Observability as a PracticeUsing the Appropriate ToolsInstrumentationData Storage and AnalyticsRolling Out Tools to Your TeamsKnowing When You Have Enough ObservabilityConclusion
Recognizing Nonengineering Observability NeedsCreating Observability Allies in PracticeCustomer Support TeamsCustomer Success and Product TeamsSales and Executive TeamsUsing Observability Versus Business Intelligence ToolsQuery Execution TimeAccuracyRecencyStructureTime WindowsEphemeralityUsing Observability and BI Tools Together in PracticeConclusion
A Note About Maturity ModelsWhy Observability Needs a Maturity ModelAbout the Observability Maturity ModelCapabilities Referenced in the OMMRespond to System Failure with ResilienceDeliver High-Quality CodeManage Complexity and Technical DebtRelease on a Predictable CadenceUnderstand User BehaviorUsing the OMM for Your OrganizationConclusion
Observability, Then Versus NowAdditional ResourcesPredictions for Where Observability Is Going

Overview

Observability is critical for building, changing, and understanding the software that powers complex modern systems. Teams that adopt observability are much better equipped to ship code swiftly and confidently, identify outliers and aberrant behaviors, and understand the experience of each and every user. This practical book explains the value of observable systems and shows you how to practice observability-driven development.

Authors Charity Majors, Liz Fong-Jones, and George Miranda from Honeycomb explain what constitutes good observability, show you how to improve upon what you're doing today, and provide practical dos and don'ts for migrating from legacy tooling, such as metrics, monitoring, and log management. You'll also learn the impact observability has on organizational culture (and vice versa).

You'll explore:

How the concept of observability applies to managing software at scale
The value of practicing observability when delivering complex cloud native applications and systems
The impact observability has across the entire software development lifecycle
How and why different functional teams use observability with service-level objectives
How to instrument your code to help future engineers understand the code you wrote today
How to produce quality code for context-aware system debugging and maintenance
How data-rich analytics can help you debug elusive issues