book

Distributed Tracing in Practice

by Austin Parker, Daniel Spoonhower, Jonathan Mace, Ben Sigelman, Rebecca Isaacs

April 2020

Intermediate to advanced

327 pages

10h

English

O'Reilly Media, Inc.

Read now

Unlock full access

Distributed Architectures and YouDeep SystemsThe Difficulties of Understanding Distributed ArchitecturesHow Does Distributed Tracing Help?Distributed Tracing and YouConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
The Pieces of a Distributed Tracing DeploymentDistributed Tracing, Microservices, Serverless, Oh My!The Benefits of TracingSetting the Table
White Box Versus Black BoxApplication Versus SystemAgents Versus LibrariesPropagating ContextInterprocess PropagationIntraprocess PropagationThe Shape of Distributed TracingTracing-Friendly Microservices and ServerlessTracing in a MonolithTracing in Web and Mobile Clients
The Importance of Abstract InstrumentationOpenTelemetryOpenTracing and OpenCensusOpenTracingOpenCensusOther Notable Formats and ProjectsX-RayZipkinInteroperability and Migration StrategiesWhy Use Open Source Instrumentation?InteroperabilityPortabilityEcosystem and Implicit Visibility
Tracing by ExampleInstalling the Sample ApplicationAdding Basic Distributed TracingCustom InstrumentationWhere to Start—Nodes and EdgesFramework InstrumentationService Mesh InstrumentationCreating Your Service GraphWhat’s in a Span?Effective NamingEffective TaggingEffective LoggingUnderstanding Performance ConsiderationsTrace-Driven DevelopmentDeveloping with TracesTesting with TracesCreating an Instrumentation PlanMaking the Case for InstrumentationInstrumentation Quality ChecklistKnowing When to Stop InstrumentingSmart and Sustainable Instrumentation Growth
Organizational AdoptionStart Close to Your UsersStart Centrally: Load Balancers and GatewaysLeverage Infrastructure: RPC Frameworks and Service MeshesMake Adoption RepeatableTracer ArchitectureIn-Process LibrariesSidecars and AgentsCollectorsCentralized Storage and AnalysisIncremental DeploymentData Provenance, Security, and FederationFrontend Service TelemetryServer-Side Telemetry for Managed Services
Application OverheadLatencyThroughputInfrastructure CostsNetworkStorageSamplingMinimum RequirementsStrategiesSelecting TracesOff-the-Shelf ETL Solutions
The Three Pillars DefinedMetricsLoggingDistributed TracingFatal Flaws of the Three PillarsDesign GoalsAssessing the Three PillarsThree Pipes (Not Pillars)Observability Goals and ActivitiesTwo Goals in ObservabilityTwo Fundamental Activities in ObservabilityA New ScorecardThe Path Ahead
Measuring PerformancePercentilesHistogramsDefining the Critical PathApproaches to Improving PerformanceIndividual TracesBiased Sampling and Trace ComparisonTrace SearchMultimodal AnalysisAggregate AnalysisCorrelation Analysis

Defining the ProblemHuman Factors(Avoiding) Finger-Pointing“Suppressing” the MessengerIncident Hand-offGood PostmortemsApproaches to Restoring PerformanceIntegration with Alerting WorkflowsIndividual TracesBiased SamplingReal-Time ResponseKnowing What’s NormalAggregate and Correlation Root Cause Analysis
Distributed Tracing: A History of PragmatismRequest-Based SystemsResponse Time MattersRequest-Oriented InformationNotable WorkPinpointMagpieX-TraceDapperWhere to Next?
The Value of Traces in AggregateExample 1: Is Network Congestion Affecting My Application?Example 2: What Services Are Required to Serve an API Endpoint?Organizing the DataA Strawperson SolutionWhat About the Trade-offs?Sampling for Aggregate AnalysisThe Processing PipelineIncorporating Heterogeneous DataCustom FunctionsJoining with Other Data SourcesRecap and Case StudyThe Value of Traces in AggregateOrganizing the DataSampling for Aggregate AnalysisThe Processing PipelineIncorporating Heterogeneous Data
Why Spans Have PrevailedVisibilityPragmatismPortabilityCompatibilityFlexibilityWhy Spans Aren’t EnoughGraphs, Not TreesInter-Request DependenciesDecoupled DependenciesDistributed DataflowMachine LearningLow-Level Performance MetricsNew AbstractionsSeeing Causality
Limitations of Distributed TracingChallenge 1: Anticipating ProblemsChallenge 2: Completeness Versus CostsChallenge 3: Open-Ended Use CasesOther Tools Like Distributed TracingCensusA Motivating ExampleA Distributed Tracing Solution?Tag Propagation and Local Metric AggregationComparison to Distributed TracingPivot TracingDynamic InstrumentationRecurring ProblemsHow Does It Work?Dynamic ContextComparison to Distributed TracingPythiaPerformance RegressionsDesignOverheadsComparison to Distributed Tracing
Cross-Cutting ToolsUse CasesDistributed TracingCross-Component MetricsCross-Component Resource ManagementManaging Data Quality Trade-offsFailure Testing of MicroservicesEnforcing Cross-System ConsistencyRequest DuplicationRecord Lineage in Stream Processing SystemsAuditing Security PoliciesTesting in ProductionCommon ThemesShould You Care?The Tracing PlaneIs Baggage Enough?Beyond Key-Value PairsCompiling BDLBaggageContextMergingOverheads
Open Source Tracers and Trace AnalysisCommercial Tracers and Trace AnalyzersLanguage-Specific Tracing FeaturesJava and C#Go, Rust, and C++Python, JavaScript, and Other Dynamic Languages
Why a Separate Context Model?The OpenTelemetry Context ModelW3C CorrelationContext and the Correlations APIDistributed and Local ContextExamples and Potential Applications

Content preview from Distributed Tracing in Practice

Bibliography

[Abr13] Abraham, Lior, John Allen, Oleksandr Barykin, Vinayak Borkar, Bhuwan Chopra, Ciprian Gerea, Daniel Merl, Josh Metzler, David Reiss, Subbu Subramanian, Janet L. Wiener, and Okay Zed. 2013. “Scuba: Diving into Data at Facebook.” Facebook paper. https://research.fb.com/wp-content/uploads/2016/11/scuba-diving-into-data-at-facebook.pdf.
[Agu03] Aguilera, Marcos K., Jeffrey C. Mogul, Janet L. Wiener, Patrick Reynolds, and Athicha Muthitacharoen. 2003. “Performance Debugging for Distributed Systems of Black Boxes.” ACM SIGOPS Operating Systems Review 37(5): 74-89. https://pdos.csail.mit.edu/~athicha/papers/blackboxes:sosp03.pdf.
[Aka17] Akamai. 2017. “Akamai Online Retail Performance Report: Milliseconds Are Critical.” Press release. April 19, 2017. https://www.akamai.com/uk/en/about/news/press/2017-press/akamai-releases-spring-2017-state-of-online-retail-performance-report.jsp.
[Ate19] Ates, Emre, Lily Sturmann, Mert Toslali, Orran Krieger, Richard Megginson, Ayse K. Coskun, and Raja R. Sambasivan. 2019. “An Automated, Cross-Layer Instrumentation Framework for Diagnosing Performance Problems in Distributed Applications.” SoCC ’19: Proceedings of the ACM Symposium on Cloud Computing (November 2019): 165–70. https://doi.org/10.1145/3357223.3362704.
[Bar04] Barham, Paul, Austin Donnelly, Rebecca Isaacs, and Richard Mortier. 2004. “Using Magpie for Request Extraction and Workload Modelling.” Proceedings of the 1st ACM SIGOPS European Workshop, Leuven, Belgium, ...