book

Distributed Tracing in Practice

by Austin Parker, Daniel Spoonhower, Jonathan Mace, Ben Sigelman, Rebecca Isaacs

April 2020

Intermediate to advanced

327 pages

10h

English

O'Reilly Media, Inc.

Read now

Unlock full access

Distributed Architectures and YouDeep SystemsThe Difficulties of Understanding Distributed ArchitecturesHow Does Distributed Tracing Help?Distributed Tracing and YouConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
The Pieces of a Distributed Tracing DeploymentDistributed Tracing, Microservices, Serverless, Oh My!The Benefits of TracingSetting the Table
White Box Versus Black BoxApplication Versus SystemAgents Versus LibrariesPropagating ContextInterprocess PropagationIntraprocess PropagationThe Shape of Distributed TracingTracing-Friendly Microservices and ServerlessTracing in a MonolithTracing in Web and Mobile Clients
The Importance of Abstract InstrumentationOpenTelemetryOpenTracing and OpenCensusOpenTracingOpenCensusOther Notable Formats and ProjectsX-RayZipkinInteroperability and Migration StrategiesWhy Use Open Source Instrumentation?InteroperabilityPortabilityEcosystem and Implicit Visibility
Tracing by ExampleInstalling the Sample ApplicationAdding Basic Distributed TracingCustom InstrumentationWhere to Start—Nodes and EdgesFramework InstrumentationService Mesh InstrumentationCreating Your Service GraphWhat’s in a Span?Effective NamingEffective TaggingEffective LoggingUnderstanding Performance ConsiderationsTrace-Driven DevelopmentDeveloping with TracesTesting with TracesCreating an Instrumentation PlanMaking the Case for InstrumentationInstrumentation Quality ChecklistKnowing When to Stop InstrumentingSmart and Sustainable Instrumentation Growth
Organizational AdoptionStart Close to Your UsersStart Centrally: Load Balancers and GatewaysLeverage Infrastructure: RPC Frameworks and Service MeshesMake Adoption RepeatableTracer ArchitectureIn-Process LibrariesSidecars and AgentsCollectorsCentralized Storage and AnalysisIncremental DeploymentData Provenance, Security, and FederationFrontend Service TelemetryServer-Side Telemetry for Managed Services
Application OverheadLatencyThroughputInfrastructure CostsNetworkStorageSamplingMinimum RequirementsStrategiesSelecting TracesOff-the-Shelf ETL Solutions
The Three Pillars DefinedMetricsLoggingDistributed TracingFatal Flaws of the Three PillarsDesign GoalsAssessing the Three PillarsThree Pipes (Not Pillars)Observability Goals and ActivitiesTwo Goals in ObservabilityTwo Fundamental Activities in ObservabilityA New ScorecardThe Path Ahead
Measuring PerformancePercentilesHistogramsDefining the Critical PathApproaches to Improving PerformanceIndividual TracesBiased Sampling and Trace ComparisonTrace SearchMultimodal AnalysisAggregate AnalysisCorrelation Analysis

Defining the ProblemHuman Factors(Avoiding) Finger-Pointing“Suppressing” the MessengerIncident Hand-offGood PostmortemsApproaches to Restoring PerformanceIntegration with Alerting WorkflowsIndividual TracesBiased SamplingReal-Time ResponseKnowing What’s NormalAggregate and Correlation Root Cause Analysis
Distributed Tracing: A History of PragmatismRequest-Based SystemsResponse Time MattersRequest-Oriented InformationNotable WorkPinpointMagpieX-TraceDapperWhere to Next?
The Value of Traces in AggregateExample 1: Is Network Congestion Affecting My Application?Example 2: What Services Are Required to Serve an API Endpoint?Organizing the DataA Strawperson SolutionWhat About the Trade-offs?Sampling for Aggregate AnalysisThe Processing PipelineIncorporating Heterogeneous DataCustom FunctionsJoining with Other Data SourcesRecap and Case StudyThe Value of Traces in AggregateOrganizing the DataSampling for Aggregate AnalysisThe Processing PipelineIncorporating Heterogeneous Data
Why Spans Have PrevailedVisibilityPragmatismPortabilityCompatibilityFlexibilityWhy Spans Aren’t EnoughGraphs, Not TreesInter-Request DependenciesDecoupled DependenciesDistributed DataflowMachine LearningLow-Level Performance MetricsNew AbstractionsSeeing Causality
Limitations of Distributed TracingChallenge 1: Anticipating ProblemsChallenge 2: Completeness Versus CostsChallenge 3: Open-Ended Use CasesOther Tools Like Distributed TracingCensusA Motivating ExampleA Distributed Tracing Solution?Tag Propagation and Local Metric AggregationComparison to Distributed TracingPivot TracingDynamic InstrumentationRecurring ProblemsHow Does It Work?Dynamic ContextComparison to Distributed TracingPythiaPerformance RegressionsDesignOverheadsComparison to Distributed Tracing
Cross-Cutting ToolsUse CasesDistributed TracingCross-Component MetricsCross-Component Resource ManagementManaging Data Quality Trade-offsFailure Testing of MicroservicesEnforcing Cross-System ConsistencyRequest DuplicationRecord Lineage in Stream Processing SystemsAuditing Security PoliciesTesting in ProductionCommon ThemesShould You Care?The Tracing PlaneIs Baggage Enough?Beyond Key-Value PairsCompiling BDLBaggageContextMergingOverheads
Open Source Tracers and Trace AnalysisCommercial Tracers and Trace AnalyzersLanguage-Specific Tracing FeaturesJava and C#Go, Rust, and C++Python, JavaScript, and Other Dynamic Languages
Why a Separate Context Model?The OpenTelemetry Context ModelW3C CorrelationContext and the Correlations APIDistributed and Local ContextExamples and Potential Applications

Content preview from Distributed Tracing in Practice

Appendix A. The State of Distributed Tracing Circa 2020

One of the biggest challenges in writing this book has been figuring out what merits inclusion. We’ve attempted to provide a broad, and somewhat timeless, overview of the fundamental concepts and techniques underpinning the technology as it exists today. For everything we touched on, however, there’s more that we left out. This appendix will attempt to give you a snapshot of popular tools for distributed tracing, both open source and commercial, that you might find useful.

Open Source Tracers and Trace Analysis

These are some popular open source tracers and trace analyzers:

Zipkin: One of the most mature open source tracing tools available today. We discuss it in more depth in “Zipkin”. It provides, in addition to a trace analyzer, a number of instrumentation libraries for popular frameworks such as ASP.NET Core, a variety of Java frameworks (such as Jersey, JAXRS2, Apache HttpClient, and Spring Cloud), and more.
Jaeger: A newer open source tracing tool and a Cloud Native Computing Foundation (CNCF) project. The more lightweight nature of Jaeger has made it extremely popular for cloud-native developers, and it is the default trace analyzer for the popular service mesh Istio. Jaeger also fully supports the OpenTracing API, making it compatible with hundreds of integrations into existing frameworks and libraries.
SkyWalking: An Apache Foundation project that has found a great deal of popularity in China. Its goal is to ...