book

Mastering Distributed Tracing

by Yuri Shkuro

February 2019

Intermediate to advanced

444 pages

11h 36m

English

Packt Publishing

Read now

Unlock full access

Why subscribe?
About the author
Who this book is for

Download the example code filesDownload the color imagesConventions used
Reviews
Microservices and cloud-native applications
MetricsLogs
PrerequisitesRunning from prepackaged binariesRunning from Docker imagesRunning from the source codeGo language development environmentJaeger source code
The idea
Black-box inferenceSchema-basedMetadata propagation
Inter-request causality
Event modelSpan model
PrerequisitesProject source codeGo development environmentJava development environmentPython development environmentMySQL databaseQuery tools (curl or wget)Tracing backend (Jaeger)
Hello application in GoHello application in JavaHello application in PythonExercise summary
Step 1 – create a tracer instanceCreate a tracer in GoCreate a tracer in JavaCreate a tracer in PythonStep 2 – start a spanStart a span in GoStart a span in JavaStart a span in PythonStep 3 – annotate the spanAnnotate the span in GoAnnotate the span in JavaAnnotate the span in PythonExercise summary
Step 1 – trace individual functionsTrace individual functions in GoTrace individual functions in JavaTrace individual functions in PythonStep 2 – combine multiple spans into a single traceCombine multiple spans into a single trace in GoCombine multiple spans into a single trace in JavaCombine multiple spans into a single trace in PythonStep 3 – propagate the in-process contextIn-process context propagation in PythonIn-process context propagation in JavaIn-process context propagation in GoExercise summary
Step 1 – break up the monolithMicroservices in GoMicroservices in JavaMicroservices in PythonStep 2 – pass the context between processesPassing context between processes in GoPassing context between processes in JavaPassing context between processes in PythonStep 3 – apply OpenTracing-recommended tagsStandard tags in GoStandard tags in JavaStandard tags in PythonExercise summary
Using baggage in GoUsing baggage in JavaUsing baggage in PythonExercise summary
Open source instrumentation in GoAuto-instrumentation in JavaAuto-instrumentation in Python
PrerequisitesProject source codeJava development environmentKafka, Zookeeper, Redis, and Jaeger
ImplementationThe lib moduleAppIdMessageKafkaConfig and KafkaServiceRedisConfig and RedisServiceGiphyServiceThe chat-api serviceThe storage-service microserviceThe giphy-service microserviceRunning the applicationObserving traces
Spring instrumentationTracer resolverRedis instrumentationKafka instrumentationProducing messagesConsuming messages
Styles of instrumentation
Tracing systemsZipkin and OpenZipkinJaegerSkyWalkingX-Ray, Stackdriver, and moreStandards projectsW3C Trace ContextW3C "Data Interchange Format"OpenCensusOpenTracing
Service meshes
Project source codeJava development environmentKubernetesIstio
Head-based consistent samplingProbabilistic samplingRate limiting samplingGuaranteed-throughput probabilistic samplingAdaptive samplingLocal adaptive samplingGlobal adaptive samplingGoalsTheoryArchitectureCalculating sampling probabilityImplications of adaptive samplingExtensionsContext-sensitive samplingAd-hoc or debug samplingHow to deal with oversamplingPost-collection down-samplingThrottling
Tracing as a knowledge base
Deep, path-aware service graphsDetecting architectural problems
Critical path analysisRecognizing trace patternsLook for error markersLook for the longest span on the critical pathLook out for missing detailsAvoid sequential execution or "staircase"Be wary when things finish at exactly the same timeExemplarsLatency histograms
Brown Tracing Plane
Testing in productionDebugging in productionDeveloping in production
Three pillars of observability
Project source codeJava development environmentRunning the servers in DockerDeclaring index pattern in KibanaRunning the clients
Standard metrics via tracing instrumentationAdding context to metricsContext-aware metrics APIs
Structured loggingCorrelating logs with trace contextContext-aware logging APIsCapturing logs in the tracing systemDo we need separate logging and tracing backends?
Feature extraction
Tracing backendTrace completion triggerFeature extractorAggregator
Project source codeRunning the servers in DockerDefining index mapping in ElasticsearchJava development environmentMicroservices simulatorRunning as a Docker imageRunning from sourceVerifyDefine an index pattern in Kibana
Trace completion triggerFeature extractor
Beware of extrapolations
Why is it hard to deploy tracing instrumentation?
Standard frameworksIn-house adapter librariesTracing enabled by defaultMonoreposIntegration with existing infrastructure
Explaining the valueIntegrating with developer workflows
Why host your own?Customizations and integrationsBandwidth costOwn the data
Basic architecture: agent + collector + query serviceClientAgentCollectorQuery service and UIData mining jobsStreaming architectureMulti-tenancyCost accountingComplete isolationGranular access controlsSecurityRunning in multiple DCsCapturing origin zoneCross-zone federation
Over-samplingDebug tracesTraffic spikes due to DC failoverPerpetual tracesVery long traces
References

Overview

Mastering Distributed Tracing is an essential guide for developers and system architects looking to harness distributed tracing technology in microservices and large-scale distributed systems. Through real-world examples and clear explanations, this book teaches you to effectively use tracing tools to analyze and optimize complex systems.

What this Book will help me do

Gain a strong understanding of distributed tracing concepts and architecture.
Navigate and leverage open standards like OpenTracing and W3C Trace Context.
Learn how to implement code instrumentation in various scenarios.
Operate and maintain distributed tracing systems in production.
Use tracing insights to improve the performance of distributed and microservice architectures.

Author(s)

Yuri Shkuro, a visionary in the distributed tracing field and creator of the popular open-source tool Jaeger, combines theoretical insights and practical application in this book. With extensive industry experience, Yuri simplifies complex concepts to make distributed tracing accessible for professionals.

Who is it for?

This book is perfect for developers, architects, and system administrators working with microservices and large-scale distributed systems. Ideal for professionals wanting to build performant systems and those seeking to enhance their understanding of distributed tracing principles and tools.