book

Linux Observability with BPF

by David Calavera, Lorenzo Fontana

November 2019

Intermediate to advanced

177 pages

4h 39m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Conventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
BPF’s HistoryArchitectureConclusion
Writing BPF ProgramsBPF Program TypesSocket Filter ProgramsKprobe ProgramsTracepoint ProgramsXDP ProgramsPerf Event ProgramsCgroup Socket ProgramsCgroup Open Socket ProgramsSocket Option ProgramsSocket Map ProgramsCgroup Device ProgramsSocket Message Delivery ProgramsRaw Tracepoint ProgramsCgroup Socket Address ProgramsSocket Reuseport ProgramsFlow Dissection ProgramsOther BPF ProgramsThe BPF VerifierBPF Type FormatBPF Tail CallsConclusion
Creating BPF MapsELF Conventions to Create BPF MapsWorking with BFP MapsUpdating Elements in a BPF MapReading Elements from a BPF MapRemoving an Element from a BPF MapIterating Over Elements in a BPF MapLooking Up and Deleting ElementsConcurrent Access to Map ElementsTypes of BPF MapsHash-Table MapsArray MapsProgram Array MapsPerf Events Array MapsPer-CPU Hash MapsPer-CPU Array MapsStack Trace MapsCgroup Array MapsLRU Hash and Per-CPU Hash MapsLPM Trie MapsArray of Maps and Hash of MapsDevice Map MapsCPU Map MapsOpen Socket MapsSocket Array and Hash MapsCgroup Storage and Per-CPU Storage MapsReuseport Socket MapsQueue MapsStack MapsThe BPF Virtual FilesystemConclusion
ProbesKernel ProbesTracepointsUser-Space ProbesUser Statically Defined TracepointsVisualizing Tracing DataFlame GraphsHistogramsPerf EventsConclusion
BPFToolInstallationFeature DisplayInspecting BPF ProgramsInspecting BPF MapsInspecting Programs Attached to Specific InterfacesLoading Commands in Batch ModeDisplaying BTF InformationBPFTraceInstallationLanguage ReferenceFilteringDynamic Mappingkubectl-traceInstallationInspecting Kubernetes NodeseBPF ExporterInstallationExporting Metrics from BPFConclusion
BPF and Packet Filteringtcpdump and BPF ExpressionsPacket Filtering for Raw SocketsBPF-Based Traffic Control ClassifierTerminologyTraffic Control Classifier Program Using cls_bpfDifferences Between Traffic Control and XDPConclusion
XDP Programs OverviewOperation ModesThe Packet ProcessorXDP and iproute2 as a LoaderXDP and BCCTesting XDP ProgramsXDP Testing Using the Python Unit Testing FrameworkXDP Use CasesMonitoringDDoS MitigationLoad BalancingFirewallingConclusion
CapabilitiesSeccompSeccomp ErrorsSeccomp BPF Filter ExampleBPF LSM HooksConclusion

Sysdig eBPF God ModeFlowmill

Content preview from Linux Observability with BPF

Preface

In 2015, David was working as a core developer for Docker, the company that made containers popular. His day-to-day work was divided between helping the community and growing the project. Part of his job was reviewing the firehose of pull requests that members of the community sent us; he also had to ensure that Docker worked for all kinds of scenarios, including high-performance workloads that were running and provisioning thousands of containers at any point of time.

To diagnose performance issues at Docker, we used flame graphs, which are advanced visualizations to help you navigate that data easily. The Go programming language makes it really easy to measure and extract application performance data using an embedded HTTP endpoint and generate graphs based on that data. David wrote an article about Go’s profiler capabilities and how you can use its data to generate flame graphs. A big pitfall about the way that Docker collects performance data is that the profiler is disabled by default, so if you’re trying to debug a performance issue, the first action to take is to restart Docker. The main issue with this strategy is that by restarting the service, you’ll probably lose the relevant data that you’re trying to collect, and then you need to wait until the event you’re trying to trace happens again. In David’s article about Docker flame graphs, he mentioned this as a necessary step to measure Docker’s performance, but that it didn’t need to be this way. This realization ...