BPF Performance Tools: Linux System and Application Observability

Book Description

BPF and related observability tools give software professionals unprecedented visibility into software, helping them analyze operating system and application performance, troubleshoot code, and strengthen security. BPF Performance Tools: Linux System and Application Observability is the industry’s most comprehensive guide to using these tools for observability. Brendan Gregg, author of the industry’s definitive guide to system performance, introduces powerful new methods and tools for doing analysis that leads to more robust, reliable, and safer code.

This authoritative guide:

  • Explores a wide spectrum of software and hardware targets

  • Thoroughly covers open source BPF tools from the Linux Foundation iovisor project’s bcc and bpftrace repositories

  • Summarizes performance engineering and kernel internals you need to understand

  • Provides and discusses 150+ bpftrace tools, including 80 written specifically for this book: tools you can run as-is, without programming — or customize and develop further, using diverse interfaces and the bpftrace front-end

You’ll learn how to use BPF (eBPF) tracing tools to analyze CPUs, memory, disks, file systems, networking, languages, applications, containers, hypervisors, security, and the Linux kernel. You’ll move from basic to advanced tools and techniques, producing new metrics, stack traces, custom latency histograms, and more. It’s like having a superpower: with Gregg’s guidance and tools, you can analyze virtually everything that impacts system performance, so you can improve virtually any Linux operating system or application.

Table of Contents

  1. Cover Page
  2. About This eBook
  3. Half Title Page
  4. Title Page
  5. Copyright Page
  6. Contents at a Glance
  7. Contents
  8. Foreword
  9. Preface
    1. Why Do You Need BPF Performance Tools?
    2. About This Book
    3. New Tools
    4. About GUIs
    5. About Linux Versions
    6. What This Book Does Not Cover
    7. How This Book Is Structured
    8. Intended Audience
    9. Source Code Copyright
    10. Supplemental Material and References
    11. Conventions Used in This Book
  10. Acknowledgments
  11. About the Author
  12. Part I: Technologies
    1. Chapter 1. Introduction
      1. 1.1 What Are BPF and eBPF?
      2. 1.2 What Are Tracing, Snooping, Sampling, Profiling, and Observability?
      3. 1.3 What Are BCC, bpftrace, and IO Visor?
      4. 1.4 A First Look at BCC: Quick Wins
      5. 1.5 BPF Tracing Visibility
      6. 1.6 Dynamic Instrumentation: kprobes and uprobes
      7. 1.7 Static Instrumentation: Tracepoints and USDT
      8. 1.8 A First Look at bpftrace: Tracing open()
      9. 1.9 Back to BCC: Tracing open()
      10. 1.10 Summary
    2. Chapter 2. Technology Background
      1. 2.1 BPF Illustrated
      2. 2.2 BPF
      3. 2.3 Extended BPF (eBPF)
      4. 2.4 Stack Trace Walking
      5. 2.5 Flame Graphs
      6. 2.6 Event Sources
      7. 2.7 kprobes
      8. 2.8 uprobes
      9. 2.9 Tracepoints
      10. 2.10 USDT
      11. 2.11 Dynamic USDT
      12. 2.12 PMCs
      13. 2.13 perf_events
      14. 2.14 Summary
    3. Chapter 3. Performance Analysis
      1. 3.1 Overview
      2. 3.2 Performance Methodologies
      3. 3.3 Linux 60-Second Analysis
      4. 3.4 BCC Tool Checklist
      5. 3.5 Summary
    4. Chapter 4. BCC
      1. 4.1 BCC Components
      2. 4.2 BCC Features
      3. 4.3 BCC Installation
      4. 4.4 BCC Tools
      5. 4.5 funccount
      6. 4.6 stackcount
      7. 4.7 trace
      8. 4.8 argdist
      9. 4.9 Tool Documentation
      10. 4.10 Developing BCC Tools
      11. 4.11 BCC Internals
      12. 4.12 BCC Debugging
      13. 4.13 Summary
    5. Chapter 5. bpftrace
      1. 5.1 bpftrace Components
      2. 5.2 bpftrace Features
      3. 5.3 bpftrace Installation
      4. 5.4 bpftrace Tools
      5. 5.5 bpftrace One-Liners
      6. 5.6 bpftrace Documentation
      7. 5.7 bpftrace Programming
      8. 5.8 bpftrace Usage
      9. 5.9 bpftrace Probe Types
      10. 5.10 bpftrace Flow Control
      11. 5.11 bpftrace Operators
      12. 5.12 bpftrace Variables
      13. 5.13 bpftrace Functions
      14. 5.14 bpftrace Map Functions
      15. 5.15 bpftrace Future Work
      16. 5.16 bpftrace Internals
      17. 5.17 bpftrace Debugging
      18. 5.18 Summary
  13. Part II: Using BPF Tools
    1. Chapter 6. CPUs
      1. 6.1 Background
      2. 6.2 Traditional Tools
      3. 6.3 BPF Tools
      4. 6.4 BPF One-Liners
      5. 6.5 Optional Exercises
      6. 6.6 Summary
    2. Chapter 7. Memory
      1. 7.1 Background
      2. 7.2 Traditional Tools
      3. 7.3 BPF Tools
      4. 7.4 BPF One-Liners
      5. 7.5 Optional Exercises
      6. 7.6 Summary
    3. Chapter 8. File Systems
      1. 8.1 Background
      2. 8.2 Traditional Tools
      3. 8.3 BPF Tools
      4. 8.4 BPF One-Liners
      5. 8.5 Optional Exercises
      6. 8.6 Summary
    4. Chapter 9. Disk I/O
      1. 9.1 Background
      2. 9.2 Traditional Tools
      3. 9.3 BPF Tools
      4. 9.4 BPF One-Liners
      5. 9.5 Optional Exercises
      6. 9.6 Summary
    5. Chapter 10. Networking
      1. 10.1 Background
      2. 10.2 Traditional Tools
      3. 10.3 BPF Tools
      4. 10.4 BPF One-Liners
      5. 10.5 Optional Exercises
      6. 10.6 Summary
    6. Chapter 11. Security
      1. 11.1 Background
      2. 11.2 BPF Tools
      3. 11.3 BPF One-Liners
      4. 11.4 Summary
    7. Chapter 12. Languages
      1. 12.1 Background
      2. 12.2 C
      3. 12.3 Java
      4. 12.4 Bash Shell
      5. 12.5 Other Languages
      6. 12.6 Summary
    8. Chapter 13. Applications
      1. 13.1 Background
      2. 13.2 BPF Tools
      3. 13.3 BPF One-Liners
      4. 13.4 BPF One-Liners Examples
      5. 13.5 Summary
    9. Chapter 14. Kernel
      1. 14.1 Background
      2. 14.2 Strategy
      3. 14.3 Traditional Tools
      4. 14.4 BPF Tools
      5. 14.5 BPF One-Liners
      6. 14.6 BPF One-Liners Examples
      7. 14.7 Challenges
      8. 14.8 Summary
    10. Chapter 15. Containers
      1. 15.1 Background
      2. 15.2 Traditional Tools
      3. 15.3 BPF Tools
      4. 15.4 BPF One-Liners
      5. 15.5 Optional Exercises
      6. 15.6 Summary
    11. Chapter 16. Hypervisors
      1. 16.1 Background
      2. 16.2 Traditional Tools
      3. 16.3 Guest BPF Tools
      4. 16.4 Host BPF Tools
      5. 16.5 Summary
  14. Part III: Additional Topics
    1. Chapter 17. Other BPF Performance Tools
      1. 17.1 Vector and Performance Co-Pilot (PCP)
      2. 17.2 Grafana and Performance Co-Pilot (PCP)
      3. 17.3 Cloudflare eBPF Prometheus Exporter (with Grafana)
      4. 17.4 kubectl-trace
      5. 17.5 Other Tools
      6. 17.6 Summary
    2. Chapter 18. Tips, Tricks, and Common Problems
      1. 18.1 Typical Event Frequency and Overhead
      2. 18.2 Sample at 49 or 99 Hertz
      3. 18.3 Yellow Pigs and Gray Rats
      4. 18.4 Write Target Software
      5. 18.5 Learn Syscalls
      6. 18.6 Keep It Simple
      7. 18.7 Missing Events
      8. 18.8 Missing Stacks Traces
      9. 18.9 Missing Symbols (Function Names) When Printing
      10. 18.10 Missing Functions When Tracing
      11. 18.11 Feedback Loops
      12. 18.12 Dropped Events
  15. Part IV: Appendixes
    1. Appendix A. bpftrace One-Liners
      1. Chapter 6 CPUs
      2. Chapter 7 Memory
      3. Chapter 8 File Systems
      4. Chapter 9 Disk I/O
      5. Chapter 10 Networking
      6. Chapter 11 Security
      7. Chapter 13 Applications
      8. Chapter 14 Kernel
    2. Appendix B. bpftrace Cheat Sheet
      1. Synopsis
      2. Probes
      3. Probe Aliases
      4. Vars
      5. Actions
      6. Asynchronous Actions
      7. Switches
    3. Appendix C. BCC Tool Development
      1. Resources
      2. Five Tips
      3. Tool Examples
      4. More Info
    4. Appendix D. C BPF
      1. Why Program in C?
      2. Five Tips
      3. C Programs
      4. perf C
      5. More Info
    5. Appendix E. BPF Instructions
      1. Helper Macros
      2. Instructions
      3. Encoding
      4. References
  16. Glossary
  17. Bibliography
  18. Index
  19. Code Snippets

Product Information

  • Title: BPF Performance Tools: Linux System and Application Observability
  • Author(s): Brendan Gregg
  • Release date: December 2019
  • Publisher(s): Addison-Wesley Professional
  • ISBN: 9780136588870