BPF Performance Tools

BPF Performance Tools

by Brendan Gregg
Released December 2019
Publisher(s): Addison-Wesley Professional
ISBN: 9780136624523

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Use BPF Tools to Optimize Performance, Fix Problems, and See Inside Running Systems

BPF-based performance tools give you unprecedented visibility into systems and applications, so you can optimize performance, troubleshoot code, strengthen security, and reduce costs. BPF Performance Tools: Linux System and Application Observability is the definitive guide to using these tools for observability.

Pioneering BPF expert Brendan Gregg presents more than 150 ready-to-run analysis and debugging tools, expert guidance on applying them, and step-by-step tutorials on developing your own. You’ll learn how to analyze CPUs, memory, disks, file systems, networking, languages, applications, containers, hypervisors, security, and the kernel. Gregg guides you from basic to advanced tools, helping you generate deeper, more useful technical insights for improving virtually any Linux system or application.

  • Learn essential tracing concepts and both core BPF front-ends: BCC and bpftrace

  • Master 150+ powerful BPF tools, including dozens created just for this book, and available for download

  • Discover practical strategies, tips, and tricks for more effective analysis

  • Analyze compiled, JIT-compiled, and interpreted code in multiple languages: C, Java, bash shell, and more

  • Generate metrics, stack traces, and custom latency histograms

  • Use complementary tools when they offer quick, easy wins

  • Explore advanced tools built on BPF: PCP and Grafana for remote monitoring, eBPF Exporter, and kubectl-trace for tracing Kubernetes

  • Foreword by Alexei Starovoitov, creator of the new BPF

BPF Performance Tools will be an indispensable resource for all administrators, developers, support staff, and other IT professionals working with any recent Linux distribution in any enterprise or cloud environment.

Table of contents

  1. Cover Page
  2. About This eBook
  3. Half Title Page
  4. Title Page
  5. Copyright Page
  6. Contents at a Glance
  7. Contents
  8. Foreword
  9. Preface
    1. Why Do You Need BPF Performance Tools?
    2. About This Book
    3. New Tools
    4. About GUIs
    5. About Linux Versions
    6. What This Book Does Not Cover
    7. How This Book Is Structured
    8. Intended Audience
    9. Source Code Copyright
    10. Supplemental Material and References
    11. Conventions Used in This Book
  10. Acknowledgments
  11. About the Author
  12. Part I: Technologies
    1. Chapter 1. Introduction
      1. 1.1 What Are BPF and eBPF?
      2. 1.2 What Are Tracing, Snooping, Sampling, Profiling, and Observability?
      3. 1.3 What Are BCC, bpftrace, and IO Visor?
      4. 1.4 A First Look at BCC: Quick Wins
      5. 1.5 BPF Tracing Visibility
      6. 1.6 Dynamic Instrumentation: kprobes and uprobes
      7. 1.7 Static Instrumentation: Tracepoints and USDT
      8. 1.8 A First Look at bpftrace: Tracing open()
      9. 1.9 Back to BCC: Tracing open()
      10. 1.10 Summary
    2. Chapter 2. Technology Background
      1. 2.1 BPF Illustrated
      2. 2.2 BPF
      3. 2.3 Extended BPF (eBPF)
      4. 2.4 Stack Trace Walking
      5. 2.5 Flame Graphs
      6. 2.6 Event Sources
      7. 2.7 kprobes
      8. 2.8 uprobes
      9. 2.9 Tracepoints
      10. 2.10 USDT
      11. 2.11 Dynamic USDT
      12. 2.12 PMCs
      13. 2.13 perf_events
      14. 2.14 Summary
    3. Chapter 3. Performance Analysis
      1. 3.1 Overview
      2. 3.2 Performance Methodologies
      3. 3.3 Linux 60-Second Analysis
      4. 3.4 BCC Tool Checklist
      5. 3.5 Summary
    4. Chapter 4. BCC
      1. 4.1 BCC Components
      2. 4.2 BCC Features
      3. 4.3 BCC Installation
      4. 4.4 BCC Tools
      5. 4.5 funccount
      6. 4.6 stackcount
      7. 4.7 trace
      8. 4.8 argdist
      9. 4.9 Tool Documentation
      10. 4.10 Developing BCC Tools
      11. 4.11 BCC Internals
      12. 4.12 BCC Debugging
      13. 4.13 Summary
    5. Chapter 5. bpftrace
      1. 5.1 bpftrace Components
      2. 5.2 bpftrace Features
      3. 5.3 bpftrace Installation
      4. 5.4 bpftrace Tools
      5. 5.5 bpftrace One-Liners
      6. 5.6 bpftrace Documentation
      7. 5.7 bpftrace Programming
      8. 5.8 bpftrace Usage
      9. 5.9 bpftrace Probe Types
      10. 5.10 bpftrace Flow Control
      11. 5.11 bpftrace Operators
      12. 5.12 bpftrace Variables
      13. 5.13 bpftrace Functions
      14. 5.14 bpftrace Map Functions
      15. 5.15 bpftrace Future Work
      16. 5.16 bpftrace Internals
      17. 5.17 bpftrace Debugging
      18. 5.18 Summary
  13. Part II: Using BPF Tools
    1. Chapter 6. CPUs
      1. 6.1 Background
      2. 6.2 Traditional Tools
      3. 6.3 BPF Tools
      4. 6.4 BPF One-Liners
      5. 6.5 Optional Exercises
      6. 6.6 Summary
    2. Chapter 7. Memory
      1. 7.1 Background
      2. 7.2 Traditional Tools
      3. 7.3 BPF Tools
      4. 7.4 BPF One-Liners
      5. 7.5 Optional Exercises
      6. 7.6 Summary
    3. Chapter 8. File Systems
      1. 8.1 Background
      2. 8.2 Traditional Tools
      3. 8.3 BPF Tools
      4. 8.4 BPF One-Liners
      5. 8.5 Optional Exercises
      6. 8.6 Summary
    4. Chapter 9. Disk I/O
      1. 9.1 Background
      2. 9.2 Traditional Tools
      3. 9.3 BPF Tools
      4. 9.4 BPF One-Liners
      5. 9.5 Optional Exercises
      6. 9.6 Summary
    5. Chapter 10. Networking
      1. 10.1 Background
      2. 10.2 Traditional Tools
      3. 10.3 BPF Tools
      4. 10.4 BPF One-Liners
      5. 10.5 Optional Exercises
      6. 10.6 Summary
    6. Chapter 11. Security
      1. 11.1 Background
      2. 11.2 BPF Tools
      3. 11.3 BPF One-Liners
      4. 11.4 Summary
    7. Chapter 12. Languages
      1. 12.1 Background
      2. 12.2 C
      3. 12.3 Java
      4. 12.4 Bash Shell
      5. 12.5 Other Languages
      6. 12.6 Summary
    8. Chapter 13. Applications
      1. 13.1 Background
      2. 13.2 BPF Tools
      3. 13.3 BPF One-Liners
      4. 13.4 BPF One-Liners Examples
      5. 13.5 Summary
    9. Chapter 14. Kernel
      1. 14.1 Background
      2. 14.2 Strategy
      3. 14.3 Traditional Tools
      4. 14.4 BPF Tools
      5. 14.5 BPF One-Liners
      6. 14.6 BPF One-Liners Examples
      7. 14.7 Challenges
      8. 14.8 Summary
    10. Chapter 15. Containers
      1. 15.1 Background
      2. 15.2 Traditional Tools
      3. 15.3 BPF Tools
      4. 15.4 BPF One-Liners
      5. 15.5 Optional Exercises
      6. 15.6 Summary
    11. Chapter 16. Hypervisors
      1. 16.1 Background
      2. 16.2 Traditional Tools
      3. 16.3 Guest BPF Tools
      4. 16.4 Host BPF Tools
      5. 16.5 Summary
  14. Part III: Additional Topics
    1. Chapter 17. Other BPF Performance Tools
      1. 17.1 Vector and Performance Co-Pilot (PCP)
      2. 17.2 Grafana and Performance Co-Pilot (PCP)
      3. 17.3 Cloudflare eBPF Prometheus Exporter (with Grafana)
      4. 17.4 kubectl-trace
      5. 17.5 Other Tools
      6. 17.6 Summary
    2. Chapter 18. Tips, Tricks, and Common Problems
      1. 18.1 Typical Event Frequency and Overhead
      2. 18.2 Sample at 49 or 99 Hertz
      3. 18.3 Yellow Pigs and Gray Rats
      4. 18.4 Write Target Software
      5. 18.5 Learn Syscalls
      6. 18.6 Keep It Simple
      7. 18.7 Missing Events
      8. 18.8 Missing Stacks Traces
      9. 18.9 Missing Symbols (Function Names) When Printing
      10. 18.10 Missing Functions When Tracing
      11. 18.11 Feedback Loops
      12. 18.12 Dropped Events
  15. Part IV: Appendixes
    1. Appendix A. bpftrace One-Liners
      1. Chapter 6 CPUs
      2. Chapter 7 Memory
      3. Chapter 8 File Systems
      4. Chapter 9 Disk I/O
      5. Chapter 10 Networking
      6. Chapter 11 Security
      7. Chapter 13 Applications
      8. Chapter 14 Kernel
    2. Appendix B. bpftrace Cheat Sheet
      1. Synopsis
      2. Probes
      3. Probe Aliases
      4. Vars
      5. Actions
      6. Asynchronous Actions
      7. Switches
    3. Appendix C. BCC Tool Development
      1. Resources
      2. Five Tips
      3. Tool Examples
      4. More Info
    4. Appendix D. C BPF
      1. Why Program in C?
      2. Five Tips
      3. C Programs
      4. perf C
      5. More Info
    5. Appendix E. BPF Instructions
      1. Helper Macros
      2. Instructions
      3. Encoding
      4. References
  16. Glossary
  17. Bibliography
  18. Index
  19. Code Snippets

Product information

  • Title: BPF Performance Tools
  • Author(s): Brendan Gregg
  • Release date: December 2019
  • Publisher(s): Addison-Wesley Professional
  • ISBN: 9780136624523