Linux Observability with BPF

Preface

In 2015, David was working as a core developer for Docker, the company that made containers popular. His day-to-day work was divided between helping the community and growing the project. Part of his job was reviewing the firehose of pull requests that members of the community sent us; he also had to ensure that Docker worked for all kinds of scenarios, including high-performance workloads that were running and provisioning thousands of containers at any point of time.

To diagnose performance issues at Docker, we used flame graphs, which are advanced visualizations to help you navigate that data easily. The Go programming language makes it really easy to measure and extract application performance data using an embedded HTTP endpoint and generate graphs based on that data. David wrote an article about Go’s profiler capabilities and how you can use its data to generate flame graphs. A big pitfall about the way that Docker collects performance data is that the profiler is disabled by default, so if you’re trying to debug a performance issue, the first action to take is to restart Docker. The main issue with this strategy is that by restarting the service, you’ll probably lose the relevant data that you’re trying to collect, and then you need to wait until the event you’re trying to trace happens again. In David’s article about Docker flame graphs, he mentioned this as a necessary step to measure Docker’s performance, but that it didn’t need to be this way. This realization ...