book

Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools

Name: Optimizing Linux® Performance: A Hands-On Guide to Linux® Performance Tools
Author: Phillip G. Ezolt
ISBN: 0131486829

by Phillip G. Ezolt

March 2005

Intermediate to advanced

384 pages

7h 5m

English

Pearson

Read now

Unlock full access

Copyright
Dedication
Hewlett-Packard® Professional Books
Preface
Why Is Performance Important?Linux: Strengths and WeaknessHow Can This Book Help You?Why Learn How to Use Performance Tools?Can I Tune for Performance?Who Should Read This Book?How Is This Book Organized?
Acknowledgments
About the Author
1. Performance Hunting Tips
1.1. General Tips1.1.1. Take Copious Notes (Save Everything)1.1.2. Automate Redundant Tasks1.1.3. Choose Low-Overhead Tools If Possible1.1.4. Use Multiple Tools to Understand the Problem1.1.5. Trust Your Tools1.1.6. Use the Experience of Others (Cautiously)1.2. Outline of a Performance Investigation1.2.1. Finding a Metric, Baseline, and Target1.2.1.1. Establish a Metric1.2.1.2. Establish a Baseline1.2.1.3. Establish a Target1.2.2. Track Down the Approximate Problem1.2.3. See Whether the Problem Has Already Been Solved1.2.4. The Case Begins (Start to Investigate)1.2.5. Document, Document, Document1.3. Chapter Summary
2. Performance Tools: System CPU
2.1. CPU Performance Statistics2.1.1. Run Queue Statistics2.1.2. Context Switches2.1.3. Interrupts2.1.4. CPU Utilization2.2. Linux Performance Tools: CPU2.2.1. vmstat (Virtual Memory Statistics)2.2.1.1. CPU Performance-Related Options2.2.1.2. Example Usage2.2.2. top (v. 2.0.x)2.2.2.1. CPU Performance-Related Options2.2.2.2. Example Usage2.2.3. top (v. 3.x.x)2.2.3.1. CPU Performance-Related Options2.2.3.2. Example Usage2.2.4. procinfo (Display Info from the /proc File System)2.2.4.1. CPU Performance-Related Options2.2.4.2. Example Usage2.2.5. gnome-system-monitor2.2.5.1. CPU Performance-Related Options2.2.5.2. Example Usage2.2.6. mpstat (Multiprocessor Stat)2.2.6.1. CPU Performance-Related Options2.2.6.2. Example Usage2.2.7. sar (System Activity Reporter)2.2.7.1. CPU Performance-Related Options2.2.7.2. Example Usage2.2.8. oprofile2.2.8.1. CPU Performance-Related Options2.2.8.2. Example Usage2.3. Chapter Summary
3. Performance Tools: System Memory
3.1. Memory Performance Statistics3.1.1. Memory Subsystem and Performance3.1.2. Memory Subsystem (Virtual Memory)3.1.2.1. Swap (Not Enough Physical Memory)3.1.2.2. Buffers and Cache (Too Much Physical Memory)3.1.2.3. Active Versus Inactive Memory3.1.2.4. High Versus Low Memory3.1.2.5. Kernel Usage of Memory (Slabs)3.2. Linux Performance Tools: CPU and Memory3.2.1. vmstat (Virtual Memory Statistics) II3.2.1.1. System-Wide Memory-Related Options3.2.1.2. Example Usage3.2.2. top (2.x and 3.x)3.2.2.1. Memory Performance-Related Options3.2.2.2. Example Usage3.2.3. procinfo II3.2.3.1. Memory Performance-Related Options3.2.3.2. Example Usage3.2.4. gnome-system-monitor (II)3.2.4.1. Memory Performance-Related Options3.2.4.2. Example Usage3.2.5. free3.2.5.1. Memory Performance-Related Options3.2.5.2. Example Usage3.2.6. slabtop3.2.6.1. Memory Performance-Related Options3.2.6.2. Example Usage3.2.7. sar (II)3.2.7.1. Memory Performance-Related Options3.2.7.2. Example Usage3.2.8. /proc/meminfo3.2.8.1. Memory Performance-Related Options3.8.2.2. Example Usage3.3. Chapter Summary
4. Performance Tools: Process-Specific CPU
4.1. Process Performance Statistics4.1.1. Kernel Time Versus User Time4.1.2. Library Time Versus Application Time4.1.3. Subdividing Application Time4.2. The Tools4.2.1. time4.2.1.1. CPU Performance-Related Options4.2.1.2. Example Usage4.2.2. strace4.2.2.1. CPU Performance-Related Options4.2.2.2. Example Usage4.2.3. ltrace4.2.3.1. CPU Performance-Related Options4.2.3.2. Example Usage4.2.4. ps (Process Status)4.2.4.1. CPU Performance-Related Options4.2.4.2. Example Usage4.2.5. ld.so (Dynamic Loader)4.2.5.1. CPU Performance-Related Options4.2.5.2. Example Usage4.2.6. gprof4.2.6.1. CPU Performance-Related Options4.2.6.2. Example Usage4.2.7. oprofile (II)4.2.7.1. CPU Performance-Related Options4.2.7.2. Example Usage4.2.8. Languages: Static (C and C++) Versus Dynamic (Java and Mono)4.3. Chapter Summary
5. Performance Tools: Process-Specific Memory
5.1. Linux Memory Subsystem5.2. Memory Performance Tools5.2.1. ps5.2.1.1. Memory Performance-Related Options5.2.1.2. Example Usage5.2.2. /proc/<PID>5.2.2.1. Memory Performance-Related Options5.2.2.2. Example Usage5.2.3. memprof5.2.3.1. Memory Performance-Related Options5.2.3.2. Example Usage5.2.4. valgrind (cachegrind)5.2.4.1. Memory Performance-Related Options5.2.4.2. Example Usage5.2.5. kcachegrind5.2.5.1. Memory Performance-Related Options5.2.5.2. Example Usage5.2.6. oprofile (III)5.2.6.1. Memory Performance-Related Options5.2.6.2. Example Usage5.2.7. ipcs5.2.7.1. Memory Performance-Related Options5.2.7.2. Example Usage5.2.8. Dynamic Languages (Java, Mono)5.3. Chapter Summary

6. Performance Tools: Disk I/O
6.1. Introduction to Disk I/O6.2. Disk I/O Performance Tools6.2.1. vmstat (ii)6.2.1.1. Disk I/O Performance-Related Options and Outputs6.2.1.2. Example Usage6.2.2. iostat6.2.2.1. Disk I/O Performance-Related Options and Outputs6.2.2.2. Example Usage6.2.3. sar6.2.3.1. Disk I/O Performance-Related Options and Outputs6.2.3.2. Example Usage6.2.4. lsof (List Open Files)6.2.4.1. Disk I/O Performance-Related Options and Outputs6.2.4.2. Example Usage6.3. What’s Missing?6.4. Chapter Summary
7. Performance Tools: Network
7.1. Introduction to Network I/O7.1.1. Network Traffic in the Link Layer7.1.2. Protocol-Level Network Traffic7.2. Network Performance Tools7.2.1. mii-tool (Media-Independent Interface Tool)7.2.1.1. Network I/O Performance-Related Options7.2.1.2. Example Usage7.2.2. ethtool7.2.2.1. Network I/O Performance-Related Options7.2.2.2. Example Usage7.2.3. ifconfig (Interface Configure)7.2.3.1. Network I/O Performance-Related Options7.2.3.2. Example Usage7.2.4. ip7.2.4.1. Network I/O Performance-Related Options7.2.4.2. Example Usage7.2.5. sar7.2.5.1. Network I/O Performance-Related Options7.2.5.2. Example Usage7.2.6. gkrellm7.2.6.1. Network I/O Performance-Related Options7.2.6.2. Example Usage7.2.7. iptraf7.2.7.1. Network I/O Performance-Related Options7.2.7.2. Example Usage7.2.8. netstat7.2.8.1. Network I/O Performance-Related Options7.2.8.2. Example Usage7.2.9. etherape7.2.9.1. Network I/O Performance-Related Options7.2.9.2. Example Usage7.3. Chapter Summary
8. Utility Tools: Performance Tool Helpers
8.1. Performance Tool Helpers8.1.1. Automating and Recording Commands8.1.2. Graphing and Analyzing Performance Statistics8.1.3. Investigating the Libraries That an Application Uses8.1.4. Creating and Debugging Applications8.2. Tools8.2.1. bash8.2.1.1. Performance-Related Options8.2.1.2. Example Usage8.2.2. tee8.2.2.1. Performance-Related Options8.2.2.2. Example Usage8.2.3. script8.2.3.1. Performance-Related Options8.2.3.2. Example Usage8.2.4. watch8.2.4.1. Performance-Related Options8.2.4.2. Example Usage8.2.5. gnumeric8.2.5.1. Performance-Related Options8.2.5.2. Example Usage8.2.6. ldd8.2.6.1. Performance-Related Options8.2.6.2. Example Usage8.2.7. objdump8.2.7.1. Performance-Related Options8.2.7.2. Example Usage8.2.8. GNU Debugger (gdb)8.2.8.1. Performance-Related Options8.2.8.2. Example Usage8.2.9. gcc (GNU Compiler Collection)8.2.9.1. Performance-Related Options8.2.9.2. Example Usage8.3. Chapter Summary
9. Using Performance Tools to Find Problems
9.1. Not Always a Silver Bullet9.2. Starting the Hunt9.3. Optimizing an Application9.3.1. Is Memory Usage a Problem?9.3.2. Is Startup Time a Problem?9.3.3. Is the Loader Introducing a Delay?9.3.4. Is CPU Usage (or Length of Time to Complete) a Problem?9.3.5. Is the Application’s Disk Usage a Problem?9.3.6. Is the Application’s Network Usage a Problem?9.4. Optimizing a System9.4.1. Is the System CPU-Bound?9.4.2. Is a Single Processor CPU-Bound?9.4.3. Are One or More Processes Using Most of the System CPU?9.4.4. Are One or More Processes Using Most of an Individual CPU?9.4.5. Is the Kernel Servicing Many Interrupts?9.4.6. Where Is Time Spent in the Kernel?9.4.7. Is the Amount of Swap Space Being Used Increasing?9.4.8. Is the System I/O-Bound?9.4.9. Is the System Using Disk I/O?9.4.10. Is the System Using Network I/O?9.5. Optimizing Process CPU Usage9.5.1. Is the Process Spending Time in User or Kernel Space?9.5.2. Which System Calls Is the Process Making, and How Long Do They Take to Complete?9.5.3. In Which Functions Does the Process Spend Time?9.5.4. What Is the Call Tree to the Hot Functions?9.5.5. Do Cache Misses Correspond to the Hot Functions or Source Lines?9.6. Optimizing Memory Usage9.6.1. Is the Kernel Memory Usage Increasing?9.6.2. What Type of Memory Is the Kernel Using?9.6.3. Is a Particular Process’s Resident Set Size Increasing?9.6.4. Is Shared Memory Usage Increasing?9.6.5. Which Processes Are Using the Shared Memory?9.6.6. What Type of Memory Is the Process Using?9.6.7. What Functions Are Using All of the Stack?9.6.8. What Functions Have the Biggest Text Size?9.6.9. How Big Are the Libraries That the Process Uses?9.6.10. What Functions Are Allocating Heap Memory?9.7. Optimizing Disk I/O Usage9.7.1. Is the System Stressing a Particular Disk?9.7.2. Which Application Is Accessing the Disk?9.7.3. Which Files Are Accessed by the Application?9.8. Optimizing Network I/O Usage9.8.1. Is Any Network Device Sending/Receiving Near the Theoretical Limit?9.8.2. Is Any Network Device Generating a Large Number of Errors?9.8.3. What Type of Traffic Is Running on That Device?9.8.4. Is a Particular Process Responsible for That Traffic?9.8.5. What Remote System Is Sending the Traffic?9.8.6. Which Application Socket Is Responsible for the Traffic?9.9. The End9.10. Chapter Summary
10. Performance Hunt 1: A CPU-Bound Application (GIMP)
10.1. CPU-Bound Application10.2. Identify a Problem10.3. Find a Baseline/Set a Goal10.4. Configure the Application for the Performance Hunt10.5. Install and Configure Performance Tools10.6. Run Application and Performance Tools10.7. Analyze the Results10.8. Jump to the Web10.9. Increase the Image Cache10.10. Hitting a (Tiled) Wall10.11. Solving the Problem10.12. Verify Correctness?10.13. Next Steps10.14. Chapter Summary
11. Performance Hunt 2: A Latency-Sensitive Application (nautilus)
11.1. A Latency-Sensitive Application11.2. Identify a Problem11.3. Find a Baseline/Set a Goal11.4. Configure the Application for the Performance Hunt11.5. Install and Configure Performance Tools11.6. Run Application and Performance Tools11.7. Compile and Examine the Source11.8. Using gdb to Generate Call Traces11.9. Finding the Time Differences11.10. Trying a Possible Solution11.11. Chapter Summary
12. Performance Hunt 3: The System-Wide Slowdown (prelink)
12.1. Investigating a System-Wide Slowdown12.2. Identify a Problem12.3. Find a Baseline/Set a Goal12.4. Configure the Application for the Performance Hunt12.5. Install and Configure Performance Tools12.6. Run Application and Performance Tools12.7. Simulating a Solution12.8. Reporting the Problem12.9. Testing the Solution12.10. Chapter Summary
13. Performance Tools: What’s Next?
13.1. The State of Linux Tools13.2. What Tools Does Linux Still Need?13.2.1. Hole 1: Performance Statistics Are Scattered13.2.2. Hole 2: No Reliable and Complete Call Tree13.2.3. Hole 3: I/O Attribution13.3. Performance Tuning on Linux13.3.1. Available Source13.3.2. Easy Access to Developers13.3.3. Linux Is Still Young13.4. Chapter Summary
A. Performance Tool Locations
B. Installing oprofile
B.1 Fedora Core 2 (FC2)B.2 Enterprise Linux 3 (EL3)B.3 SUSE 9.1

Overview

The first comprehensive, expert guide for end-to-end Linux application optimization
Learn to choose the right tools—and use them together to solve real problems in real production environments

Superior application performance is more crucial than ever—and in today's complex production environments, it's tougher to ensure, too. If you use Linux, you have extraordinary advantages: complete source code access, plus an exceptional array of optimization tools. But the tools are scattered across the Internet. Many are poorly documented. And few experts know how to use them together to solve real problems. Now, one of those experts has written the definitive Linux tuning primer for every professional: Optimizing Linux® Performance.

Renowned Linux benchmarking specialist Phillip Ezolt introduces each of today's most important Linux optimization tools, showing how they fit into a proven methodology for perfecting overall application performance. Using realistic examples, Ezolt shows developers how to pinpoint exact lines of source code that are impacting performance. He teaches sysadmins and application developers how to rapidly drill down to specific bottlenecks, so they can implement solutions more quickly. You'll discover how to:

Identify bottlenecks even if you're not familiar with the underlying system
Find and choose the right performance tools for any problem
Recognize the meaning of the events you're measuring
Optimize system CPU, user CPU, memory, network I/O, and disk I/O—and understand their interrelationships
Fix CPU-bound, latency-sensitive, and I/O-bound applications, through case studies you can easily adapt to your own environment

Install and use oprofile, the advanced systemwide profiler for Linux systems

If you're new to tuning, Ezolt gives you a clear and practical introduction to all the principles and strategies you'll need. If you're migrating to Linux, you'll quickly master Linux equivalents to the tools and techniques you already know. Whatever your background or environment, this book can help you improve the performance of all your Linux applications—increasingbusiness value and user satisfaction at the same time.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 0131486829Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills