Performance Tuning for Linux® Servers

Performance Tuning for Linux® Servers

by Sandra K. Johnson, Gerrit Huizenga, Badari Pulavarty
Released May 2005
Publisher(s): IBM Press
ISBN: 9780131447530

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

  • Proven techniques for optimizing web, file, print, database, and application servers

  • Practical introductions to open source performance management tools

  • Easy-to-understand examples based on Red Hat Enterprise Linux and Novell SUSE Linux Enterprise Server

    • An indispensable guide to maximizing Linux system and application performance

    From Wall Street to Hollywood, Linux runs many of the world's most businesscritical systems. Linux performance now impacts the entire enterprise. In Performance Tuning for Linux Servers, a team of IBM's most-experienced Linux performance specialists shows you how to find bottlenecks, measure performance, and identify effective optimizations.

    This book doesn't just cover kernel tuning: it shows how to maximize the end-to-end performance of real-world applications and databases running on Linux. Throughout, the authors present realistic examples based on today's most popular enterprise Linux platforms, Intel-based Red Hat Enterprise Linux, and Novell SUSE Linux Enterprise Server. These examples are designed for simplicity, clarity, and easy adaptation to any contemporary Linux environment. You'll discover how to

  • Install and configure Linux for maximum performance from the outset

  • Evaluate and choose the right hardware architecture for your Linux environment

  • Understand Linux kernels 2.4 through 2.6: components, performance issues, and optimization opportunities

  • Master core Linux performance tuning principles and strategies

  • Utilize free, open source tools for measurement, monitoring, system tracing, and benchmarking

  • Interpret performance data to analyze your Linux server's real-world behavior

  • Optimize Linux system schedulers, memory, I/O, file systems, and networking

  • Tune web, file, database, and application servers running commercial workloads

  • Predict the impact of changes in tuning parameters or configurations

  • Tune Linux code: optimize design, timing, sockets, threads, synchronization, and more

  • Architect for maximum performance: SMP scaling, clustering, and topology

  • Integrate kernel and application tuning in end-to-end system optimization projects

    • Whether you're an administrator, developer, integrator, or consultant, Performance Tuning for Linux Serverswill help you maximize the performance and value of every Linux system and application you run.

    © Copyright Pearson Education. All rights reserved.

    Table of contents

    1. Copyright
    2. IBM Press
      1. On Demand Computing Books
      2. WebSphere Books
      3. More Books from IBM Press
      4. DB2® Books
    3. Preface
    4. Acknowledgments
    5. About the Editors
    6. About the Contributors
    7. I. Linux Overview
      1. One. Linux Installation Issues
        1. Introduction
        2. Preinstallation Planning
          1. Partition Placement
          2. Using Multiple Hard Drives
          3. Selecting File Systems
          4. Converting File Systems
          5. Configuring RAID
        3. Configurable 2.6 Kernel Features
          1. I/O Elevators
          2. Huge TLB Page Support
        4. Linux Logging Facility
          1. /var/log/messages
          2. /var/log/XFree86.0.log
          3. Log Rotation
          4. Logger
          5. Customized Logging
        5. System Initialization: BSD Versus System V Initialization
          1. Initialization Table (/etc/inittab)
          2. BSD inittab (Slackware)
          3. Sys V inittab (Red Hat)
        6. Summary
        7. References
          1. Bibliography
      2. Two. Kernel Overview
        1. Introduction
        2. The Evolution of Linux
        3. Linux Kernel Architecture
          1. Kernel Responsibilities
          2. Organization and Modules
          3. Using Kernel Services
          4. /proc File System—External Performance View
          5. Memory Management
            1. Address Space
            2. Physical Memory
            3. Memory Mapping
            4. Paging
            5. Swapping
        4. Process Management
          1. Processes, Tasks, and Kernel Threads
          2. Scheduling and Context Switching
        5. Interprocess Communications
          1. Signals
          2. Pipes
          3. System V IPC Mechanisms
            1. Message Queues
            2. Semaphores
            3. Shared Memory
        6. The Linux Symmetrical Multiprocessing (SMP) Model
          1. Types of Multiprocessing
          2. Concurrency and Data Serialization
          3. Locks, Lock Granularity, and Locking Overhead
          4. Cache Coherency
          5. Processor Affinity
        7. File Systems
          1. Virtual File System (VFS)
          2. ext2fs
          3. LVM and RAID
          4. Volume Groups
          5. Device Special Files
          6. devfs
        8. New Features in Linux 2.6
        9. Summary
        10. References
          1. Bibliography
      3. Three. Overview of Server Architectures
        1. Introduction
        2. Linux Servers
        3. Processors and Multiprocessing
          1. Server Topologies
          2. Mixing Processors
        4. Memory
        5. I/O
        6. Linux Enterprise Servers
        7. Linux Clusters
          1. High-Performance Clusters
          2. High-Availability Clusters
        8. Examples of Server Systems
          1. IBM Mainframes—zSeries
            1. Hardware Design Overview
            2. Reliability
            3. IBM Mainframe I/O Design
          2. Blades
          3. NUMA
            1. Hardware Implementations
              1. Types of Nodes
              2. Types of Interconnects
              3. Latency Ratios
            2. Specific NUMA Implementations
              1. MPIO on NUMA Systems
            3. Timers
        9. Summary
    8. II. Performance Analysis Tools
      1. Four. System Performance Monitoring
        1. Introduction
        2. Background on Linux and Performance Analysis
        3. CPU Utilization
          1. vmstat
          2. top and gtop
          3. sar
            1. CPU Utilization
            2. Interrupt Rate
            3. Interrupt Distribution
        4. Memory Utilization
          1. /proc/meminfo and /proc/slabinfo
          2. ps
          3. vmstat
        5. I/O Utilization
          1. iostat
          2. sar
        6. Network Utilization
          1. Network Statistics
          2. Displaying Interface Information
          3. TCP/IP Protocol Statistics
          4. nfsstat
        7. Summary
        8. References
          1. Bibliography
      2. Five. System Trace Tools
        1. Introduction
        2. Requirements for System Tracing
        3. The top Utility
        4. strace
        5. OProfile
          1. opcontrol
            1. Using the Counter Support
            2. Separate Profiles for Libraries and the Kernel
            3. Saving Session Data
            4. oprof_start
          2. Data Profiling Tools
            1. oprofpp
            2. op_time
            3. op_to_source
            4. op_merge
        6. Performance Inspector
          1. Above Idle
          2. Per-Thread Time
          3. Trace Profiling
          4. Instruction Tracing
          5. Java Profiler
          6. Java Lock Monitor
          7. Performance Inspector Executable Tools
            1. swtrace
            2. post
            3. pipcntr
            4. ptt
            5. pttstats
        7. Summary
        8. References
          1. Bibliography
      3. Six. Benchmarks as an Aid to Understanding Workload Performance
        1. Introduction
        2. Benchmarking to Improve Your Workload
        3. What Types of Benchmarks Are There?
        4. Microbenchmarks
          1. Operating System Benchmarks
            1. LMbench
              1. LMbench Sample Output
            2. AIM7 and AIM9
              1. AIM7
              2. AIM9
              3. Sample AIM9 Output
            3. Reaim
            4. SPEC SDET
          2. Disk Benchmarks
            1. Bonnie/Bonnie++
            2. IOzone
            3. IOmeter
            4. tiobench
            5. dbench
          3. Network Benchmarks
            1. Netperf
            2. SPEC SFS
          4. Application Benchmarks
            1. Java Benchmarks
            2. PostMark
            3. Database Benchmarks
        5. Web Server Benchmarks
          1. SPECweb, SPECweb SSL, and TPC-W
            1. SPECjAppServer and ECPerf
            2. Other Application Benchmarks
        6. Summary
    9. III. System Tuning
      1. Seven. System Performance Principles and Strategy: A Benchmarking Methodology Case Study
        1. Introduction
        2. Performance Evaluation Methodologies
          1. Tracing
          2. Workload Characterization
          3. Numerical Analysis
          4. Simulation
        3. Benchmarking Methodology Case Study
        4. Analysis Methodology
          1. Hardware and Software
          2. Run Rules
          3. Setting Targets
          4. Measurement, Analysis, and Tuning
          5. Exit Strategy
        5. Benchmarks
          1. Results
        6. Summary
        7. Acknowledgments
        8. References
          1. Bibliography
      2. Eight. Scheduler Tuning
        1. Introduction
        2. Single-Processor Systems
        3. Symmetric Multiprocessing (SMP)
        4. Non-Uniform Memory Access (NUMA)
        5. Symmetric Multithreading (SMT)
        6. The 2.6 Linux Scheduler
        7. Load Balancing
        8. Tunable Parts of the Scheduler
          1. *CHILD_PENALTY
          2. *CREDIT_LIMIT
          3. *EXIT_WEIGHT
          4. *INTERACTIVE_DELTA
          5. *MAX_SLEEP_AVG
          6. *MAX_TIMESLICE
          7. *MIN_TIMESLICE
          8. *PARENT_PENALTY
          9. *PRIO_BONUS_RATIO
          10. *STARVATION_LIMIT
        9. Summary
        10. References
          1. Bibliography
      3. Nine. The Linux Virtual Memory—Performance Implications
        1. Introduction
        2. Memory and Address Space
          1. Address Space
          2. User Address Space
          3. The VM Area Structures
          4. Kernel Address Space
            1. Kernel Module Section
            2. Kernel Image Section
        3. High-Memory Support
          1. The highmem Interface
        4. Paging and Swapping
          1. Replacement Policy
          2. Page Replacement and Memory Balancing
        5. The Linux Page Tables
        6. New Features in Linux 2.6
          1. rmap and objrmap
          2. Largepages Support
          3. Page Allocation and Replacement
          4. Slab Allocator
          5. VM Tunables
          6. CPU Scheduler
        7. Summary
        8. References
          1. Bibliography
      4. Ten. I/O Subsystems—Performance Implications
        1. Introduction
        2. I/O Scheduling and the Block I/O (BIO) Layer
          1. The 2.6 Linux I/O Schedulers
          2. The 2.4 Linux I/O Scheduler
          3. The 2.6 Deadline I/O Scheduler
          4. Scheduler Tunables
          5. The 2.6 Anticipatory I/O Scheduler
          6. The 2.6 CFQ Scheduler
          7. The 2.6 noop I/O Scheduler
          8. I/O Scheduler—Performance Implications
        3. Read and Write Request Batches
        4. Read Anticipation Heuristic
        5. I/O Components That Affect Performance
        6. Addressing an I/O Device
        7. Summary
        8. References
          1. Bibliography
      5. Eleven. File System Tuning
        1. Introduction
        2. File System Fundamentals
          1. File System Implementation Considerations
          2. Creating Optimized File Systems
          3. Basic File System Terms
        3. Journaled File Systems
          1. When Good File Systems Go Bad
          2. Transactions Are the Solution
          3. A Wealth of Options
        4. Disks Factor into File System Performance
          1. Physical Disk Storage Considerations
          2. Disk Anatomy
          3. Transfer Rate
          4. hdparm
        5. Fragmenting a File System
        6. File Synchronization
          1. How to Assure Data or File Integrity
          2. Using Function Calls
          3. Using File Descriptors
          4. Performance Implications of sync/fsync
        7. bdflush Parameters
        8. Asynchronous Input and Output
          1. Using Asynchronous I/O
          2. Asynchronous I/O Functions
        9. Raw Disk I/O
          1. Setting up Raw I/O on Linux
          2. Example
        10. Ext2 and Ext3
          1. Ext2 Organization
          2. Block Allocation in the Ext2 File System
          3. Creating an Ext2 File System
          4. Ext3 Extensions for the Ext2 File System
          5. Kernel Configuration Support for Ext3
          6. Working with Ext3
          7. Creating an Ext3 Partition
          8. Converting an Ext2 File System to Ext3
          9. Using a Separate Journal Device on an Ext3 File System
          10. Ext2/Ext3 Utilities
        11. ReiserFS
          1. Kernel Configuration Support for ReiserFS
          2. Working with ReiserFS
          3. Creating a ReiserFS File System
          4. Increasing Speed with an External Log
          5. Mounting the File System
          6. Mount Options
          7. Tuning ReiserFS
          8. ReiserFS Utilities
        12. Journaled File System (JFS)
          1. Kernel Configuration Support for JFS
          2. Working with JFS
          3. Creating a JFS File System
          4. Increasing Speed with an External Log for JFS
          5. Mounting the File System
          6. Mount Options
          7. Tuning JFS
          8. JFS Utilities
        13. Next-Generation File System (XFS)
          1. Kernel Configuration Support for XFS
          2. Working with XFS File Systems
          3. Creating an XFS File System
          4. Full File System
          5. Increasing Speed with an External Log for XFS
          6. Mounting the File System
          7. Mount Options
          8. Tuning XFS
          9. XFS Utilities
        14. Summary
        15. References
          1. Bibliography
      6. Twelve. Network Tuning
        1. Introduction
        2. The Network Protocol Stack
        3. Kernel Parameter Tuning Mechanisms
        4. Kernel Auto Tuning
        5. Core Kernel Parameter Descriptions
          1. An Introduction to Socket Buffers
            1. Default Socket Buffer Size
            2. Maximum Socket Buffer Size
            3. netdev_max_backlog
            4. somaxconn
            5. optmem_max
        6. TCP/IPv4 Protocol Kernel Parameters
          1. TCP Buffer and Memory Management
            1. tcp_rmem
            2. tcp_wmem
            3. tcp_mem
          2. TCP Options
            1. tcp_window_scaling
            2. tcp_sack
            3. tcp_dsack
            4. tcp_fack
          3. TCP Connection Management
            1. tcp_max_syn_backlog
            2. tcp_synack_retries
            3. tcp_retries2
          4. TCP Keep-Alive Management
            1. tcp_keepalive_time
            2. tcp_keepalive_intvl
            3. tcp_keepalive_probes
          5. IP Port Space Range
            1. ip_local_port_range
        7. Summary
        8. References
          1. Bibliography
      7. Thirteen. Interprocess Communication
        1. Introduction
        2. What Is Interprocess Communication?
        3. Linux SysV IPC Resources and the ipcs Command
          1. The ipcs Command
          2. IPC Identifiers and Their Limits
        4. Semaphore Parameters
          1. semmni
          2. semmns
          3. semmsl
          4. semopm
          5. semvmx
          6. Unused Semaphore Parameters in Linux
        5. Message Queue Parameters
          1. msgmni
          2. msgmax
          3. msgmnb
          4. Unused Message Queue Parameters
        6. Shared Memory Segment Parameters
          1. shmmni
          2. shmmax
          3. shmmin
          4. shmall
        7. Dynamically Modifying the Configurable IPC Parameters
          1. Using /proc
          2. Using sysctl
        8. Configuring IPC Parameters Statically
        9. Pipes
        10. Summary
      8. Fourteen. Code Tuning
        1. Introduction
        2. General Principles
        3. Profiling to Understand the Application
        4. Compiler Options as Tuning Tools
          1. Basic Performance Tuning
          2. One Word About Compiler Optimizations
        5. Code Tuning
        6. Algorithm: Achieving Performance Through Design Choices
          1. Problems and Solution Possibilities
          2. A Problem—How Fast Can We Connect, Create a File, Read It, and Disconnect?
          3. The Program
          4. Designing the Code
          5. The Server
          6. Timing
          7. Sockets
          8. Threads
          9. Synchronization
          10. FILE I/O
          11. The Client
          12. Code Discussion
          13. Compilation Options
          14. Libraries
        7. Summary
    10. IV. Performance Characterization of Linux Server Applications
      1. Fifteen. Web Servers
        1. Introduction
        2. HTTP Requests and Responses
        3. Network Behavior of a Web Server
        4. Anatomy of a Web Server Transaction
        5. Different Models of Web Servers
        6. Tuning Web Servers
          1. Tuning for All Web Servers
          2. Apache
          3. Flash and Other Event-Driven Servers
          4. Tux
          5. Performance Tools for Evaluating Web Servers
        7. Summary
        8. References
          1. Bibliography
      2. Sixteen. File and Print Servers
        1. Introduction
        2. Types of Dedicated Network Storage Servers
        3. Optimizing the Performance of Network Storage
          1. Determining What Data to Store Remotely
          2. SAN Versus Network File Systems/NAS
          3. The Network File System Protocol
          4. Client and Server Implementation Choices
          5. Tuning the Linux Client—Some Key Concepts
            1. Protocol Layering
            2. Opportunistic Locking
            3. Metadata
            4. File Change Notification
            5. Read-only Volumes and Read-Only Files
          6. Linux File Server Tuning
            1. NFS
            2. Samba
          7. Performance Measurement
          8. Load Measurement for Improved Capacity Planning
          9. Print Server Performance
        4. References
          1. Bibliography
      3. Seventeen. Database Servers
        1. Introduction
        2. Overview of Database Architectures
        3. Database Tuning Areas to Consider
          1. I/O Tuning
          2. Queue Length and Response Times
          3. Load Balancing
          4. Global Memory
          5. Logging Device
          6. Typical Database Workloads
        4. Process Management
        5. Memory Management
        6. I/O Management
          1. Bounce Buffers
          2. Raw I/O
          3. Vectored I/O
          4. Asynchronous I/O
          5. Direct I/O
          6. Block Size I/O
          7. I/O Request Lock
        7. Summary
      4. Eighteen. Application Servers
        1. Introduction
        2. The Application Server Defined
        3. Java, J2EE, and Application Servers
        4. Performance Characterization of Application Servers
          1. Characteristics of the Application Server
          2. Application Servers on Linux
            1. The Software Performance Stack
            2. The Complexity of the J2EE Framework
            3. The Enterprise Application
            4. The Application Server
            5. The Java Virtual Machine
            6. Native Code
            7. The Hardware and the Operating System
            8. Application Server Hot Spots
              1. JVM
              2. Networking
              3. Threads
              4. Memory Usage
              5. Synchronization
              6. String Manipulation
              7. Web Server
              8. File Systems
        5. Improving Performance and High Availability
          1. SMP Scaling
          2. Clustering
          3. Topology
          4. Performance Tuning
        6. Summary
        7. References
          1. Bibliography
    11. V. Tuning Case Studies
      1. Nineteen. Case Study: Tuning the I/O Schedulers in Linux 2.6
        1. Introduction
        2. Benchmark Environment and Workload Profiles
        3. I/O Schedulers and Performance
        4. Single-CPU Single-Disk Setup
        5. 8-Way RAID-5 Setup
        6. 16-Way RAID-0 Setup
        7. AS Sequential Read Performance
        8. AS Versus Deadline Performance
        9. CFQ Performance
        10. Summary
        11. References
          1. Bibliography
      2. Twenty. Case Study: File System Tuning
        1. Introduction
        2. Analyzing File Layout
          1. Ext2/Ext3 Layout
          2. Journaled File System (JFS) File Layout
          3. ReiserFS File Layout
          4. XFS File Layout
        3. Tuning File Systems
          1. Tuning Options for Ext3: Using a Separate Journal Device
          2. Tuning Options for ReiserFS: Go Faster with an External Log
          3. Tuning Options for JFS: Go Faster with an External Log
            1. Tuning JFS with jfs_tune
          4. Tuning Options for XFS
            1. Using the Defaults
            2. Using logbufsize and logbsize
            3. Placing the Log on an External Device
        4. Measuring I/O
          1. iostat
          2. Using the iostat and sar Utilities
        5. Summary
        6. References
          1. Bibliography
      3. Twenty-One. Case Study: Network Performance on Linux
        1. Introduction
        2. Benchmarks Used in This Case Study
          1. NetBench
          2. Netperf3
          3. VolanoMark
          4. SPECWeb99
        3. Enhancements in the Linux 2.4 and 2.6 Kernels
          1. SendFile Support
          2. TCP Segmentation Offloading Support
          3. Process and IRQ Affinity in Network Load
          4. NAPI Support
          5. TCP Offload Engine Support
        4. Case Study
          1. NetBench
          2. Netperf3 (Gigabit Ethernet Tuning Case Study)
          3. VolanoMark
          4. SPECWeb99
        5. Summary
        6. References
          1. Bibliography
      4. Twenty-Two. Case Study: Commercial Workload Tuning
        1. Introduction
        2. Overview of Commercial Workload Tuning
        3. Standard Commercial Workload Model for J2EE
        4. Our Commercial Workload Model: Stock Trading
          1. System Configuration
          2. A Little More About Trade3
          3. Performance Analysis and Methodology
        5. The Performance Analysis Exercise
          1. The Web Server
          2. The Database Back End: A Bottleneck?
            1. Thread Dump
            2. Database I/O Activity
            3. Network Activity Between the Application Server and Database Server
          3. The Java Virtual Machine
          4. The Application Server
            1. Thread Pools
            2. Pass-by-Reference
            3. Data Source Prepared Statements and Connection Pool
          5. Hyperthreading
        6. Summary
        7. References
          1. Bibliography
      5. A. Tuning Kernel Parameters
        1. Introduction
        2. The sysctl Interface
          1. Using sysctl
        3. The procfs Interface
          1. Using procfs
        4. sysfs (Linux Kernel 2.6 Only)
          1. Using sysfs
        5. General Kernel Parameters
          1. Shared Memory
          2. Processes
          3. Signals
          4. Profiling/Debugging
          5. System
        6. Virtual Memory
        7. File System
        8. Network
          1. Net Core
          2. ICMP
          3. NEIGHBOR
          4. CONFIG
          5. Routing
          6. TCP
          7. IP Fragmentation

    Product information

    • Title: Performance Tuning for Linux® Servers
    • Author(s): Sandra K. Johnson, Gerrit Huizenga, Badari Pulavarty
    • Release date: May 2005
    • Publisher(s): IBM Press
    • ISBN: 9780131447530