Solaris™ Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris

Solaris™ Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris

by Richard McDougall, Jim Mauro, Brendan Gregg
Released July 2006
Publisher(s): Pearson
ISBN: 0131568191

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

"The Solaris™Internals volumes are simply the best and most comprehensive treatment of the Solaris (and OpenSolaris) Operating Environment. Any person using Solaris--in any capacity--would be remiss not to include these two new volumes in their personal library. With advanced observability tools in Solaris (like DTrace), you will more often find yourself in what was previously unchartable territory. Solaris™ Internals, Second Edition, provides us a fantastic means to be able to quickly understand these systems and further explore the Solaris architecture--especially when coupled with OpenSolaris source availability."

--Jarod Jenson, chief systems architect, Aeysis

"The Solaris™ Internals volumes by Jim Mauro and Richard McDougall must be on your bookshelf if you are interested in in-depth knowledge of Solaris operating system internals and architecture. As a senior Unix engineer for many years, I found the first edition of Solaris™ Internals the only fully comprehensive source for kernel developers, systems programmers, and systems administrators. The new second edition, with the companion performance and debugging book, is an indispensable reference set, containing many useful and practical explanations of Solaris and its underlying subsystems, including tools and methods for observing and analyzing any system running Solaris 10 or OpenSolaris."

--Marc Strahl, senior UNIX engineer

Solaris™ Performance and Tools provides comprehensive coverage of the powerful utilities bundled with Solaris 10 and OpenSolaris, including the Solaris Dynamic Tracing facility, DTrace, and the Modular Debugger, MDB. It provides a systematic approach to understanding performance and behavior, including:

  • Analyzing CPU utilization by the kernel and applications, including reading and understanding hardware counters

  • Process-level resource usage and profiling

  • Disk IO behavior and analysis

  • Memory usage at the system and application level

  • Network performance

  • Monitoring and profiling the kernel, and gathering kernel statistics

  • Using DTrace providers and aggregations

  • MDB commands and a complete MDB tutorial

    • The Solaris™ Internals volumes make a superb reference for anyone using Solaris 10 and OpenSolaris.

    Table of contents

    1. Copyright
      1. Dedication
    2. Foreword
    3. Preface
      1. About These Books
        1. Solaris™ Internals
        2. Solaris™ Performance and Tools
      2. Intended Audience
      3. OpenSolaris
      4. How the Books Are Organized
        1. Volume 1: Solaris™ Internals
        2. Volume 2: Solaris™ Performance and Tools
      5. Updates and Related Material
      6. Notational Conventions
      7. A Note from the Authors
    4. About the Authors
    5. Acknowledgments
      1. The Solaris™ Internals Community Authors
      2. The Reviewers
      3. Personal Acknowledgments from Richard
      4. Personal Acknowledgments from Jim
      5. Personal Acknowledgements from Brendan
    6. One. Observability Methods
      1. 1. Introduction to Observability Tools
        1. 1.1. Observability Tools
          1. 1.1.1. Kstat Tools
          2. 1.1.2. Process Tools
          3. 1.1.3. Forensic Tools
        2. 1.2. Drill-Down Analysis
        3. 1.3. About Part One
          1. 1.3.1. Chapter Layout
      2. 2. CPUs
        1. 2.1. Tools for CPU Analysis
        2. 2.2. vmstat Tool
        3. 2.3. CPU Utilization
        4. 2.4. CPU Saturation
        5. 2.5. psrinfo Command
        6. 2.6. uptime Command
        7. 2.7. sar Command
          1. 2.7.1. sar Default Output
          2. 2.7.2. sar -q
          3. 2.7.3. Capturing Historical Data
        8. 2.8. Clock Tick Woes
        9. 2.9. mpstat Command
        10. 2.10. Who Is Using the CPU?
        11. 2.11. CPU Run Queue Latency
        12. 2.12. CPU Statistics Internals
          1. 2.12.1. usr, sys, idl Times
          2. 2.12.2. Load Averages
          3. 2.12.3. pr_pctcpu Field
        13. 2.13. Using DTrace to Explain Events from Performance Tools
        14. 2.14. DTrace Versions of runq-sz, %runocc
        15. 2.15. DTrace Probes for CPU States
      3. 3. Processes
        1. 3.1. Tools for Process Analysis
        2. 3.2. Process Statistics Summary: prstat
          1. 3.2.1. Thread Summary: prstat -L
          2. 3.2.2. Process Microstates: prstat -m
          3. 3.2.3. Sorting by a Key: prstat -s
          4. 3.2.4. User Summary: prstat -t
          5. 3.2.5. Project Summary: prstat -J
          6. 3.2.6. Zone Summary: prstat -Z
        3. 3.3. Process Status: ps
          1. 3.3.1. /usr/bin/ps Command
          2. 3.3.2. /usr/ucb/ps
        4. 3.4. Tools for Listing and Controlling Processes
          1. 3.4.1. Process Tree: ptree
          2. 3.4.2. Grepping for Processes: pgrep
          3. 3.4.3. Killing Processes: pkill
          4. 3.4.4. Temporarily Stop a Process: pstop
          5. 3.4.5. Making a Process Runnable: prun
          6. 3.4.6. Wait for Process Completion: pwait
          7. 3.4.7. Reap a Zombie Process: preap
        5. 3.5. Process Introspection Commands
          1. 3.5.1. Process Stack: pstack
          2. 3.5.2. Process Memory Map: pmap -x
          3. 3.5.3. Process File Table: pfiles
          4. 3.5.4. Execution Time Statistics for a Process: ptime
          5. 3.5.5. Process Signal Disposition: psig
          6. 3.5.6. Process Libraries: pldd
          7. 3.5.7. Process Flags: pflags
          8. 3.5.8. Process Credentials: pcred
          9. 3.5.9. Process Arguments: pargs
          10. 3.5.10. Process Working Directory: pwdx
        6. 3.6. Examining User-Level Locks in a Process
        7. 3.7. Tracing Processes
          1. 3.7.1. Using truss to Trace Processes
          2. 3.7.2. Using apptrace to Trace Processes
          3. 3.7.3. Using DTrace to Trace Process Functions
          4. 3.7.4. Using DTrace to Aggregate Process Functions
          5. 3.7.5. Using DTrace to Peer Inside Processes
          6. 3.7.6. Using DTrace to Sample Stack Backtraces
        8. 3.8. Java Processes
          1. 3.8.1. Process Stack on a Java Virtual Machine: pstack
          2. 3.8.2. JVM Profiling
          3. 3.8.3. Tuning Java Garbage Collection
          4. 3.8.4. Using DTrace on Java Applications
      4. 4. Disk Behavior and Analysis
        1. 4.1. Terms for Disk Analysis
        2. 4.2. Random vs. Sequential I/O
          1. 4.2.1. Demonstration of Sequential Disk Activity
          2. 4.2.2. Demonstration of Random Disk Activity
        3. 4.3. Storage Arrays
        4. 4.4. Sector Zoning
        5. 4.5. Max I/O Size
        6. 4.6. iostat Utility
        7. 4.7. Disk Utilization
        8. 4.8. Disk Saturation
        9. 4.9. Disk Throughput
        10. 4.10. iostat Reference
          1. 4.10.1. iostat Default
          2. 4.10.2. iostat -D
          3. 4.10.3. iostat -l n
          4. 4.10.4. iostat -n
          5. 4.10.5. iostat -x
          6. 4.10.6. iostat -p, -P
          7. 4.10.7. iostat -e
          8. 4.10.8. iostat -E
        11. 4.11. Reading iostat
          1. 4.11.1. Event Size Ratio
          2. 4.11.2. Service Times
        12. 4.12. iostat Internals
        13. 4.13. sar -d
        14. 4.14. Trace Normal Form (TNF) Tracing for I/O
        15. 4.15. DTrace for I/O
          1. 4.15.1. io Probes
          2. 4.15.2. I/O Size One-Liners
          3. 4.15.3. A More Elaborate Example
          4. 4.15.4. I/O Size Aggregation
          5. 4.15.5. I/O Seek Aggregation
          6. 4.15.6. I/O File Names
        16. 4.16. Disk I/O Time
          1. 4.16.1. Simple Disk Event
          2. 4.16.2. Concurrent Disk Events
          3. 4.16.3. Adaptive Disk I/O Time Algorithm
          4. 4.16.4. Other Response Times
          5. 4.16.5. Time by Layer
        17. 4.17. DTraceToolkit Commands
          1. 4.17.1. iotop Script
          2. 4.17.2. iosnoop Script
          3. 4.17.3. Plotting Disk Activity
          4. 4.17.4. Plotting Concurrent Activity
        18. 4.18. DTraceTazTool
      5. 5. File Systems
        1. 5.1. Layers of File System and I/O
        2. 5.2. Observing Physical I/O
        3. 5.3. File System Latency
        4. 5.4. Causes of Read/Write File System Latency
          1. 5.4.1. Disk I/O Wait
          2. 5.4.2. Block or Metadata Cache Misses
          3. 5.4.3. I/O Breakup
          4. 5.4.4. Locking in the File System
          5. 5.4.5. Metadata Updates
        5. 5.5. Observing File System “Top End” Activity
        6. 5.6. File System Caches
          1. 5.6.1. Page Cache
          2. 5.6.2. Bypassing the Page Cache with Direct I/O
          3. 5.6.3. The Directory Name Lookup Cache
          4. 5.6.4. Block Buffer Cache
          5. 5.6.5. UFS Inode Cache
          6. 5.6.6. Monitoring UFS Caches with fcachestat
        7. 5.7. NFS Statistics
          1. 5.7.1. NFS Client Statistics: nfsstat -c
          2. 5.7.2. NFS Server Statistics: nfsstat -s
      6. 6. Memory
        1. 6.1. Tools for Memory Analysis
        2. 6.2. vmstat(1M) Command
        3. 6.3. Types of Paging
          1. 6.3.1. File I/O Paging: “Good” Paging
          2. 6.3.2. Anonymous Memory Paging: “Bad” Paging
          3. 6.3.3. Per-Zone Paging Statistics
        4. 6.4. Physical Memory Allocation
          1. 6.4.1. Total Physical Memory
          2. 6.4.2. Free Memory
          3. 6.4.3. Using the memstat Command in MDB
        5. 6.5. Relieving Memory Pressure
        6. 6.6. Scan Rate as a Memory Health Indicator
          1. 6.6.1. Using prstat to Estimate Memory Slowdowns
        7. 6.7. Process Virtual and Resident Set Size
        8. 6.8. Using pmap to Inspect Process Memory Usage
        9. 6.9. Calculating Process Memory Usage with ps and pmap
        10. 6.10. Displaying Page-Size Information with pmap
        11. 6.11. Using DTrace for Memory Analysis
          1. 6.11.1. Using DTrace to Estimate Memory Slowdowns
        12. 6.12. Obtaining Memory Kstats
        13. 6.13. Using the Perl Kstat API to Look at Memory Statistics
        14. 6.14. System Memory Allocation Kstats
        15. 6.15. Kernel Memory with kstat
        16. 6.16. System Paging Kstats
        17. 6.17. Observing MMU Performance Impact with trapstat
        18. 6.18. Swap Space
          1. 6.18.1. Swap Allocation
          2. 6.18.2. Swap Statistics
          3. 6.18.3. Swap Summary: swap -s
          4. 6.18.4. Listing Physical Swap Devices: swap -l
          5. 6.18.5. Determining Swapped-Out Threads
          6. 6.18.6. Monitoring Physical Swap Activity
          7. 6.18.7. MemTool prtswap
          8. 6.18.8. Display of Swap Reservations with pmap
      7. 7. Networks
        1. 7.1. Terms for Network Analysis
        2. 7.2. Packets Are Not Bytes
        3. 7.3. Network Utilization
        4. 7.4. Network Saturation
        5. 7.5. Network Errors
        6. 7.6. Misconfigurations
        7. 7.7. Systemwide Statistics
          1. 7.7.1. netstat Command
          2. 7.7.2. kstat Command
          3. 7.7.3. nx.se Tool
          4. 7.7.4. nicstat Tool
          5. 7.7.5. SNMP
          6. 7.7.6. checkcable Tool
          7. 7.7.7. ping Tool
          8. 7.7.8. traceroute Tool
          9. 7.7.9. snoop Tool
          10. 7.7.10. TTCP
          11. 7.7.11. pathchar Tool
          12. 7.7.12. ntop Tool
          13. 7.7.13. NFS Client Statistics: nfsstat -c
          14. 7.7.14. NFS Server Statistics: nfsstat -s
        8. 7.8. Per-Process Network Statistics
          1. 7.8.1. tcptop Tool
          2. 7.8.2. tcpsnoop Tool
        9. 7.9. TCP Statistics
          1. 7.9.1. TCP Statistics Internals
          2. 7.9.2. TCP Statistics from Kstat
          3. 7.9.3. TCP Statistics Reference
          4. 7.9.4. TCP Statistics from DTrace
        10. 7.10. IP Statistics
          1. 7.10.1. IP Statistics Internals
          2. 7.10.2. IP Statistics from Kstat
          3. 7.10.3. IP Statistics Reference
          4. 7.10.4. IP Statistics from DTrace
        11. 7.11. ICMP Statistics
          1. 7.11.1. ICMP Statistics from Kstat
          2. 7.11.2. ICMP Statistics Reference
          3. 7.11.3. ICMP Statistics from DTrace
          4. 7.11.4. Tracing Raw Network Functions
      8. 8. Performance Counters
        1. 8.1. Introducing CPU Caches
        2. 8.2. cpustat Command
          1. 8.2.1. Cache Hit Ratio, Cache Misses
          2. 8.2.2. Listing PICs and Events
          3. 8.2.3. PIC Examples: UltraSPARC IIi
          4. 8.2.4. PIC Examples: The UltraSPARC T1 Processor
          5. 8.2.5. Event Multiplexing
          6. 8.2.6. Using cpustat with Multiple CPUs
          7. 8.2.7. Cycles per Instruction
          8. 8.2.8. PIC Examples: UltraSPARC IV
        3. 8.3. cputrack Command
        4. 8.4. busstat Command
          1. 8.4.1. Listing Supported Buses
          2. 8.4.2. Listing Bus Events
          3. 8.4.3. Monitoring Bus Events
          4. 8.4.4. Event Multiplexing
          5. 8.4.5. Example: UltraSPARC T1
      9. 9. Kernel Monitoring
        1. 9.1. Tools for Kernel Monitoring
        2. 9.2. Profiling the Kernel and Drivers
          1. 9.2.1. Profiling the Kernel with lockstat -I
        3. 9.3. Analyzing Kernel Locks
          1. 9.3.1. Adaptive Locks
          2. 9.3.2. Spin Locks
          3. 9.3.3. Reader/Writer Locks
          4. 9.3.4. Thread Locks
          5. 9.3.5. Analyzing Locks with lockstat
        4. 9.4. DTrace lockstat Provider
        5. 9.5. DTrace Kernel Profiling
        6. 9.6. Interrupt Statistics: vmstat -i
        7. 9.7. Interrupt Analysis: intrstat
    7. Two. Observability Infrastructure
      1. 10. Dynamic Tracing
        1. 10.1. Introduction to DTrace
        2. 10.2. The Basics
          1. 10.2.1. D Program Structure
          2. 10.2.2. Providers and Probes
          3. 10.2.3. Aggregations
          4. 10.2.4. Variables
          5. 10.2.5. Probe Arguments
          6. 10.2.6. Mixing Providers
          7. 10.2.7. Accessing Global Kernel Data
          8. 10.2.8. Assorted Actions of Interest
        3. 10.3. Inspecting Java Applications with DTrace
          1. 10.3.1. Inspecting Applications with the DTrace jstack Action
          2. 10.3.2. Adding Probes to Pre-Mustang Releases
        4. 10.4. DTrace Architecture
        5. 10.5. Summary
        6. 10.6. Probe Reference
          1. 10.6.1. The I/O Provider
            1. 10.6.1.1. bufinfo_t structure
            2. 10.6.1.2. devinfo_t
            3. 10.6.1.3. fileinfo_t
          2. 10.6.2. Virtual Memory Provider Probes
          3. 10.6.3. The Sched Provider
            1. 10.6.3.1. Arguments
          4. 10.6.4. DTrace Lockstat Provider
            1. 10.6.4.1. Adaptive Lock Probes
            2. 10.6.4.2. Spin Lock Probes
            3. 10.6.4.3. Thread Locks
            4. 10.6.4.4. Readers/Writer Lock Probes
          5. 10.6.5. The Java Virtual Machine Provider
            1. 10.6.5.1. VM Life Cycle Probes
            2. 10.6.5.2. Thread Life Cycle Probes
            3. 10.6.5.3. Class-Loading Probes
            4. 10.6.5.4. Garbage Collection Probes
            5. 10.6.5.5. Method Compilation Probes
            6. 10.6.5.6. Monitor Probes
            7. 10.6.5.7. Application Tracking Probes
            8. 10.6.5.8. The hotspot_jni Provider
        7. 10.7. MDB Reference
      2. 11. Kernel Statistics
        1. 11.1. C-Level Kstat Interface
          1. 11.1.1. Data Structure Overview
          2. 11.1.2. Getting Started
          3. 11.1.3. Data Types
            1. 11.1.3.1. KSTAT_TYPE_RAW
            2. 11.1.3.2. KSTAT_TYPE_NAMED
            3. 11.1.3.3. KSTAT_TYPE_TIMER
            4. 11.1.3.4. KSTAT_TYPE_INTR
            5. 11.1.3.5. KSTAT_TYPE_IO
              1. Accumulated Time and Queue Length Statistics
          4. 11.1.4. Kstat Names
          5. 11.1.5. Functions
          6. 11.1.6. Management of Chain Updates
          7. 11.1.7. Putting It All Together
        2. 11.2. Command-Line Interface
          1. 11.2.1. The kstat Command
          2. 11.2.2. Real-World Example That Uses kstat and nawk
        3. 11.3. Using Perl to Access kstats
          1. 11.3.1. The Tied-Hash Interface to the kstat Facility
          2. 11.3.2. The update() Method
          3. 11.3.3. 64-Bit Values
          4. 11.3.4. Getting Started with Perl
          5. 11.3.5. netstatMulti Implemented in Perl
        4. 11.4. Snooping a Program’s kstat Use with DTrace
        5. 11.5. Adding Statistics to the Solaris Kernel
          1. 11.5.1. A kstat Provider Walkthrough
          2. 11.5.2. I/O Statistics
        6. 11.6. Additional Information
    8. Three. Debugging
      1. 12. The Modular Debugger
        1. 12.1. Introduction to the Modular Debugger
          1. 12.1.1. MDB
          2. 12.1.2. MDB Features
          3. 12.1.3. Terms
        2. 12.2. MDB Concepts
          1. 12.2.1. Building Blocks
          2. 12.2.2. Targets
          3. 12.2.3. Debugger Commands
          4. 12.2.4. Walker
          5. 12.2.5. Debugger Modules
          6. 12.2.6. Macros
          7. 12.2.7. Modularity
      2. 13. An MDB Tutorial
        1. 13.1. Invoking MDB
          1. 13.1.1. Logging Output to a File
        2. 13.2. MDB Command Syntax
          1. 13.2.1. Expressions
          2. 13.2.2. Symbols
          3. 13.2.3. Formatting Metacharacters
          4. 13.2.4. Formatting Characters
          5. 13.2.5. dcmds
          6. 13.2.6. Walkers
          7. 13.2.7. Macros
          8. 13.2.8. Pipelines
          9. 13.2.9. Piping to UNIX Commands
          10. 13.2.10. Obtaining Symbolic Type Information
          11. 13.2.11. Variables
          12. 13.2.12. Walkers, Variables, and Expressions Combined
        3. 13.3. Working with Debugging Targets
          1. 13.3.1. Displaying Stacks
          2. 13.3.2. Displaying Registers
          3. 13.3.3. Disassembling the Target
          4. 13.3.4. Setting Breakpoints
        4. 13.4. GDB-to-MDB Reference
        5. 13.5. dcmd and Walker Reference
          1. 13.5.1. Commands
          2. 13.5.2. Comments
          3. 13.5.3. Expressions
          4. 13.5.4. Symbols
          5. 13.5.5. dcmds
          6. 13.5.6. Variables
          7. 13.5.7. Read Formats
          8. 13.5.8. Write Formats
          9. 13.5.9. Search Formats
          10. 13.5.10. General dcmds
          11. 13.5.11. Target-Related dcmds
          12. 13.5.12. CTF-Related
          13. 13.5.13. Kernel: proc-Related
          14. 13.5.14. Kernel: Thread-Related
          15. 13.5.15. Kernel: Synchronization-Related
          16. 13.5.16. Kernel: CPU-Related
          17. 13.5.17. Kernel: Memory-Related
          18. 13.5.18. Kernel: kmem-Related
          19. 13.5.19. Process: Target-Related
          20. 13.5.20. Kernel: kmdb-Related
      3. 14. Debugging Kernels
        1. 14.1. Working with Kernel Cores
          1. 14.1.1. Locating and Attaching the Target
          2. 14.1.2. Examining Kernel Core Summary Information
          3. 14.1.3. Examining the Message Buffer
          4. 14.1.4. Obtaining a Stack Trace of the Running Thread
          5. 14.1.5. Which Process?
          6. 14.1.6. Disassembling the Suspect Code
          7. 14.1.7. Displaying General-Purpose Registers
          8. 14.1.8. Navigating the Stack Backtrace
          9. 14.1.9. Looking at the Status of the CPUs
          10. 14.1.10. Traversing Stack Frames in SPARC Architectures
          11. 14.1.11. Listing Processes and Process Stacks
          12. 14.1.12. Global Memory Summary
          13. 14.1.13. Listing Network Connections
          14. 14.1.14. Listing All Kernel Threads
          15. 14.1.15. Other Notable Kernel dcmds
        2. 14.2. Examining User Process Stacks within a Kernel Image
          1. 14.2.1. Enabling Process Pages in a Dump
          2. 14.2.2. Invoking MDB to Examine the Kernel Image
          3. 14.2.3. Locating the Target Process
          4. 14.2.4. Extracting the User-Mode Stack Frame Pointers
        3. 14.3. Switching MDB to Debug a Specific Process
          1. 14.3.1. Constructing the Process Stack
          2. 14.3.2. Examining the Process Memory
        4. 14.4. kmdb, the Kernel Modular Debugger
          1. 14.4.1. Diagnosing with kmdb and moddebug
            1. 14.4.1.1. Starting kmdb from the Console
          2. 14.4.2. Booting with the Kernel Debugger
          3. 14.4.3. Configuring a tty Console on x86
          4. 14.4.4. Investigating Hangs
          5. 14.4.5. Collecting Information about Panics
          6. 14.4.6. Working with Debugging Targets
          7. 14.4.7. Setting Breakpoints
          8. 14.4.8. Forcing a Crash Dump with halt -d
          9. 14.4.9. Forcing a Dump with kmdb
        5. 14.5. Kernel Built-In MDB dcmds
    9. Appendices
      1. A. Tunables and Settings
        1. A.1. Tunable Parameters in Solaris
          1. A.1.1. /etc/default Directory
          2. A.1.2. prctl Command
          3. A.1.3. /etc/system File
          4. A.1.4. driver.conf File
          5. A.1.5. ndd Command
          6. A.1.6. routeadm(1)
        2. A.2. System V IPC Tunables for Databases
      2. B. DTrace One-Liners
        1. B.1. DTrace One-Liners
        2. B.2. DTrace Longer One-Liners
      3. C. Java DTrace Scripts
        1. C.1. dvm_probe_test.d
        2. C.2. DVM Agent Provider Interface
      4. D. Sample Perl Kstat Utilities
        1. D.1. A Simple Kstat Walker
        2. D.2. A Perl Version of Uptime
        3. D.3. A Network Statistics Utility
        4. D.4. A Performance Utility for CPU, Memory, Disk, and Net
    10. Bibliography

    Product information

    • Title: Solaris™ Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris
    • Author(s): Richard McDougall, Jim Mauro, Brendan Gregg
    • Release date: July 2006
    • Publisher(s): Pearson
    • ISBN: 0131568191