book

How Linux Works, 2nd Edition

by Brian Ward

November 2014

Beginner to intermediate

392 pages

11h

English

No Starch Press

Read now

Unlock full access

Who Should Read This Book?PrerequisitesHow to Read This BookA Hands-On ApproachHow This Book is OrganizedWhat’s New in the Second Edition?A Note on Terminology
1.1 Levels and Layers of Abstraction in a Linux System1.2 Hardware: Understanding Main Memory1.3 The Kernel1.3.1 Process Management1.3.2 Memory Management1.3.3 Device Drivers and Management1.3.4 System Calls and Support1.4 User Space1.5 Users1.6 Looking Forward
2.1 The Bourne Shell: /bin/sh2.2 Using the Shell2.2.1 The Shell Window2.2.2 cat2.2.3 Standard Input and Standard Output2.3 Basic Commands2.3.1 ls2.3.2 cp2.3.3 mv2.3.4 touch2.3.5 rm2.3.6 echo2.4 Navigating Directories2.4.1 cd2.4.2 mkdir2.4.3 rmdir2.4.4 Shell Globbing (Wildcards)2.5 Intermediate Commands2.5.1 grep2.5.2 less2.5.3 pwd2.5.4 diff2.5.5 file2.5.6 find and locate2.5.7 head and tail2.5.8 sort2.6 Changing Your Password and Shell2.7 Dot Files2.8 Environment and Shell Variables2.9 The Command Path2.10 Special Characters2.11 Command-Line Editing2.12 Text Editors2.13 Getting Online Help2.14 Shell Input and Output2.14.1 Standard Error2.14.2 Standard Input Redirection2.15 Understanding Error Messages2.15.1 Anatomy of a UNIX Error Message2.15.2 Common ErrorsNo such file or directoryFile existsNot a directory, Is a directoryNo space left on devicePermission deniedOperation not permittedSegmentation fault, Bus error2.16 Listing and Manipulating Processes2.16.1 Command Options2.16.2 Killing Processes2.16.3 Job Control2.16.4 Background Processes2.17 File Modes and Permissions2.17.1 Modifying Permissions2.17.2 Symbolic Links2.17.3 Creating Symbolic Links2.18 Archiving and Compressing Files2.18.1 gzip2.18.2 tarUnpacking tar filesTable-of-Contents Mode2.18.3 Compressed Archives (.tar.gz)2.18.4 zcat2.18.5 Other Compression Utilities2.19 Linux Directory Hierarchy Essentials2.19.1 Other Root Subdirectories2.19.2 The /usr Directory2.19.3 Kernel Location2.20 Running Commands as the Superuser2.20.1 sudo2.20.2 /etc/sudoers2.21 Looking Forward
3.1 Device Files3.2 The sysfs Device Path3.3 dd and Devices3.4 Device Name Summary3.4.1 Hard Disks: /dev/sd*3.4.2 CD and DVD Drives: /dev/sr*3.4.3 PATA Hard Disks: /dev/hd*3.4.4 Terminals: /dev/tty*, /dev/pts/*, and /dev/ttyDisplay Modes and Virtual Consoles3.4.5 Serial Ports: /dev/ttyS*3.4.6 Parallel Ports: /dev/lp0 and /dev/lp13.4.7 Audio Devices: /dev/snd/*, /dev/dsp, /dev/audio, and More3.4.8 Creating Device Files3.5 udev3.5.1 devtmpfs3.5.2 udevd Operation and Configuration3.5.3 udevadm3.5.4 Monitoring Devices3.6 In-Depth: SCSI and the Linux Kernel3.6.1 USB Storage and SCSI3.6.2 SCSI and ATA3.6.3 Generic SCSI Devices3.6.4 Multiple Access Methods for a Single Device
4.1 Partitioning Disk Devices4.1.1 Viewing a Partition TableInitial Kernel Read4.1.2 Changing Partition Tables4.1.3 Disk and Partition Geometry4.1.4 Solid-State Disks (SSDs)4.2 Filesystems4.2.1 Filesystem Types4.2.2 Creating a Filesystem4.2.3 Mounting a Filesystem4.2.4 Filesystem UUID4.2.5 Disk Buffering, Caching, and Filesystems4.2.6 Filesystem Mount OptionsShort OptionsLong Options4.2.7 Remounting a Filesystem4.2.8 The /etc/fstab Filesystem Table4.2.9 Alternatives to /etc/fstab4.2.10 Filesystem Capacity4.2.11 Checking and Repairing FilesystemsChecking ext3 and ext4 FilesystemsThe Worst Case4.2.12 Special-Purpose Filesystems4.3 swap space4.3.1 Using a Disk Partition as Swap Space4.3.2 Using a File as Swap Space4.3.3 How Much Swap Do You Need?4.4 Looking Forward: Disks and User Space4.5 Inside a Traditional Filesystem4.5.1 Viewing Inode Details4.5.2 Working with Filesystems in User Space4.5.3 The Evolution of Filesystems
5.1 Startup Messages5.2 Kernel Initialization and Boot Options5.3 Kernel Parameters5.4 Boot Loaders5.4.1 Boot Loader Tasks5.4.2 Boot Loader Overview5.5 GRUB Introduction5.5.1 Exploring Devices and Partitions with the GRUB Command LineListing DevicesFile Navigation5.5.2 GRUB ConfigurationReviewing Grub.cfgGenerating a New Configuration File5.5.3 GRUB InstallationInstalling GRUB on Your SystemInstalling GRUB on an External Storage DeviceInstalling GRUB with UEFI5.6 UEFI Secure Boot Problems5.7 Chainloading Other Operating Systems5.8 Boot Loader Details5.8.1 MBR Boot5.8.2 UEFI Boot5.8.3 How GRUB Works
6.1 Introduction to init6.2 System V Runlevels6.3 Identifying Your init6.4 systemd6.4.1 Units and Unit Types6.4.2 systemd DependenciesOrderingConditional Dependencies6.4.3 systemd ConfigurationUnit FilesEnabling Units and the [Install] SectionVariables and Specifiers6.4.4 systemd Operation6.4.5 Adding Units to systemdRemoving Units6.4.6 systemd Process Tracking and Synchronization6.4.7 systemd On-Demand and Resource-Parallelized StartupBoot Optimization with Auxiliary UnitsAn Example Socket Unit and ServiceInstances and Handoff6.4.8 systemd System V Compatibility6.4.9 systemd Auxiliary Programs6.5 Upstart6.5.1 Upstart Initialization Procedure6.5.2 Upstart JobsViewing JobsJob State Transitions6.5.3 Upstart ConfigurationA Service Job: tty1Process Tracking and the Upstart expect Stanza6.5.4 Upstart Operation6.5.5 Upstart Logs6.5.6 Upstart Runlevels and System V Compatibility6.6 System V initrespawnctrlaltdelsysinit6.6.1 System V init: Startup Command Sequence6.6.2 The System V init Link FarmStarting and Stopping ServicesModifying the Boot Sequence6.6.3 run-parts6.6.4 Controlling System V init6.7 Shutting Down Your System6.8 The Initial RAM Filesystem6.9 Emergency Booting and Single-User Mode

7.1 The Structure of /etc7.2 System Logging7.2.1 The System Logger7.2.2 Configuration FilesFacility and PriorityExtended SyntaxTroubleshootingLogging: Past and Future7.3 User Management Files7.3.1 The /etc/passwd File7.3.2 Special Users7.3.3 The /etc/shadow File7.3.4 Manipulating Users and PasswordsChanging /etc/passwd as the Superuser7.3.5 Working with Groups7.4 getty and login7.5 Setting the Time7.5.1 Kernel Time Representation and Time Zones7.5.2 Network Time7.6 Scheduling Recurring Tasks with cron7.6.1 Installing Crontab Files7.6.2 System Crontab Files7.6.3 The Future of cron7.7 Scheduling One-Time Tasks with at7.8 Understanding User IDs and User Switching7.8.1 Process Ownership, Effective UID, Real UID, and Saved UIDTypical Setuid Program BehaviorSecurity Implications7.9 User Identification and Authentication7.9.1 Using Libraries for User Information7.10 PAM7.10.1 PAM ConfigurationFunction TypesControl Arguments and Stacked RulesModule Arguments7.10.2 Notes on PAM7.10.3 PAM and Passwords7.11 Looking Forward
8.1 Tracking Processes8.2 Finding Open Files with lsof8.2.1 Reading the lsof Output8.2.2 Using lsof8.3 Tracing Program Execution and System Calls8.3.1 strace8.3.2 ltrace8.4 Threads8.4.1 Single-Threaded and Multithreaded Processes8.4.2 Viewing Threads8.5 Introduction to Resource Monitoring8.6 Measuring CPU Time8.7 Adjusting Process Priorities8.8 Load Averages8.8.1 Using uptime8.8.2 High Loads8.9 Memory8.9.1 How Memory Works8.9.2 Page FaultsMinor Page FaultsMajor Page FaultsWatching Page Faults8.10 Monitoring CPU and Memory Performance with vmstat8.11 I/O Monitoring8.11.1 Using iostat8.11.2 Per-Process I/O Utilization and Monitoring: iotop8.12 Per-Process Monitoring with pidstat8.13 Further Topics
9.1 Network Basics9.1.1 Packets9.2 Network Layers9.3 The Internet Layer9.3.1 Viewing Your Computer’s IP Addresses9.3.2 Subnets9.3.3 Common Subnet Masks and CIDR Notation9.4 Routes and the Kernel Routing Table9.4.1 The Default Gateway9.5 Basic ICMP and DNS Tools9.5.1 ping9.5.2 traceroute9.5.3 DNS and host9.6 The Physical Layer and Ethernet9.7 Understanding Kernel Network Interfaces9.8 Introduction to Network Interface Configuration9.8.1 Manually Adding and Deleting Routes9.9 Boot-Activated Network Configuration9.10 Problems with Manual and Boot-Activated Network Configuration9.11 Network Configuration Managers9.11.1 NetworkManager Operation9.11.2 Interacting with NetworkManager9.11.3 NetworkManager ConfigurationUnmanaged InterfacesDispatching9.12 Resolving Hostnames9.12.1 /etc/hosts9.12.2 resolv.conf9.12.3 Caching and Zero-Configuration DNS9.12.4 /etc/nsswitch.conf9.13 Localhost9.14 The Transport Layer: TCP, UDP, and Services9.14.1 TCP Ports and Connections9.14.2 Establishing TCP Connections9.14.3 Port Numbers and /etc/services9.14.4 Characteristics of TCP9.14.5 UDP9.15 Revisiting a Simple Local Network9.16 Understanding DHCP9.16.1 The Linux DHCP Client9.16.2 Linux DHCP Servers9.17 Configuring Linux as a Router9.17.1 Internet Uplinks9.18 Private Networks9.19 Network Address Translation (IP Masquerading)9.20 Routers and Linux9.21 Firewalls9.21.1 Linux Firewall Basics9.21.2 Setting Firewall Rules9.21.3 Firewall Strategies9.22 Ethernet, IP, and ARP9.23 Wireless Ethernet9.23.1 iw9.23.2 Wireless Security9.24 Summary
10.1 The Basics of Services10.1.1 A Closer Look10.2 Network Servers10.3 Secure Shell (SSH)10.3.1 The SSHD ServerHost KeysStarting the SSH Server10.3.2 The SSH ClientSSH File Transfer ClientsSSH Clients for Non-Unix Platforms10.4 The inetd and xinetd Daemons10.4.1 TCP Wrappers: tcpd, /etc/hosts.allow, and /etc/hosts.deny10.5 Diagnostic Tools10.5.1 lsofFiltering by Protocol and PortFiltering by Connection Status10.5.2 tcpdumpPrimitivesOperatorsWhen Not to Use tcpdump10.5.3 netcat10.5.4 Port Scanning10.6 Remote Procedure Call (RPC)10.7 Network Security10.7.1 Typical Vulnerabilities10.7.2 Security Resources10.8 Looking Forward10.9 Sockets: How Processes Communicate with the Network10.10 Unix Domain Sockets10.10.1 Advantages for Developers10.10.2 Listing Unix Domain Sockets
11.1 Shell Script Basics11.1.1 Limitations of Shell Scripts11.2 Quoting and Literals11.2.1 Literals11.2.2 Single Quotes11.2.3 Double Quotes11.2.4 Passing a Literal Single Quote11.3 Special Variables11.3.1 Individual Arguments: $1, $2, ...11.3.2 Number of Arguments: $#11.3.3 All Arguments: $@11.3.4 Script Name: $011.3.5 Process ID: $$11.3.6 Exit Code: $?11.4 Exit Codes11.5 Conditionals11.5.1 Getting Around Empty Parameter Lists11.5.2 Using Other Commands for Tests11.5.3 elif11.5.4 && and || Logical Constructs11.5.5 Testing ConditionsFile TestsString TestsArithmetic Tests11.5.6 Matching Strings with case11.6 Loops11.6.1 for Loops11.6.2 while Loops11.7 Command Substitution11.8 Temporary File Management11.9 Here Documents11.10 Important Shell Script Utilities11.10.1 basename11.10.2 awk11.10.3 sed11.10.4 xargs11.10.5 expr11.10.6 exec11.11 Subshells11.12 Including Other Files in Scripts11.13 Reading User Input11.14 When (Not) to Use Shell Scripts
12.1 Quick Copy12.2 rsync12.2.1 rsync Basics12.2.2 Making Exact Copies of a Directory Structure12.2.3 Using the Trailing Slash12.2.4 Excluding Files and Directories12.2.5 Transfer Integrity, Safeguards, and Verbose Modes12.2.6 Compression12.2.7 Limiting Bandwidth12.2.8 Transferring Files to Your Computer12.2.9 Further rsync Topics12.3 Introduction to File Sharing12.4 Sharing Files with Samba12.4.1 Configuring the Server12.4.2 Server Access Control12.4.3 PasswordsAdding and Deleting UsersChanging Passwords12.4.4 Starting the Server12.4.5 Diagnostics and Log Files12.4.6 Configuring a File Share12.4.7 Home Directories12.4.8 Sharing Printers12.4.9 Using the Samba Client12.4.10 Accessing Files as a ClientUsing the CIFS Filesystem12.5 NFS Clients12.6 Further Network File Service Options and Limitations
13.1 Guidelines for Creating Startup Files13.2 When to Alter Startup Files13.3 Shell Startup File Elements13.3.1 The Command PathAdding Dot (.) to the Path13.3.2 The Manual Page Path13.3.3 The Prompt13.3.4 Aliases13.3.5 The Permissions Mask13.4 Startup File Order and Examples13.4.1 The bash ShellLogin ShellsNon-Login ShellsThe Consequences of Two Kinds of ShellsExample .bashrcChecking for Login and Interactive Shells13.4.2 The tcsh ShellExample .cshrc13.5 Default User Settings13.5.1 Shell Defaults13.5.2 Editor13.5.3 Pager13.6 Startup File Pitfalls13.7 Further Startup Topics
14.1 Desktop Components14.1.1 Window Managers14.1.2 Toolkits14.1.3 Desktop Environments14.1.4 Applications14.2 A Closer Look at the X Window System14.2.1 Display Managers14.2.2 Network Transparency14.3 Exploring X Clients14.3.1 X Events14.3.2 Understanding X Input and Preference SettingsInput Devices (General)MouseKeyboardDesktop Backgroundxset14.4 The Future of X14.5 D-Bus14.5.1 System and Session Instances14.5.2 Monitoring D-Bus Messages14.6 Printing14.6.1 CUPS14.6.2 Format Conversion and Print Filters14.7 Other Desktop Topics
15.1 The C Compiler15.1.1 Multiple Source Files15.1.2 Header (Include) Files and DirectoriesFixing Include File ProblemsWhat Is the C Preprocessor (cpp)?15.1.3 Linking with Libraries15.1.4 Shared LibrariesListing Shared Library DependenciesHow ld.so Finds Shared LibrariesLinking Programs Against Shared LibrariesProblems with Shared Libraries15.2 make15.2.1 A Sample Makefile15.2.2 Built-in Rules15.2.3 Final Program Build15.2.4 Staying Up-to-Date15.2.5 Command-Line Arguments and Options15.2.6 Standard Macros and Variables15.2.7 Conventional Targets15.2.8 Organizing a Makefile15.3 Debuggers15.4 Lex and Yacc15.5 Scripting Languages15.5.1 Python15.5.2 Perl15.5.3 Other Scripting Languages15.6 Java15.7 Looking Forward: Compiling Packages
16.1 Software Build Systems16.2 Unpacking C Source Packages16.2.1 Where to Start16.3 GNU Autoconf16.3.1 An Autoconf Example16.3.2 Installing Using a Packaging Tool16.3.3 configure Script OptionsUsing Separate Build Directories16.3.4 Environment Variables16.3.5 Autoconf Targets16.3.6 Autoconf Log Files16.3.7 pkg-configHow pkg-config WorksInstalling pkg-config Files in Nonstandard Locations16.4 Installation Practice16.4.1 Where to Install16.5 Applying a Patch16.6 Troubleshooting Compiles and Installations16.6.1 Specific Errors16.7 Looking Forward
17.1 Web Servers and Applications17.2 Databases17.2.1 Types of Databases17.3 Virtualization17.4 Distributed and On-Demand Computing17.5 Embedded Systems17.6 Final Remarks

Content preview from How Linux Works, 2nd Edition

Chapter 8. A Closer Look at Processes and Resource Utilization

This chapter takes you deeper into the relationships between processes, the kernel, and system resources. There are three basic kinds of hardware resources: CPU, memory, and I/O. Processes vie for these resources, and the kernel’s job is to allocate resources fairly. The kernel itself is also a resource—a software resource that processes use to perform tasks such as creating new processes and communicating with other processes.

Many of the tools that you see in this chapter are often thought of as performance-monitoring tools. They’re particularly helpful if your system is slowing to ...