book

Unix Power Tools, 3rd Edition

by Jerry Peek, Shelley Powers, Tim O'Reilly, Mike Loukides

October 2002

Intermediate to advanced

1154 pages

49h 36m

English

O'Reilly Media, Inc.

Read now

Unlock full access

UNIX Power Tools, 3rd Edition
A Note Regarding Supplemental Files
How to Use This Book
Preface
A Book for BrowsingLike an AlmanacLike a News MagazineLike a Hypertext DatabasePrograms on the WebAbout Unix VersionsCross-ReferencesWhat’s New in the Third EditionTypefaces and Other ConventionsThe AuthorsThe Fine PrintRequest for CommentsAcknowledgments for the First EditionAcknowledgments for the Second EditionAcknowledgments for the Third Edition
I. Basic Unix Environment
1. Introduction
1.1. What’s Special About Unix?1.2. Power Grows on You1.3. The Core of Unix1.4. Communication with Unix1.5. Programs Are Designed to Work Together1.6. There Are Many Shells1.7. Which Shell Am I Running?1.8. Anyone Can Program the Shell1.9. Internal and External Commands1.10. The Kernel and Daemons1.11. Filenames1.12. Filename Extensions1.13. Wildcards1.14. The Tree Structure of the Filesystem1.15. Your Home Directory1.16. Making Pathnames1.17. File Access Permissions1.18. The Superuser (Root)1.19. When Is a File Not a File?1.20. Scripting1.21. Unix Networking and Communications1.22. The X Window System
2. Getting Help
2.1. The man Command2.2. whatis: One-Line Command Summaries2.3. whereis: Finding Where a Command Is Located2.4. Searching Online Manual Pages2.5. How Unix Systems Remember Their Names2.6. Which Version Am I Using?2.7. What tty Am I On?2.8. Who’s On?2.9. The info Command
II. Customizing Your Environment
3. Setting Up Your Unix Shell
3.1. What Happens When You Log In3.2. The Mac OS X Terminal Application3.3. Shell Setup Files — Which, Where, and Why3.4. Login Shells, Interactive Shells3.4.1. Login Shells3.4.2. Interactive Shells3.5. What Goes in Shell Setup Files?3.6. Tip for Changing Account Setup: Keep a Shell Ready3.7. Use Absolute Pathnames in Shell Setup Files3.8. Setup Files Aren’t Read When You Want?3.9. Gotchas in set prompt Test3.10. Automatic Setups for Different Terminals3.11. Terminal Setup: Testing TERM3.12. Terminal Setup: Testing Remote Hostname and X Display3.13. Terminal Setup: Testing Port3.14. Terminal Setup: Testing Environment Variables3.15. Terminal Setup: Searching Terminal Table3.16. Terminal Setup: Testing Window Size3.17. Terminal Setup: Setting and Testing Window Name3.18. A .cshrc.$HOST File for Per Host Setup3.19. Making a “Login” Shell3.20. RC Files3.21. Make Your Own Manpages Without Learning troff3.22. Writing a Simple Manpage with the -man Macros
4. Interacting with Your Environment
4.1. Basics of Setting the Prompt4.2. Static Prompts4.3. Dynamic Prompts4.4. Simulating Dynamic Prompts4.5. C-Shell Prompt Causes Problems in vi, rsh, etc.4.6. Faster Prompt Setting with Built-ins4.7. Multiline Shell Prompts4.8. Session Info in Window Title or Status Line4.9. A “Menu Prompt” for Naive Users4.10. Highlighting and Color in Shell Prompts4.11. Right-Side Prompts4.12. Show Subshell Level with $SHLVL4.13. What Good Is a Blank Shell Prompt?4.14. dirs in Your Prompt: Better Than $cwd4.15. External Commands Send Signals to Set Variables4.16. Preprompt, Pre-execution, and Periodic Commands4.17. Running Commands When You Log Out4.18. Running Commands at Bourne/Korn Shell Logout4.19. Stop Accidental Bourne-Shell Logouts

5. Getting the Most out of Terminals, xterm, and X Windows
5.1. There’s a Lot to Know About Terminals5.2. The Idea of a Terminal Database5.3. Setting the Terminal Type When You Log In5.4. Querying Your Terminal Type: qterm5.5. Querying Your xterm Size: resize5.6. Checklist: Terminal Hangs When I Log In5.6.1. Output Stopped?5.6.2. Job Stopped?5.6.3. Program Waiting for Input?5.6.4. Stalled Data Connection?5.6.5. Aborting Programs5.7. Find Out Terminal Settings with stty5.8. Setting Your Erase, Kill, and Interrupt Characters5.9. Working with xterm and Friends5.10. Login xterms and rxvts5.11. Working with Scrollbars5.12. How Many Lines to Save?5.13. Simple Copy and Paste in xterm5.14. Defining What Makes Up a Word for Selection Purposes5.15. Setting the Titlebar and Icon Text5.16. The Simple Way to Pick a Font5.17. The xterm Menus5.18. Changing Fonts Dynamically5.18.1. VT Fonts Menu5.18.2. Enabling Escape Sequence and Selection5.19. Working with xclipboard5.20. Problems with Large Selections5.21. Tips for Copy and Paste Between Windows5.22. Running a Single Command with xterm -e5.23. Don’t Quote Arguments to xterm -e
6. Your X Environment
6.1. Defining Keys and Button Presses with xmodmap6.2. Using xev to Learn Keysym Mappings6.3. X Resource Syntax6.4. X Event Translations6.5. Setting X Resources: Overview6.6. Setting Resources with the -xrm Option6.7. How -name Affects Resources6.8. Setting Resources with xrdb6.9. Listing the Current Resources for a Client: appres6.10. Starting Remote X Clients6.10.1. Starting Remote X Clients from Interactive Logins6.10.2. Starting a Remote Client with rsh and ssh
III. Working with Files and Directories
7. Directory Organization
7.1. What? Me, Organized?7.2. Many Homes7.3. Access to Directories7.4. A bin Directory for Your Programs and Scripts7.5. Private (Personal) Directories7.6. Naming Files7.7. Make More Directories!7.8. Making Directories Made Easier
8. Directories and Files
8.1. Everything but the find Command8.2. The Three Unix File Times8.3. Finding Oldest or Newest Files with ls -t and ls -u8.4. List All Subdirectories with ls -R8.5. The ls -d Option8.6. Color ls8.6.1. Trying It8.6.2. Configuring It8.6.3. The -- color Option8.6.4. Another color ls8.7. Some GNU ls Features8.8. A csh Alias to List Recently Changed Files8.9. Showing Hidden Files with ls -A and -a8.10. Useful ls Aliases8.11. Can’t Access a File? Look for Spaces in the Name8.12. Showing Nonprintable Characters in Filenames8.13. Counting Files by Types8.14. Listing Files by Age and Size8.15. newer: Print the Name of the Newest File8.16. oldlinks: Find Unconnected Symbolic Links8.17. Picking a Unique Filename Automatically
9. Finding Files with find
9.1. How to Use find9.2. Delving Through a Deep Directory Tree9.3. Don’t Forget -print9.4. Looking for Files with Particular Names9.5. Searching for Old Files9.6. Be an Expert on find Search Operators9.7. The Times That find Finds9.8. Exact File-Time Comparisons9.9. Running Commands on What You Find9.10. Using -exec to Create Custom Tests9.11. Custom -exec Tests Applied9.12. Finding Many Things with One Command9.13. Searching for Files by Type9.14. Searching for Files by Size9.15. Searching for Files by Permission9.16. Searching by Owner and Group9.17. Duplicating a Directory Tree9.18. Using “Fast find” Databases9.19. Wildcards with “Fast find” Database9.20. Finding Files (Much) Faster with a find Database9.21. grepping a Directory Tree9.22. lookfor: Which File Has That Word?9.23. Using Shell Arrays to Browse Directories9.23.1. Using the Stored Lists9.23.2. Expanding Ranges9.24. Finding the (Hard) Links to a File9.25. Finding Files with -prune9.26. Quick finds in the Current Directory9.27. Skipping Parts of a Tree in find9.28. Keeping find from Searching Networked Filesystem
10. Linking, Renaming, and Copying Files
10.1. What’s So Complicated About Copying Files10.2. What’s Really in a Directory?10.3. Files with Two or More Names10.4. More About Links10.4.1. Differences Between Hard and Symbolic Links10.4.2. Links to a Directory10.5. Creating and Removing Links10.6. Stale Symbolic Links10.7. Linking Directories10.8. Showing the Actual Filenames for Symbolic Links10.9. Renaming, Copying, or Comparing a Set of Files10.10. Renaming a List of Files Interactively10.11. One More Way to Do It10.12. Copying Directory Trees with cp -r10.13. Copying Directory Trees with tar and Pipes
11. Comparing Files
11.1. Checking Differences with diff11.2. Comparing Three Different Versions with diff311.3. Context diffs11.4. Side-by-Side diffs: sdiff11.5. Choosing Sides with sdiff11.6. Problems with diff and Tabstops11.7. cmp and diff11.8. Comparing Two Files with comm11.9. More Friendly comm Output11.10. make Isn’t Just for Programmers!11.11. Even More Uses for make
12. Showing What’s in a File
12.1. Cracking the Nut12.2. What Good Is a cat?12.3. “less” is More12.4. Show Nonprinting Characters with cat -v or od -c12.5. What’s in That Whitespace?12.6. Finding File Types12.7. Squash Extra Blank Lines12.8. How to Look at the End of a File: tail12.9. Finer Control on tail12.10. How to Look at Files as They Grow12.11. GNU tail File Following12.12. Printing the Top of a File12.13. Numbering Lines
13. Searching Through Files
13.1. Different Versions of grep13.2. Searching for Text with grep13.3. Finding Text That Doesn’t Match13.4. Extended Searching for Text with egrep13.5. grepping for a List of Patterns13.6. Approximate grep: agrep13.7. Search RCS Files with rcsgrep13.7.1. rcsgrep, rcsegrep, rcsfgrep13.7.2. rcsegrep.fast13.8. GNU Context greps13.9. A Multiline Context grep Using sed13.10. Compound Searches13.11. Narrowing a Search Quickly13.12. Faking Case-Insensitive Searches13.13. Finding a Character in a Column13.14. Fast Searches and Spelling Checks with “look”13.15. Finding Words Inside Binary Files13.16. A Highlighting grep
14. Removing Files
14.1. The Cycle of Creation and Destruction14.2. How Unix Keeps Track of Files: Inodes14.3. rm and Its Dangers14.4. Tricks for Making rm Safer14.5. Answer “Yes” or “No” Forever with yes14.6. Remove Some, Leave Some14.7. A Faster Way to Remove Files Interactively14.8. Safer File Deletion in Some Directories14.9. Safe Delete: Pros and Cons14.10. Deletion with Prejudice: rm -f14.11. Deleting Files with Odd Names14.12. Using Wildcards to Delete Files with Strange Names14.13. Handling a Filename Starting with a Dash (-)14.14. Using unlink to Remove a File with a Strange Name14.15. Removing a Strange File by its i-number14.16. Problems Deleting Directories14.17. Deleting Stale Files14.18. Removing Every File but One14.19. Using find to Clear Out Unneeded Files
15. Optimizing Disk Space
15.1. Disk Space Is Cheap15.2. Instead of Removing a File, Empty It15.3. Save Space with “Bit Bucket” Log Files and Mailboxes15.4. Save Space with a Link15.5. Limiting File Sizes15.5.1. limit and ulimit15.5.2. Other Ideas15.6. Compressing Files to Save Space15.7. Save Space: tar and compress a Directory Tree15.8. How Much Disk Space?15.9. Compressing a Directory Tree: Fine-Tuning15.10. Save Space in Executable Files with strip15.11. Disk Quotas
IV. Basic Editing
16. Spell Checking, Word Counting, and Textual Analysis
16.1. The Unix spell Command16.2. Check Spelling Interactively with ispell16.3. How Do I Spell That Word?16.4. Inside spell16.5. Adding Words to ispell’s Dictionary16.6. Counting Lines, Words, and Characters: wc16.7. Find a a Doubled Word16.8. Looking for Closure16.9. Just the Words, Please
17. vi Tips and Tricks
17.1. The vi Editor: Why So Much Material?17.2. What We Cover17.3. Editing Multiple Files with vi17.4. Edits Between Files17.5. Local Settings for vi17.6. Using Buffers to Move or Copy Text17.7. Get Back What You Deleted with Numbered Buffers17.8. Using Search Patterns and Global Commands17.8.1. Global Searches17.9. Confirming Substitutions in vi17.10. Keep Your Original File, Write to a New File17.11. Saving Part of a File17.12. Appending to an Existing File17.13. Moving Blocks of Text by Patterns17.14. Useful Global Commands (with Pattern Matches)17.15. Counting Occurrences; Stopping Search Wraps17.16. Capitalizing Every Word on a Line17.17. Per-File Setups in Separate Files17.18. Filtering Text Through a Unix Command17.19. vi File Recovery Versus Networked Filesystems17.20. Be Careful with vi -r Recovered Buffers17.21. Shell Escapes: Running One UnixCommand While Using Another17.22. vi Compound Searches17.23. vi Word Abbreviation17.24. Using vi Abbreviations as Commands (Cut and Paste Between vi’s)17.25. Fixing Typos with vi Abbreviations17.26. vi Line Commands Versus Character Commands17.27. Out of Temporary Space? Use Another Directory17.28. Neatening Lines17.29. Finding Your Place with Undo17.30. Setting Up vi with the .exrc File
18. Creating Custom Commands in vi
18.1. Why Type More Than You Have To?18.2. Save Time and Typing with the vi map Commands18.2.1. Command Mode Maps18.2.2. Text-Input Mode Maps18.3. What You Lose When You Use map!18.4. vi @-Functions18.4.1. Defining and Using Simple @-Functions18.4.2. Combining @-Functions18.4.3. Reusing a Definition18.4.4. Newlines in an @-Function18.5. Keymaps for Pasting into a Window Running vi18.6. Protecting Keys from Interpretation by ex18.7. Maps for Repeated Edits18.8. More Examples of Mapping Keys in vi18.9. Repeating a vi Keymap18.10. Typing in Uppercase Without CAPS LOCK18.11. Text-Input Mode Cursor Motion with No Arrow Keys18.12. Don’t Lose Important Functions with vi Maps: Use noremap18.13. vi Macro for Splitting Long Lines18.14. File-Backup Macros
19. GNU Emacs
19.1. Emacs: The Other Editor19.2. Emacs Features: A Laundry List19.3. Customizations and How to Avoid Them19.4. Backup and Auto-Save Files19.5. Putting Emacs in Overwrite Mode19.6. Command Completion19.7. Mike’s Favorite Timesavers19.8. Rational Searches19.9. Unset PWD Before Using Emacs19.10. Inserting Binary Characters into Files19.11. Using Word-Abbreviation Mode19.11.1. Trying Word Abbreviations for One Session19.11.2. Making Word Abbreviations Part of Your Startup19.12. Directories for Emacs Hacks19.13. An Absurd Amusement
20. Batch Editing
20.1. Why Line Editors Aren’t Dinosaurs20.2. Writing Editing Scripts20.3. Line Addressing20.4. Useful ex Commands20.5. Running Editing Scripts Within vi20.6. Change Many Files by Editing Just One20.7. ed/ex Batch Edits: A Typical Example20.8. Batch Editing Gotcha: Editors Fail on Big Files20.9. patch: Generalized Updating of Files That Differ20.10. Quick Reference: awk20.10.1. Command-Line Syntax20.10.2. Patterns and Procedures20.10.2.1. Patterns20.10.2.2. Procedures20.10.2.3. Simple pattern-procedure examples20.10.3. awk System Variables20.10.4. Operators20.10.5. Variables and Array Assignments20.10.6. Group Listing of awk Commands20.10.7. Alphabetical Summary of Commands20.11. Versions of awk
21. You Can’t Quite Call This Editing
21.1. And Why Not?21.2. Neatening Text with fmt21.3. Alternatives to fmt21.4. Clean Up Program Comment Blocks21.4.1. The recomment Script21.4.2. fmt -p21.5. Remove Mail/News Headers with behead21.6. Low-Level File Butchery with dd21.7. offset: Indent Text21.8. Centering Lines in a File21.9. Splitting Files at Fixed Points: split21.10. Splitting Files by Context: csplit21.11. Hacking on Characters with tr21.12. Encoding “Binary” Files into ASCII21.12.1. uuencoding21.12.2. MIME Encoding21.13. Text Conversion with dd21.14. Cutting Columns or Fields21.15. Making Text in Columns with pr21.15.1. One File per Column: -m21.15.2. One File, Several Columns: -number21.15.3. Order Lines Across Columns: -l21.16. Make Columns Automatically with column21.17. Straightening Jagged Columns21.18. Pasting Things in Columns21.19. Joining Lines with join21.20. What Is (or Isn’t) Unique?21.21. Rotating Text
22. Sorting
22.1. Putting Things in Order22.2. Sort Fields: How sort Sorts22.3. Changing the sort Field Delimiter22.4. Confusion with Whitespace Field Delimiters22.5. Alphabetic and Numeric Sorting22.6. Miscellaneous sort Hints22.6.1. Dealing with Repeated Lines22.6.2. Ignoring Blanks22.6.3. Case-Insensitive Sorts22.6.4. Dictionary Order22.6.5. Month Order22.6.6. Reverse Sort22.7. lensort: Sort Lines by Length22.8. Sorting a List of People by Last Name
V. Processes and the Kernel
23. Job Control
23.1. Job Control in a Nutshell23.2. Job Control Basics23.2.1. How Job Control Works23.2.2. Using Job Control from Your Shell23.3. Using jobs Effectively23.4. Some Gotchas with Job Control23.5. The “Current Job” Isn’t Always What You Expect23.6. Job Control and autowrite: Real Timesavers!23.7. System Overloaded? Try Stopping Some Jobs23.8. Notification When Jobs Change State23.9. Stop Background Output with stty tostop23.10. nohup23.11. Disowning Processes23.12. Linux Virtual Consoles23.12.1. What Are They?23.12.2. Scrolling, Using a Mouse23.13. Stopping Remote Login Sessions
24. Starting, Stopping, and Killing Processes
24.1. What’s in This Chapter24.2. fork and exec24.3. Managing Processes: Overall Concepts24.4. Subshells24.5. The ps Command24.6. The Controlling Terminal24.7. Tracking Down Processes24.7.1. System V24.7.2. BSD24.8. Why ps Prints Some Commands in Parentheses24.9. The /proc Filesystem24.9.1. Memory Information24.9.2. Kernel and System Statistics24.9.3. Statistics of the Current Process24.9.4. Statistics of Processes by PID24.9.5. A Glimpse at Hardware24.10. What Are Signals?24.11. Killing Foreground Jobs24.12. Destroying Processes with kill24.13. Printer Queue Watcher: A Restartable Daemon Shell Script24.14. Killing All Your Processes24.15. Killing Processes by Name?24.16. Kill Processes Interactively24.16.1. killall -i24.16.2. zap24.17. Processes Out of Control? Just STOP Them24.18. Cleaning Up an Unkillable Process24.19. Why You Can’t Kill a Zombie24.20. The Process Chain to Your Window24.21. Terminal Windows Without Shells24.22. Close a Window by Killing Its Process(es)24.22.1. Example #1: An xterm Window24.22.2. Example #2: A Web Browser24.22.3. Closing a Window from a Shell Script
25. Delayed Execution
25.1. Building Software Robots the Easy Way25.2. Periodic Program Execution: The cron Facility25.2.1. Execution Scheduling25.2.2. A Little Help, etc.25.3. Adding crontab Entries25.4. Including Standard Input Within a cron Entry25.5. The at Command25.6. Making Your at Jobs Quiet25.7. Checking and Removing Jobs25.8. Avoiding Other at and cron Jobs25.9. Waiting a Little While: sleep
26. System Performance and Profiling
26.1. Timing Is Everything26.2. Timing Programs26.3. What Commands Are Running and How Long Do They Take?26.4. Checking System Load: uptime26.5. Know When to Be “nice” to Other Users...and When Not To26.5.1. BSD C Shell nice26.5.2. BSD Standalone nice26.5.3. System V C Shell nice26.5.4. System V Standalone nice26.6. A nice Gotcha26.7. Changing a Running Job’s Niceness
VI. Scripting
27. Shell Interpretation
27.1. What the Shell Does27.2. How the Shell Executes Other Commands27.3. What’s a Shell, Anyway?27.3.1. How Shells Run Other Programs27.3.2. Interactive Use Versus Shell Scripts27.3.3. Types of Shells27.3.4. Shell Search Paths27.3.5. Bourne Shell Used Here27.3.6. Default Commands27.4. Command Evaluation and Accidentally Overwriting Files27.5. Output Command-Line Arguments One by One27.6. Controlling Shell Command Searches27.7. Wildcards Inside Aliases27.8. eval: When You Need Another Chance27.9. Which One Will bash Use?27.10. Which One Will the C Shell Use?27.11. Is It “2>&1 file” or “> file 2>&1”? Why?27.12. Bourne Shell Quoting27.12.1. Special Characters27.12.2. How Quoting Works27.12.3. Single Quotes Inside Single Quotes?27.12.4. Multiline Quoting27.13. Differences Between Bourne and C Shell Quoting27.13.1. Special Characters27.13.2. How Quoting Works27.14. Quoting Special Characters in Filenames27.15. Verbose and Echo Settings Show Quoting27.16. Here Documents27.17. “Special” Characters and Operators27.18. How Many Backslashes?
28. Saving Time on the Command Line
28.1. What’s Special About the Unix Command Line28.2. Reprinting Your Command Line with CTRL-r28.3. Use Wildcards to Create Files?28.4. Build Strings with { }28.5. String Editing (Colon) Operators28.6. Automatic Completion28.6.1. General Example: Filename Completion28.6.2. Menu Completion28.6.3. Command-Specific Completion28.6.4. Editor Functions for Completion28.7. Don’t Match Useless Files in Filename Completion28.8. Repeating Commands28.9. Repeating and Varying Commands28.9.1. A foreach Loop28.9.2. A for Loop28.10. Repeating a Command with Copy-and-Paste28.11. Repeating a Time-Varying Command28.12. Multiline Commands, Secondary Prompts28.13. Here Document Example #1: Unformatted Form Letters28.14. Command Substitution28.15. Handling Lots of Text with Temporary Files28.16. Separating Commands with Semicolons28.17. Dealing with Too Many Arguments28.18. Expect28.18.1. Dialback28.18.2. Automating /bin/passwd28.18.3. Testing: A Story28.18.4. Other Problems
29. Custom Commands
29.1. Creating Custom Commands29.2. Introduction to Shell Aliases29.3. C-Shell Aliases with Command-Line Arguments29.4. Setting and Unsetting Bourne-Type Aliases29.5. Korn-Shell Aliases29.6. zsh Aliases29.7. Sourceable Scripts29.8. Avoiding C-Shell Alias Loops29.9. How to Put if-then-else in a C-Shell Alias29.10. Fix Quoting in csh Aliases with makealias and quote29.11. Shell Function Basics29.11.1. Simple Functions: ls with Options29.11.2. Functions with Loops: Internet Lookup29.11.3. Setting Current Shell Environment: The work Function29.11.4. Functions Calling Functions: Factorials29.11.5. Conclusion29.12. Shell Function Specifics29.13. Propagating Shell Functions29.13.1. Exporting bash Functions29.13.2. FPATH Search Path29.13.2.1. Korn shell29.13.2.2. zsh29.14. Simulated Bourne Shell Functions and Aliases
30. The Use of History
30.1. The Lessons of History30.2. History in a Nutshell30.3. My Favorite Is !$30.4. My Favorite Is !:n*30.5. My Favorite Is ^^30.6. Using !$ for Safety with Wildcards30.7. History by Number30.8. History Substitutions30.9. Repeating a Cycle of Commands30.10. Running a Series of Commands on a File30.11. Check Your History First with :p30.12. Picking Up Where You Left Off30.12.1. bash, ksh, zsh30.12.2. C Shells30.13. Pass History to Another Shell30.14. Shell Command-Line Editing30.14.1. vi Editing Mode30.14.2. Emacs Editing Mode30.14.3. tcsh Editing30.14.4. ksh Editing30.14.5. bash Editing30.14.6. zsh Editing30.15. Changing History Characters with histchars30.16. Instead of Changing History Characters
31. Moving Around in a Hurry
31.1. Getting Around the Filesystem31.2. Using Relative and Absolute Pathnames31.3. What Good Is a Current Directory?31.4. How Does Unix Find Your Current Directory?31.5. Saving Time When You Change Directories: cdpath31.6. Loop Control: break and continue31.7. The Shells’ pushd and popd Commands31.8. Nice Aliases for pushd31.9. Quick cds with Aliases31.10. cd by Directory Initials31.11. Finding (Anyone’s) Home Directory, Quickly31.12. Marking Your Place with a Shell Variable31.13. Automatic Setup When You Enter/Exit a Directory
32. Regular Expressions (Pattern Matching)
32.1. That’s an Expression32.2. Don’t Confuse Regular Expressions with Wildcards32.3. Understanding Expressions32.4. Using Metacharacters in Regular Expressions32.5. Regular Expressions: The Anchor Characters ^ and $32.6. Regular Expressions: Matching a Character with a Character Set32.7. Regular Expressions: Match Any Character with . (Dot)32.8. Regular Expressions: Specifying a Range of Characters with [...]32.9. Regular Expressions: Exceptions in a Character Set32.10. Regular Expressions: Repeating Character Sets with *32.11. Regular Expressions: Matching a Specific Number of Sets with \ { and \ }32.12. Regular Expressions: Matching Words with \ < and \ >32.13. Regular Expressions: Remembering Patterns with \ (, \ ), and \132.14. Regular Expressions: Potential Problems32.15. Extended Regular Expressions32.16. Getting Regular Expressions Right32.17. Just What Does a Regular Expression Match?32.18. Limiting the Extent of a Match32.19. I Never Meta Character I Didn’t Like32.20. Valid Metacharacters for Different Unix Programs32.21. Pattern Matching Quick Reference with Examples32.21.1. Examples of Searching32.21.2. Examples of Searching and Replacing
33. Wildcards
33.1. File-Naming Wildcards33.2. Filename Wildcards in a Nutshell33.3. Who Handles Wildcards?33.4. What if a Wildcard Doesn’t Match?33.5. Maybe You Shouldn’t Use Wildcards in Pathnames33.6. Getting a List of Matching Files with grep -l33.7. Getting a List of Nonmatching Files33.7.1. Using grep -c33.7.2. The vgrep Script33.8. nom: List Files That Don’t Match a Wildcard
34. The sed Stream Editor
34.1. sed Sermon^H^H^H^H^H^HSummary34.2. Two Things You Must Know About sed34.3. Invoking sed34.4. Testing and Using a sed Script: checksed, runsed34.4.1. checksed34.4.2. runsed34.5. sed Addressing Basics34.6. Order of Commands in a Script34.7. One Thing at a Time34.8. Delimiting a Regular Expression34.9. Newlines in a sed Replacement34.10. Referencing the Search String in a Replacement34.11. Referencing Portions of a Search String34.12. Search and Replacement: One Match Among Many34.13. Transformations on Text34.14. Hold Space: The Set-Aside Buffer34.15. Transforming Part of a Line34.16. Making Edits Across Line Boundaries34.17. The Deliberate Scrivener34.18. Searching for Patterns Split Across Lines34.19. Multiline Delete34.20. Making Edits Everywhere Except...34.21. The sed Test Command34.22. Uses of the sed Quit Command34.23. Dangers of the sed Quit Command34.24. sed Newlines, Quoting, and Backslashes in a Shell Script
35. Shell Programming for the Uninitiated
35.1. Writing a Simple Shell Program35.2. Everyone Should Learn Some Shell Programming35.3. What Environment Variables Are Good For35.4. Parent-Child Relationships35.5. Predefined Environment Variables35.6. The PATH Environment Variable35.7. PATH and path35.8. The DISPLAY Environment Variable35.9. Shell Variables35.10. Test String Values with Bourne-Shell case35.11. Pattern Matching in case Statements35.12. Exit Status of Unix Processes35.13. Test Exit Status with the if Statement35.14. Testing Your Success35.15. Loops That Test Exit Status35.15.1. Looping Until a Command Succeeds35.15.2. Looping Until a Command Fails35.16. Set Exit Status of a Shell (Script)35.17. Trapping Exits Caused by Interrupts35.18. read: Reading from the Keyboard35.19. Shell Script “Wrappers” for awk, sed, etc.35.20. Handling Command-Line Arguments in Shell Scripts35.20.1. With the “$@” Parameter35.20.2. With a Loop35.20.3. Counting Arguments with $#35.21. Handling Command-Line Arguments with a for Loop35.22. Handling Arguments with while and shift35.23. Loop Control: break and continue35.24. Standard Command-Line Parsing35.25. The Bourne Shell set Command35.25.1. Setting Options35.25.2. Setting (and Parsing) Parameters35.25.3. (Avoiding?) set with No Arguments35.25.4. Watch Your Quoting35.25.5. Can’t Set $035.26. test: Testing Files and Strings35.27. Picking a Name for a New Command35.28. Finding a Program Name and Giving Your Program Multiple Names35.29. Reading Files with the . and source Commands35.30. Using Shell Functions in Shell Scripts
36. Shell Programming for the Initiated
36.1. Beyond the Basics36.2. The Story of : # #!36.3. Don’t Need a Shell for Your Script? Don’t Use One36.4. Making #! Search the PATH36.5. The exec Command36.6. The Unappreciated Bourne Shell “:” Operator36.7. Parameter Substitution36.8. Save Disk Space and Programming: Multiple Names for a Program36.9. Finding the Last Command-Line Argument36.10. How to Unset All Command-Line Parameters36.11. Standard Input to a for Loop36.12. Making a for Loop with Multiple Variables36.13. Using basename and dirname36.13.1. Introduction to basename and dirname36.13.2. Use with Loops36.14. A while Loop with Several Loop Control Commands36.15. Overview: Open Files and File Descriptors36.16. n>&m: Swap Standard Output and Standard Error36.17. A Shell Can Read a Script from Its Standard Input, but...36.18. Shell Scripts On-the-Fly from Standard Input36.19. Quoted hereis Document Terminators: sh Versus csh36.20. Turn Off echo for “Secret” Answers36.21. Quick Reference: expr36.21.1. Syntax36.21.2. Examples36.22. Testing Characters in a String with expr36.23. Grabbing Parts of a String36.23.1. Matching with expr36.23.2. Using echo with awk or cut36.23.3. Using set and IFS36.23.4. Using sed36.24. Nested Command Substitution36.25. Testing Two Strings with One case Statement36.26. Outputting Text to an X Window36.27. Shell Lockfile
37. Shell Script Debugging and Gotchas
37.1. Tips for Debugging Shell Scripts37.1.1. Use -xv37.1.2. Unmatched Operators37.1.3. Exit Early37.1.4. Missing or Extra esac, ;;, fi, etc.37.1.5. Line Numbers Reset Inside Redirected Loops37.2. Bourne Shell Debugger Shows a Shell Variable37.3. Stop Syntax Errors in Numeric Tests37.4. Stop Syntax Errors in String Tests37.5. Quoting and Command-Line Parameters37.6. How Unix Keeps Time37.7. Copy What You Do with script37.8. Cleaning script Files37.9. Making an Arbitrary-Size File for Testing
VII. Extending and Managing Your Environment
38. Backing Up Files
38.1. What Is This “Backup” Thing?38.2. tar in a Nutshell38.3. Make Your Own Backups38.4. More Ways to Back Up38.5. How to Make Backups to a Local Device38.5.1. What to Back Up38.5.2. Backing Up to Tape38.5.3. Backing Up to Floppies or Zip Disks38.5.4. To gzip, or Not to gzip?38.6. Restoring Files from Tape with tar38.6.1. Restoring a Few Files38.6.2. Remote Restoring38.7. Using tar to a Remote Tape Drive38.8. Using GNU tar with a Remote Tape Drive38.9. On-Demand Incremental Backups of a Project38.10. Using Wildcards with tar38.10.1. Without GNU tar38.10.2. With GNU tar38.10.3. Wildcard Gotchas in GNU tar38.11. Avoid Absolute Paths with tar38.12. Getting tar’s Arguments in the Right Order38.13. The cpio Tape Archiver38.14. Industrial Strength Backups
39. Creating and Reading Archives
39.1. Packing Up and Moving39.2. Using tar to Create and Unpack Archives39.3. GNU tar Sampler39.4. Managing and Sharing Files with RCS and CVS39.5. RCS Basics39.6. List RCS Revision Numbers with rcsrevs39.7. CVS Basics39.8. More CVS
40. Software Installation
40.1. /usr/bin and Other Software Directories40.2. The Challenges of Software Installation on Unix40.3. Which make?40.4. Simplifying the make Process40.5. Using Debian’s dselect40.5.1. Choosing the Access Method40.5.2. Updating Information on Available Packages40.5.3. Choosing Packages for Installation or Removal40.5.4. Exiting the Select Function40.5.5. Installing Packages40.5.6. Configuring Packages40.5.7. Removing Packages40.5.8. Exiting dselect40.6. Installing Software with Debian’s Apt-Get40.6.1. Configuring the sources.list File40.6.2. Using apt-get40.6.2.1. Updating information on available packages40.6.2.2. Installing a package40.6.2.3. Upgrading installed packages40.7. Interruptable gets with wget40.8. The curl Application and One-Step GNU-Darwin Auto-Installer for OS X40.9. Installation with FreeBSD Ports40.10. Installing with FreeBSD Packages40.11. Finding and Installing RPM Packaged Software
41. Perl
41.1. High-Octane Shell Scripting41.2. Checking your Perl Installation41.3. Compiling Perl from Scratch41.4. Perl Boot Camp, Part 1: Typical Script Anatomy41.5. Perl Boot Camp, Part 2: Variables and Data Types41.5.1. Scalars41.5.2. Arrays41.5.3. Hashes41.5.4. References41.6. Perl Boot Camp, Part 3: Branching and Looping41.7. Perl Boot Camp, Part 4: Pattern Matching41.8. Perl Boot Camp, Part 5: Perl Knows Unix41.9. Perl Boot Camp, Part 6: Modules41.10. Perl Boot Camp, Part 7: perldoc41.11. CPAN41.11.1. Installing Modules the Easy Way41.11.2. Installing Modules the Hard Way41.11.3. Browsing the CPAN Web Site41.12. Make Custom grep Commands (etc.) with Perl41.13. Perl and the Internet41.13.1. Be Your Own Web Browser with LWP41.13.2. Sending Mail with Mail::Sendmail41.13.3. CGI Teaser
42. Python
42.1. What Is Python?42.2. Installation and Distutils42.3. Python Basics42.3.1. Indentation42.3.2. Functions42.3.3. Everything’s an Object42.3.4. Modules and Packages42.3.5. I/O and Formatting42.3.6. wxPython42.4. Python and the Web42.5. urllib42.6. urllib242.7. htmllib and HTMLParser42.8. cgi42.9. mod_python42.10. What About Perl?
VIII. Communication and Connectivity
43. Redirecting Input and Output
43.1. Using Standard Input and Output43.2. One Argument with a cat Isn’t Enough43.3. Send (Only) Standard Error Down a Pipe43.4. Problems Piping to a Pager43.5. Redirection in C Shell: Capture Errors, Too?43.6. Safe I/O Redirection with noclobber43.7. The ( ) Subshell Operators43.7.1. Combining Several Commands43.7.2. Temporary Change of Directory and Environment43.8. Send Output Two or More Places43.9. How to tee Several Commands into One Place43.10. Redirecting Output to More Than One Place43.11. Named Pipes: FIFOs43.12. What Can You Do with an Empty File?
44. Devices
44.1. Quick Introduction to Hardware44.2. Reading Kernel Boot Output44.3. Basic Kernel Configuration44.4. Disk Partitioning44.5. Filesystem Types and /etc/fstab44.6. Mounting and Unmounting Removable Filesystems44.7. Loopback Mounts44.8. Network Devices — ifconfig44.9. Mounting Network Filesystems — NFS, SMBFS44.10. Win Is a Modem Not a Modem?44.11. Setting Up a Dialup PPP Session44.12. USB Configuration44.13. Dealing with Sound Cards and Other Annoying Hardware44.14. Decapitating Your Machine — Serial Consoles
45. Printing
45.1. Introduction to Printing45.2. Introduction to Printing on Unix45.2.1. lpr-Style Printing Commands45.2.2. lp-Style Printing Commands45.3. Printer Control with lpc45.4. Using Different Printers45.5. Using Symbolic Links for Spooling45.6. Formatting Plain Text: pr45.7. Formatting Plain Text: enscript45.8. Printing Over a Network45.9. Printing Over Samba45.9.1. Printing to Unix Printers from Windows45.9.2. Printing to Windows Printers from Unix45.10. Introduction to Typesetting45.11. A Bit of Unix Typesetting History45.12. Typesetting Manpages: nroff45.13. Formatting Markup Languages — troff, LATEX, HTML, and So On45.14. Printing Languages — PostScript, PCL, DVI, PDF45.15. Converting Text Files into a Printing Language45.16. Converting Typeset Files into a Printing Language45.17. Converting Source Files Automagically Within the Spooler45.18. The Common Unix Printing System (CUPS)45.19. The Portable Bitmap Package
46. Connectivity
46.1. TCP/IP — IP Addresses and Ports46.1.1. Internet Protocol (IP)46.1.2. Layer 4 Protocols: TCP, UDP, and ICMP46.2. /etc/services Is Your Friend46.3. Status and Troubleshooting46.4. Where, Oh Where Did That Packet Go?46.5. The Director of Operations: inetd46.6. Secure Shell (SSH)46.7. Configuring an Anonymous FTP Server46.8. Mail — SMTP, POP, and IMAP46.9. Domain Name Service (DNS)46.10. Dynamic Host Configuration Protocol (DHCP)46.11. Gateways and NAT46.12. Firewalls46.13. Gatewaying from a Personal LAN over a Modem
47. Connecting to MS Windows
47.1. Building Bridges47.2. Installing and Configuring Samba47.3. Securing Samba47.4. SWAT and GUI SMB Browsers47.5. Printing with Samba47.6. Connecting to SMB Shares from Unix47.7. Sharing Desktops with VNC47.7.1. Connecting to a Windows VNC server47.7.2. Setting up VNC on Unix47.8. Of Emulators and APIs47.8.1. VMWare47.8.2. Wine47.9. Citrix: Making Windows Multiuser47.9.1. Citrix Metaframe47.9.2. rdesktop47.9.3. Hob
IX. Security
48. Security Basics
48.1. Understanding Points of Vulnerability48.2. CERT Security Checklists48.3. Keeping Up with Security Alerts48.4. What We Mean by Buffer Overflow48.5. What We Mean by DoS48.6. Beware of Sluggish Performance48.6.1. Check Processes48.6.2. Checking Swap Space48.6.3. Check Network Connections48.6.4. Other Checks48.7. Intruder Detection48.8. Importance of MOTD48.9. The Linux proc Filesystem48.10. Disabling inetd48.11. Disallow rlogin and rsh48.12. TCP Wrappers
49. Root, Group, and User Management
49.1. Unix User/Group Infrastructure49.2. When Does a User Become a User49.3. Forgetting the root Password49.4. Setting an Exact umask49.5. Group Permissions in a Directory with the setgid Bit49.6. Groups and Group Ownership49.7. Add Users to a Group to Deny Permissions49.8. Care and Feeding of SUID and SGID Scripts49.9. Substitute Identity with su49.10. Never Log In as root49.11. Providing Superpowers with sudo49.12. Enabling Root in Darwin49.13. Disable logins
50. File Security, Ownership, and Sharing
50.1. Introduction to File Ownership and Security50.2. Tutorial on File and Directory Permissions50.2.1. User, Group, and World50.2.2. Which Group is Which?50.3. Who Will Own a New File?50.4. Protecting Files with the Sticky Bit50.5. Using chmod to Change File Permission50.6. The Handy chmod = Operator50.7. Protect Important Files: Make Them Unwritable50.8. cx, cw, c-w: Quick File Permission Changes50.9. A Loophole: Modifying Files Without Write Access50.10. A Directory That People Can Access but Can’t List50.11. Juggling Permissions50.12. File Verification with md5sum50.13. Shell Scripts Must Be Readable and (Usually) Executable50.14. Why Can’t You Change File Ownership?50.15. How to Change File Ownership Without chown
51. SSH
51.1. Enabling Remote Access on Mac OS X51.2. Protecting Access Through SSH51.3. Free SSH with OpenSSH51.4. SSH Problems and Solutions51.5. General and Authentication Problems51.6. Key and Agent Problems51.7. Server and Client Problems
Glossary
Index
About the Authors
Colophon
Copyright

Content preview from Unix Power Tools, 3rd Edition

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The image on the cover of Unix Power Tools, Third Edition, is an AC Dyno-Mite DC drill made by the Millers Falls Company, circa 1950.

Jeffrey Holcomb was the production editor for Unix Power Tools, Third Edition. Leanne Soylemez and Jeffrey Holcomb were the copyeditors. Mary Brady, Linley Dolby, and Claire Cloutier provided quality control. Genevieve d’Entremont, Julie Flanagan, Andrew Savikas, Brian Sawyer, and Sue Willing were the compositors. Ellen Troutman-Zaig wrote the index.

Edie Freedman designed the cover of this book. Emma Colby produced the cover layout with QuarkXPress 4.1 using Adobe’s ITC Garamond font.

David Futato designed the interior layout. This book was converted to FrameMaker 5.5.6 with a format conversion tool created by Erik Ray, Jason McIntosh, Neil Walls, and Mike Sierra that uses Perl and XML technologies. The text font is Linotype Birka; the heading font is Adobe Helvetica Neue Condensed; and the code font is LucasFont’s TheSans Mono Condensed. The illustrations that appear in the book were produced by Robert Romano and Jessamyn Read using Macromedia FreeHand 9 and Adobe Photoshop 6. This colophon was written by Jeffrey Holcomb.

The online edition of this book was created by the Safari production group (John ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

UNIX and Linux System Administration Handbook, 5th Edition

Publisher Resources

ISBN: 0596003307Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Unix Power Tools, 3rd Edition

by Jerry Peek, Shelley Powers, Tim O'Reilly, Mike Loukides

Colophon

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.