book

sed & awk, 2nd Edition

by Dale Dougherty, Arnold Robbins

March 1997

Intermediate to advanced

432 pages

11h 31m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note Regarding Supplemental Files
Dedication
Preface
Scope of This Handbook
Availability of sed and awk
DOS VersionsOther Sources of Information About sed and awkSample Programs
Obtaining Example Source Code
FTPFtpmailBITFTPUUCP
Conventions Used in This Handbook
About the Second Edition
Acknowledgments from the First Edition
Comments and Questions

1. Power Tools for Editing
1.1. May You Solve Interesting Problems
1.2. A Stream Editor
1.3. A Pattern-Matching Programming Language
1.4. Four Hurdles to Mastering sed and awk
2. Understanding Basic Operations
2.1. Awk, by Sed and Grep, out of Ed
2.2. Command-Line Syntax
2.2.1. Scripting2.2.2. Sample Mailing List
2.3. Using sed
2.3.1. Specifying Simple Instructions2.3.1.1. Command garbled2.3.2. Script Files2.3.2.1. Saving output2.3.2.2. Suppressing automatic display of input lines2.3.2.3. Mixing options (POSIX)2.3.2.4. Summary of options
2.4. Using awk
2.4.1. Running awk2.4.2. Error Messages2.4.3. Summary of Options
2.5. Using sed and awk Together
3. Understanding Regular Expression Syntax
3.1. That’s an Expression
3.2. A Line-Up of Characters
3.2.1. The Ubiquitous Backslash3.2.2. A Wildcard3.2.3. Writing Regular Expressions3.2.4. Character Classes3.2.4.1. A range of characters3.2.4.2. Excluding a class of characters3.2.4.3. POSIX character class additions3.2.5. Repeated Occurrences of a Character3.2.6. What’s the Word? Part I3.2.7. Positional Metacharacters3.2.7.1. Phrases3.2.8. A Span of Characters3.2.9. Alternative Operations3.2.10. Grouping Operations3.2.11. What’s the Word? Part II3.2.12. Your Replacement Is Here3.2.12.1. The extent of the match3.2.13. Limiting the Extent
3.3. I Never Metacharacter I Didn’t Like
4. Writing sed Scripts
4.1. Applying Commands in a Script
4.1.1. The Pattern Space
4.2. A Global Perspective on Addressing
4.2.1. Grouping Commands
4.3. Testing and Saving Output
4.3.1. testsed4.3.2. runsed
4.4. Four Types of sed Scripts
4.4.1. Multiple Edits to the Same File4.4.2. Making Changes Across a Set of Files4.4.3. Extracting Contents of a File4.4.3.1. Extracting a macro definition4.4.3.2. Generating an outline4.4.4. Edits To Go
4.5. Getting to the PromiSed Land
5. Basic sed Commands
5.1. About the Syntax of sed Commands
5.2. Comment
5.3. Substitution
5.3.1. Replacement Metacharacters5.3.1.1. Correcting index entries
5.4. Delete
5.5. Append, Insert, and Change
5.6. List
5.6.1. Stripping Out Non-Printable Characters from nroff Files
5.7. Transform
5.8. Print
5.9. Print Line Number
5.10. Next
5.11. Reading and Writing Files
5.11.1. Checking Out Reference Pages
5.12. Quit
6. Advanced sed Commands
6.1. Multiline Pattern Space
6.1.1. Append Next Line6.1.1.1. Converting an Interleaf file6.1.2. Multiline Delete6.1.3. Multiline Print
6.2. A Case for Study
6.3. Hold That Line
6.3.1. A Capital Transformation6.3.2. Correcting Index Entries (Part II)6.3.3. Building Blocks of Text
6.4. Advanced Flow Control Commands
6.4.1. Branching6.4.2. The Test Command6.4.3. One More Case
6.5. To Join a Phrase
7. Writing Scripts for awk
7.1. Playing the Game
7.2. Hello, World
7.3. Awk’s Programming Model
7.4. Pattern Matching
7.4.1. Describing Your Script
7.5. Records and Fields
7.5.1. Referencing and Separating Fields7.5.2. Field Splitting: The Full Story
7.6. Expressions
7.6.1. Averaging Student Grades
7.7. System Variables
7.7.1. Working with Multiline Records7.7.2. Balance the Checkbook
7.8. Relational and Boolean Operators
7.8.1. Getting Information About Files
7.9. Formatted Printing
7.10. Passing Parameters Into a Script
7.11. Information Retrieval
7.11.1. Finding a Glitch
8. Conditionals, Loops, and Arrays
8.1. Conditional Statements
8.1.1. Conditional Operator
8.2. Looping
8.2.1. While Loop8.2.2. Do Loop8.2.3. For Loop8.2.4. Deriving Factorials
8.3. Other Statements That Affect Flow Control
8.4. Arrays
8.4.1. Associative Arrays8.4.2. Testing for Membership in an Array8.4.3. A Glossary Lookup Script8.4.4. Using split( ) to Create Arrays8.4.5. Making Conversions8.4.6. Deleting Elements of an Array
8.5. An Acronym Processor
8.5.1. Multidimensional Arrays
8.6. System Variables That Are Arrays
8.6.1. An Array of Command-Line Parameters8.6.2. An Array of Environment Variables
9. Functions
9.1. Arithmetic Functions
9.1.1. Trigonometric Functions9.1.2. Integer Function9.1.3. Random Number Generation9.1.4. Pick ‘em
9.2. String Functions
9.2.1. Substrings9.2.2. String Length9.2.3. Substitution Functions9.2.4. Converting Case9.2.5. The match( ) Function
9.3. Writing Your Own Functions
9.3.1. Writing a Sort Function9.3.2. Maintaining a Function Library9.3.3. Another Sorted Example
10. The Bottom Drawer
10.1. The getline Function
10.1.1. Reading Input from Files10.1.2. Assigning the Input to a Variable10.1.3. Reading Input from a Pipe
10.2. The close( ) Function
10.3. The system( ) Function
10.4. A Menu-Based Command Generator
10.5. Directing Output to Files and Pipes
10.5.1. Directing Output to a Pipe10.5.2. Working with Multiple Files
10.6. Generating Columnar Reports
10.7. Debugging
10.7.1. Make a Copy10.7.2. Before and After Photos10.7.3. Finding Out Where the Problem Is10.7.4. Commenting Out Loud10.7.5. Slash and Burn10.7.6. Getting Defensive About Your Script
10.8. Limitations
10.9. Invoking awk Using the #! Syntax
11. A Flock of awks
11.1. Original awk
11.1.1. Escape Sequences11.1.2. Exponentiation11.1.3. The C Conditional Expression11.1.4. Variables as Boolean Patterns11.1.5. Faking Dynamic Regular Expressions11.1.6. Control Flow11.1.7. Field Separating11.1.8. Arrays11.1.9. The getline Function11.1.10. Functions11.1.11. Built-In Variables
11.2. Freely Available awks
11.2.1. Common Extensions11.2.1.1. Deleting all elements of an array11.2.1.2. Obtaining individual characters11.2.1.3. Flushing buffered output11.2.1.4. Special filenames11.2.1.5. The nextfile statement11.2.1.6. Regular expression record separators (gawk and mawk)11.2.2. Bell Labs awk11.2.3. GNU awk (gawk)11.2.3.1. Command line options11.2.3.2. An awk program search path11.2.3.3. Line continuation11.2.3.4. Extended regular expressions11.2.3.5. Regular expression record terminators11.2.3.6. Separating fields11.2.3.7. Additional special files11.2.3.8. Additional variables11.2.3.9. Additional functions11.2.3.10. A general substitution function11.2.3.11. Time management for programmers11.2.4. Michael’s awk (mawk)
11.3. Commercial awks
11.3.1. MKS awk11.3.2. Thompson Automation awk (tawk)11.3.2.1. Tawk language extensions11.3.2.2. Additional built-in tawk functions11.3.3. Videosoft VSAwk
11.4. Epilogue
12. Full-Featured Applications
12.1. An Interactive Spelling Checker
12.1.1. BEGIN Procedure12.1.2. Main Procedure12.1.3. END Procedure12.1.4. Supporting Functions12.1.5. The spellcheck Shell Script
12.2. Generating a Formatted Index
12.2.1. The masterindex Program12.2.2. Standardizing Input12.2.3. Sorting the Entries12.2.4. Handling Page Numbers12.2.5. Merging Entries with the Same Keys12.2.6. Formatting the Index12.2.6.1. The masterindex shell script
12.3. Spare Details of the masterindex Program
12.3.1. How to Hide a Special Character12.3.2. Rotating Two Parts12.3.3. Finding a Replacement12.3.4. A Function for Reporting Errors12.3.5. Handling See Also Entries12.3.6. Alternative Ways to Sort
13. A Miscellany of Scripts
13.1. uutot.awk—Report UUCP Statistics
13.1.1. Program Notes for uutot.awk
13.2. phonebill—Track Phone Usage
13.2.1. Program Notes for phonebill
13.3. combine—Extract Multipart uuencoded Binaries
13.3.1. Program Notes for combine
13.4. mailavg—Check Size of Mailboxes
13.4.1. Program Notes for mailavg
13.5. adj—Adjust Lines for Text Files
13.5.1. Program Notes for adj
13.6. readsource—Format Program Source Files for troff
13.6.1. Program Notes for readsource
13.7. gent—Get a termcap Entry
13.7.1. Program Notes for gent
13.8. plpr—lpr Preprocessor
13.8.1. Program Notes for plpr
13.9. transpose—Perform a Matrix Transposition
13.9.1. Program Notes for transpose
13.10. m1—Simple Macro Processor
13.10.1. Program Notes for m1
A. Quick Reference for sed
A.1. Command-Line Syntax
A.2. Syntax of sed Commands
A.2.1. Pattern AddressingA.2.2. Regular Expression Metacharacters for sed
A.3. Command Summary for sed
B. Quick Reference for awk
B.1. Command-Line Syntax
B.1.1. Shell Wrapper for Invoking awk
B.2. Language Summary for awk
B.2.1. Records and FieldsB.2.2. Format of a ScriptB.2.2.1. Line terminationB.2.2.2. CommentsB.2.3. PatternsB.2.4. Regular ExpressionsB.2.5. ExpressionsB.2.5.1. ConstantsB.2.5.2. Escape sequencesB.2.5.3. VariablesB.2.5.4. ArraysB.2.5.5. System variablesB.2.5.6. OperatorsB.2.6. Statements and Functions
B.3. Command Summary for awk
B.3.1. Format Expressions Used in printf and sprintf
C. Supplement for Chapter 12
C.1. Full Listing of spellcheck.awk
C.2. Listing of masterindex Shell Script
C.3. Documentation for masterindex
C.3.1. Background Details
C.3.2. Coding Index Entries
C.3.3. Output Format
C.3.4. Compiling a Master Index
Index
About the Authors
Colophon
Copyright

Content preview from sed & awk, 2nd Edition

Getting to the PromiSed Land

You have now seen four different types of sed scripts, as well as how they are embedded inside shell scripts to create easy-to-use applications. More and more, as you work with sed, you will develop methods for creating and testing sed scripts. You will come to rely upon these methods and gain confidence that you know what your script is doing and why.

Here are a few tips:

Know Thy Input! Carefully examine your input file, using grep, before designing your script.
Sample Before Buying. Start with a small sample of occurrences in a test file. Run your script on the sample and make sure the script is working. Remember, it’s just as important to make sure the script doesn’t work where you don’t want it to. Then increase the size of the sample. Try to increase the complexity of the input.
Think Before Doing. Work carefully, testing each command that you add to a script. Compare the output against the input file to see what has changed. Prove to yourself that your script is complete. Your script may work perfectly, based on your assumptions of what is in the input file, but your assumptions may be wrong.
Be Pragmatic! Try to accomplish what you can with your sed script, but it doesn’t have to do 100 percent of the job. If you encounter difficult situations, check and see how frequently they occur. Sometimes it’s better to do a few remaining edits manually.

As you gain experience, add your own “scripting tips” to this list. You will also find that these tips apply ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

The AWK Programming Language, 2nd Edition

Publisher Resources

ISBN: 1565922255Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

sed & awk, 2nd Edition

by Dale Dougherty, Arnold Robbins

Getting to the PromiSed Land

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.