book

sed & awk, 2nd Edition

by Dale Dougherty, Arnold Robbins

March 1997

Intermediate to advanced

432 pages

11h 31m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note Regarding Supplemental Files
Dedication
Preface
Scope of This Handbook
Availability of sed and awk
DOS VersionsOther Sources of Information About sed and awkSample Programs
Obtaining Example Source Code
FTPFtpmailBITFTPUUCP
Conventions Used in This Handbook
About the Second Edition
Acknowledgments from the First Edition
Comments and Questions

1. Power Tools for Editing
1.1. May You Solve Interesting Problems
1.2. A Stream Editor
1.3. A Pattern-Matching Programming Language
1.4. Four Hurdles to Mastering sed and awk
2. Understanding Basic Operations
2.1. Awk, by Sed and Grep, out of Ed
2.2. Command-Line Syntax
2.2.1. Scripting2.2.2. Sample Mailing List
2.3. Using sed
2.3.1. Specifying Simple Instructions2.3.1.1. Command garbled2.3.2. Script Files2.3.2.1. Saving output2.3.2.2. Suppressing automatic display of input lines2.3.2.3. Mixing options (POSIX)2.3.2.4. Summary of options
2.4. Using awk
2.4.1. Running awk2.4.2. Error Messages2.4.3. Summary of Options
2.5. Using sed and awk Together
3. Understanding Regular Expression Syntax
3.1. That’s an Expression
3.2. A Line-Up of Characters
3.2.1. The Ubiquitous Backslash3.2.2. A Wildcard3.2.3. Writing Regular Expressions3.2.4. Character Classes3.2.4.1. A range of characters3.2.4.2. Excluding a class of characters3.2.4.3. POSIX character class additions3.2.5. Repeated Occurrences of a Character3.2.6. What’s the Word? Part I3.2.7. Positional Metacharacters3.2.7.1. Phrases3.2.8. A Span of Characters3.2.9. Alternative Operations3.2.10. Grouping Operations3.2.11. What’s the Word? Part II3.2.12. Your Replacement Is Here3.2.12.1. The extent of the match3.2.13. Limiting the Extent
3.3. I Never Metacharacter I Didn’t Like
4. Writing sed Scripts
4.1. Applying Commands in a Script
4.1.1. The Pattern Space
4.2. A Global Perspective on Addressing
4.2.1. Grouping Commands
4.3. Testing and Saving Output
4.3.1. testsed4.3.2. runsed
4.4. Four Types of sed Scripts
4.4.1. Multiple Edits to the Same File4.4.2. Making Changes Across a Set of Files4.4.3. Extracting Contents of a File4.4.3.1. Extracting a macro definition4.4.3.2. Generating an outline4.4.4. Edits To Go
4.5. Getting to the PromiSed Land
5. Basic sed Commands
5.1. About the Syntax of sed Commands
5.2. Comment
5.3. Substitution
5.3.1. Replacement Metacharacters5.3.1.1. Correcting index entries
5.4. Delete
5.5. Append, Insert, and Change
5.6. List
5.6.1. Stripping Out Non-Printable Characters from nroff Files
5.7. Transform
5.8. Print
5.9. Print Line Number
5.10. Next
5.11. Reading and Writing Files
5.11.1. Checking Out Reference Pages
5.12. Quit
6. Advanced sed Commands
6.1. Multiline Pattern Space
6.1.1. Append Next Line6.1.1.1. Converting an Interleaf file6.1.2. Multiline Delete6.1.3. Multiline Print
6.2. A Case for Study
6.3. Hold That Line
6.3.1. A Capital Transformation6.3.2. Correcting Index Entries (Part II)6.3.3. Building Blocks of Text
6.4. Advanced Flow Control Commands
6.4.1. Branching6.4.2. The Test Command6.4.3. One More Case
6.5. To Join a Phrase
7. Writing Scripts for awk
7.1. Playing the Game
7.2. Hello, World
7.3. Awk’s Programming Model
7.4. Pattern Matching
7.4.1. Describing Your Script
7.5. Records and Fields
7.5.1. Referencing and Separating Fields7.5.2. Field Splitting: The Full Story
7.6. Expressions
7.6.1. Averaging Student Grades
7.7. System Variables
7.7.1. Working with Multiline Records7.7.2. Balance the Checkbook
7.8. Relational and Boolean Operators
7.8.1. Getting Information About Files
7.9. Formatted Printing
7.10. Passing Parameters Into a Script
7.11. Information Retrieval
7.11.1. Finding a Glitch
8. Conditionals, Loops, and Arrays
8.1. Conditional Statements
8.1.1. Conditional Operator
8.2. Looping
8.2.1. While Loop8.2.2. Do Loop8.2.3. For Loop8.2.4. Deriving Factorials
8.3. Other Statements That Affect Flow Control
8.4. Arrays
8.4.1. Associative Arrays8.4.2. Testing for Membership in an Array8.4.3. A Glossary Lookup Script8.4.4. Using split( ) to Create Arrays8.4.5. Making Conversions8.4.6. Deleting Elements of an Array
8.5. An Acronym Processor
8.5.1. Multidimensional Arrays
8.6. System Variables That Are Arrays
8.6.1. An Array of Command-Line Parameters8.6.2. An Array of Environment Variables
9. Functions
9.1. Arithmetic Functions
9.1.1. Trigonometric Functions9.1.2. Integer Function9.1.3. Random Number Generation9.1.4. Pick ‘em
9.2. String Functions
9.2.1. Substrings9.2.2. String Length9.2.3. Substitution Functions9.2.4. Converting Case9.2.5. The match( ) Function
9.3. Writing Your Own Functions
9.3.1. Writing a Sort Function9.3.2. Maintaining a Function Library9.3.3. Another Sorted Example
10. The Bottom Drawer
10.1. The getline Function
10.1.1. Reading Input from Files10.1.2. Assigning the Input to a Variable10.1.3. Reading Input from a Pipe
10.2. The close( ) Function
10.3. The system( ) Function
10.4. A Menu-Based Command Generator
10.5. Directing Output to Files and Pipes
10.5.1. Directing Output to a Pipe10.5.2. Working with Multiple Files
10.6. Generating Columnar Reports
10.7. Debugging
10.7.1. Make a Copy10.7.2. Before and After Photos10.7.3. Finding Out Where the Problem Is10.7.4. Commenting Out Loud10.7.5. Slash and Burn10.7.6. Getting Defensive About Your Script
10.8. Limitations
10.9. Invoking awk Using the #! Syntax
11. A Flock of awks
11.1. Original awk
11.1.1. Escape Sequences11.1.2. Exponentiation11.1.3. The C Conditional Expression11.1.4. Variables as Boolean Patterns11.1.5. Faking Dynamic Regular Expressions11.1.6. Control Flow11.1.7. Field Separating11.1.8. Arrays11.1.9. The getline Function11.1.10. Functions11.1.11. Built-In Variables
11.2. Freely Available awks
11.2.1. Common Extensions11.2.1.1. Deleting all elements of an array11.2.1.2. Obtaining individual characters11.2.1.3. Flushing buffered output11.2.1.4. Special filenames11.2.1.5. The nextfile statement11.2.1.6. Regular expression record separators (gawk and mawk)11.2.2. Bell Labs awk11.2.3. GNU awk (gawk)11.2.3.1. Command line options11.2.3.2. An awk program search path11.2.3.3. Line continuation11.2.3.4. Extended regular expressions11.2.3.5. Regular expression record terminators11.2.3.6. Separating fields11.2.3.7. Additional special files11.2.3.8. Additional variables11.2.3.9. Additional functions11.2.3.10. A general substitution function11.2.3.11. Time management for programmers11.2.4. Michael’s awk (mawk)
11.3. Commercial awks
11.3.1. MKS awk11.3.2. Thompson Automation awk (tawk)11.3.2.1. Tawk language extensions11.3.2.2. Additional built-in tawk functions11.3.3. Videosoft VSAwk
11.4. Epilogue
12. Full-Featured Applications
12.1. An Interactive Spelling Checker
12.1.1. BEGIN Procedure12.1.2. Main Procedure12.1.3. END Procedure12.1.4. Supporting Functions12.1.5. The spellcheck Shell Script
12.2. Generating a Formatted Index
12.2.1. The masterindex Program12.2.2. Standardizing Input12.2.3. Sorting the Entries12.2.4. Handling Page Numbers12.2.5. Merging Entries with the Same Keys12.2.6. Formatting the Index12.2.6.1. The masterindex shell script
12.3. Spare Details of the masterindex Program
12.3.1. How to Hide a Special Character12.3.2. Rotating Two Parts12.3.3. Finding a Replacement12.3.4. A Function for Reporting Errors12.3.5. Handling See Also Entries12.3.6. Alternative Ways to Sort
13. A Miscellany of Scripts
13.1. uutot.awk—Report UUCP Statistics
13.1.1. Program Notes for uutot.awk
13.2. phonebill—Track Phone Usage
13.2.1. Program Notes for phonebill
13.3. combine—Extract Multipart uuencoded Binaries
13.3.1. Program Notes for combine
13.4. mailavg—Check Size of Mailboxes
13.4.1. Program Notes for mailavg
13.5. adj—Adjust Lines for Text Files
13.5.1. Program Notes for adj
13.6. readsource—Format Program Source Files for troff
13.6.1. Program Notes for readsource
13.7. gent—Get a termcap Entry
13.7.1. Program Notes for gent
13.8. plpr—lpr Preprocessor
13.8.1. Program Notes for plpr
13.9. transpose—Perform a Matrix Transposition
13.9.1. Program Notes for transpose
13.10. m1—Simple Macro Processor
13.10.1. Program Notes for m1
A. Quick Reference for sed
A.1. Command-Line Syntax
A.2. Syntax of sed Commands
A.2.1. Pattern AddressingA.2.2. Regular Expression Metacharacters for sed
A.3. Command Summary for sed
B. Quick Reference for awk
B.1. Command-Line Syntax
B.1.1. Shell Wrapper for Invoking awk
B.2. Language Summary for awk
B.2.1. Records and FieldsB.2.2. Format of a ScriptB.2.2.1. Line terminationB.2.2.2. CommentsB.2.3. PatternsB.2.4. Regular ExpressionsB.2.5. ExpressionsB.2.5.1. ConstantsB.2.5.2. Escape sequencesB.2.5.3. VariablesB.2.5.4. ArraysB.2.5.5. System variablesB.2.5.6. OperatorsB.2.6. Statements and Functions
B.3. Command Summary for awk
B.3.1. Format Expressions Used in printf and sprintf
C. Supplement for Chapter 12
C.1. Full Listing of spellcheck.awk
C.2. Listing of masterindex Shell Script
C.3. Documentation for masterindex
C.3.1. Background Details
C.3.2. Coding Index Entries
C.3.3. Output Format
C.3.4. Compiling a Master Index
Index
About the Authors
Colophon
Copyright

Content preview from sed & awk, 2nd Edition

m1—Simple Macro Processor

Contributed by Jon Bentley

The m1 program is a “little brother” to the m4 macro processor found on UNIX systems. It was originally published in the article m1: A Mini Macro Processor, in Computer Language, June 1990, Volume 7, Number 6, pages 47-61. This program was brought to my attention by Ozan Yigit. Jon Bentley kindly sent me his current version of the program, as well as an early draft of his article (I was having trouble getting a copy of the published one). A PostScript version of this paper is included with the example programs, available from O’Reilly’s FTP server (see the Preface). I wrote these introductory notes, and the program notes below. [A.R.]

A macro processor copies its input to its output, while performing several jobs. The tasks are:

Define and expand macros. Macros have two parts, a name and a body. All occurrences of a macro’s name are replaced with the macro’s body.
Include files. Special include directives in a data file are replaced with the contents of the named file. Includes can usually be nested, with one included file including another. Included files are processed for macros.
Conditional text inclusion and exclusion. Different parts of the text can be included in the final output, often based upon whether a macro is or isn’t defined.
Depending on the macro processor, comment lines can appear that will be removed from the final output.

If you’re a C or C++ programmer, you’re already familiar with the built-in preprocessor in those ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

The AWK Programming Language, 2nd Edition

Publisher Resources

ISBN: 1565922255Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

sed & awk, 2nd Edition

by Dale Dougherty, Arnold Robbins

m1—Simple Macro Processor

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.