book

Effective awk Programming, 3rd Edition

Name: Effective awk Programming, 3rd Edition
Author: Arnold Robbins
ISBN: 9780596000707

by Arnold Robbins

May 2001

Intermediate to advanced

450 pages

11h 50m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note Regarding Supplemental Files
Dedication
Foreword
Preface
History of awk and gawkA Rose by Any Other NameUsing This BookTypographical ConventionsDark CornersThe GNU Project and This BookHow to ContributeAcknowledgments
I. The awk Language and gawk
1. Getting Started with awk
1.1. How to Run awk Programs1.1.1. One-Shot Throwaway awk Programs1.1.2. Running awk Without Input Files1.1.3. Running Long Programs1.1.4. Executable awk Programs1.1.5. Comments in awk Programs1.1.6. Shell-Quoting Issues1.2. Datafiles for the Examples1.3. Some Simple Examples1.4. An Example with Two Rules1.5. A More Complex Example1.6. awk Statements Versus Lines1.7. Other Features of awk1.8. When to Use awk
2. Regular Expressions
2.1. How to Use Regular Expressions2.2. Escape Sequences2.3. Regular Expression Operators2.4. Using Character Lists2.5. gawk-Specific Regexp Operators2.6. Case Sensitivity in Matching2.7. How Much Text Matches?2.8. Using Dynamic Regexps
3. Reading Input Files
3.1. How Input Is Split into Records3.2. Examining Fields3.3. Non-constant Field Numbers3.4. Changing the Contents of a Field3.5. Specifying How Fields Are Separated3.5.1. Using Regular Expressions to Separate Fields3.5.2. Making Each Character a Separate Field3.5.3. Setting FS from the Command Line3.5.4. Field-Splitting Summary3.6. Reading Fixed-Width Data3.7. Multiple-Line Records3.8. Explicit Input with getline3.8.1. Using getline with No Arguments3.8.2. Using getline into a Variable3.8.3. Using getline from a File3.8.4. Using getline into a Variable from a File3.8.5. Using getline from a Pipe3.8.6. Using getline into a Variable from a Pipe3.8.7. Using getline from a Coprocess3.8.8. Using getline into a Variable from a Coprocess3.8.9. Points to Remember About getline3.8.10. Summary of getline Variants
4. Printing Output
4.1. The print Statement4.2. Examples of print Statements4.3. Output Separators4.4. Controlling Numeric Output with print4.5. Using printf Statements for Fancier Printing4.5.1. Introduction to the printf Statement4.5.2. Format-Control Letters4.5.3. Modifiers for printf Formats4.5.4. Examples Using printf4.6. Redirecting Output of print and printf4.7. Special Filenames in gawk4.7.1. Special Files for Standard Descriptors4.7.2. Special Files for Process-Related Information4.7.3. Special Files for Network Communications4.7.4. Special Filename Caveats4.8. Closing Input and Output Redirections
5. Expressions
5.1. Constant Expressions5.1.1. Numeric and String Constants5.1.2. Octal and Hexadecimal Numbers5.1.3. Regular Expression Constants5.2. Using Regular Expression Constants5.3. Variables5.3.1. Using Variables in a Program5.3.2. Assigning Variables on the Command Line5.4. Conversion of Strings and Numbers5.5. Arithmetic Operators5.6. String Concatenation5.7. Assignment Expressions5.8. Increment and Decrement Operators5.9. True and False in awk5.10. Variable Typing and Comparison Expressions5.11. Boolean Expressions5.12. Conditional Expressions5.13. Function Calls5.14. Operator Precedence (How Operators Nest)

6. Patterns, Actions, and Variables
6.1. Pattern Elements6.1.1. Regular Expressions as Patterns6.1.2. Expressions as Patterns6.1.3. Specifying Record Ranges with Patterns6.1.4. The BEGIN and END Special Patterns6.1.4.1. Startup and cleanup actions6.1.4.2. Input/Output from BEGIN and END rules6.1.5. The Empty Pattern6.2. Using Shell Variables in Programs6.3. Actions6.4. Control Statements in Actions6.4.1. The if-else Statement6.4.2. The while Statement6.4.3. The do-while Statement6.4.4. The for Statement6.4.5. The break Statement6.4.6. The continue Statement6.4.7. The next Statement6.4.8. Using gawk’s nextfile Statement6.4.9. The exit Statement6.5. Built-in Variables6.5.1. Built-in Variables That Control awk6.5.2. Built-in Variables That Convey Information6.5.3. Using ARGC and ARGV
7. Arrays in awk
7.1. Introduction to Arrays7.2. Referring to an Array Element7.3. Assigning Array Elements7.4. Basic Array Example7.5. Scanning All Elements of an Array7.6. The delete Statement7.7. Using Numbers to Subscript Arrays7.8. Using Uninitialized Variables as Subscripts7.9. Multidimensional Arrays7.10. Scanning Multidimensional Arrays7.11. Sorting Array Values and Indices with gawk
8. Functions
8.1. Built-in Functions8.1.1. Calling Built-in Functions8.1.2. Numeric Functions8.1.3. String-Manipulation Functions8.1.3.1. More about \ and & with sub, gsub, and gensub8.1.4. Input/Output Functions8.1.5. Using gawk’s Timestamp Functions8.1.6. Bit-Manipulation Functions of gawk8.1.7. Using gawk’s String-Translation Functions8.2. User-Defined Functions8.2.1. Function Definition Syntax8.2.2. Function Definition Examples8.2.3. Calling User-Defined Functions8.2.4. The return Statement8.2.5. Functions and Their Effects on Variable Typing
9. Internationalization with gawk
9.1. Internationalization and Localization9.2. GNU gettext9.3. Internationalizing awk Programs9.4. Translating awk Programs9.4.1. Extracting Marked Strings9.4.2. Rearranging printf Arguments9.4.3. awk Portability Issues9.5. A Simple Internationalization Example9.6. gawk Can Speak Your Language
10. Advanced Features of gawk
10.1. Allowing Nondecimal Input Data10.2. Two-Way Communications with Another Process10.3. Using gawk for Network Programming10.4. Using gawk with BSD Portals10.5. Profiling Your awk Programs
11. Running awk and gawk
11.1. Invoking awk11.2. Command-Line Options11.3. Other Command-Line Arguments11.4. The AWKPATH Environment Variable11.5. Obsolete Options and/or Features11.6. Known Bugs in gawk
II. Using awk and gawk
12. A Library of awk Functions
12.1. Naming Library Function Global Variables12.2. General Programming12.2.1. Implementing nextfile as a Function12.2.2. Assertions12.2.3. Rounding Numbers12.2.4. The Cliff Random Number Generator12.2.5. Translating Between Characters and Numbers12.2.6. Merging an Array into a String12.2.7. Managing the Time of Day12.3. Datafile Management12.3.1. Noting Datafile Boundaries12.3.2. Rereading the Current File12.3.3. Checking for Readable Datafiles12.3.4. Treating Assignments as Filenames12.4. Processing Command-Line Options12.5. Reading the User Database12.6. Reading the Group Database
13. Practical awk Programs
13.1. Running the Example Programs13.2. Reinventing Wheels for Fun and Profit13.2.1. Cutting out Fields and Columns13.2.2. Searching for Regular Expressions in Files13.2.3. Printing out User Information13.2.4. Splitting a Large File into Pieces13.2.5. Duplicating Output into Multiple Files13.2.6. Printing Nonduplicated Lines of Text13.2.7. Counting Things13.3. A Grab Bag of awk Programs13.3.1. Finding Duplicated Words in a Document13.3.2. An Alarm Clock Program13.3.3. Transliterating Characters13.3.4. Printing Mailing Labels13.3.5. Generating Word-Usage Counts13.3.6. Removing Duplicates from Unsorted Text13.3.7. Extracting Programs from Texinfo Source Files13.3.8. A Simple Stream Editor13.3.9. An Easy Way to Use Library Functions
14. Internetworking with gawk
14.1. Networking with gawk14.1.1. gawk’s Networking Mechanisms14.1.1.1. The fields of the special filename14.1.1.2. Comparing protocols14.1.1.3. /inet/tcp14.1.1.4. /inet/udp14.1.1.5. /inet/raw14.1.2. Establishing a TCP Connection14.1.3. Troubleshooting Connection Problems14.1.4. Interacting with a Network Service14.1.5. Setting up a Service14.1.6. Reading Email14.1.7. Reading a Web Page14.1.8. A Primitive Web Service14.1.9. A Web Service with Interaction14.1.9.1. A Simple CGI Library14.1.10. A Simple Web Server14.1.11. Network Programming Caveats14.2. Some Applications and Techniques14.2.1. PANIC: An Emergency Web Server14.2.2. GETURL: Retrieving Web Pages14.2.3. REMCONF: Remote Configuration of Embedded Systems14.2.4. URLCHK: Look for Changed Web Pages14.2.5. WEBGRAB: Extract Links from a Page14.2.6. STATIST: Graphing a Statistical Distribution14.2.7. MOBAGWHO: A Simple Mobile Agent14.3. Related Links
III. Appendixes
A. The Evolution of the awk Language
A.1. Major Changes Between V7 and SVR3.1A.2. Changes Between SVR3.1 and SVR4A.3. Changes Between SVR4 and POSIX awkA.4. Extensions in the Bell Laboratories awkA.5. Extensions in gawk Not in POSIX awkA.6. Major Contributors to gawk
B. Installing gawk
B.1. The gawk DistributionB.1.1. Getting the gawk DistributionB.1.2. Extracting the DistributionB.1.3. Contents of the gawk DistributionB.2. Compiling and Installing gawk on UnixB.2.1. Compiling gawk for UnixB.2.2. Additional Configuration OptionsB.2.3. The Configuration ProcessB.3. Installation on PC Operating SystemsB.3.1. Installing a Prepared Distribution for PC SystemsB.3.2. Compiling gawk for PC Operating SystemsB.3.3. Using gawk on PC Operating SystemsB.4. Reporting Problems and BugsB.5. Other Freely Available awk Implementations
C. Implementation Notes
C.1. Downward Compatibility and DebuggingC.2. Making Additions to gawkC.2.1. Adding New FeaturesC.2.2. Porting gawk to a New Operating SystemC.3. Adding New Built-in Functions to gawkC.3.1. A Minimal Introduction to gawk InternalsC.3.2. Directory and File Operation Built-insC.3.2.1. Using chdir and statC.3.2.2. C code for chdir and statC.3.2.3. Integrating the extensionsC.4. Probable Future Extensions
D. Basic Programming Concepts
D.1. What a Program DoesD.2. Data Values in a ComputerD.3. Floating-Point Number Caveats
E. GNU General Public License
E.1. PreambleE.2. Terms and Conditions for Copying, Distribution, and ModificationE.3. NO WARRANTYE.4. END OF TERMS AND CONDITIONSE.4.1. How to Apply These Terms to Your New Programs
F. GNU Free Documentation License
F.1. ADDENDUM: How to Use This License for Your Documents
Glossary
Index
About the Author
Colophon
Copyright

Overview

Effective awk Programming,3rd Edition, focuses entirely onawk, exploring it in the greatest depth of the threeawk titles we carry. It's an excellent companion piece tothe more broadly focused second edition. This book providescomplete coverage of the gawk 3.1 language as well as themost up-to-date coverage of the POSIX standard for awkavailable anywhere. Author Arnold Robbins clearly distinguishesstandard awk features from GNU awk(gawk)-specific features, shines light into many of the"dark corners" of the language (areas to watch out for whenprogramming), and devotes two full chapters to example programs. Abrand new chapter is devoted to TCP/IP networking with gawk. Heincludes a summary of how the awk language evolved. The bookalso covers:

Internationalization of gawk

Interfacing to i18n at the awk level

Two-way pipes

TCP/IP networking via the two-way pipe interface

The new PROCINFO array, which provides information aboutrunning gawk

Profiling and pretty-printing awk programs

In addition to covering the awk language, this book servesas the official "User's Guide" for the GNU implementation ofawk (gawk), describing in an integrated fashion theextensions available to the System V Release 4 version ofawk that are also available in gawk. As the officialgawk User's Guide, this book will also be availableelectronically, and can be freely copied and distributed under theterms of the Free Software Foundation's Free Documentation License(FDL). A portion of the proceeds from sales of this book will go tothe Free Software Foundation to support further development of freeand open source software. The third edition of Effective awkProgramming is a GNU Manual and is published by O'Reilly &Associates under the Free Software Foundation'sFreeDocumentation License (FDL). A portion of the proceeds fromthe sale of this book is donated to the Free Software Foundation tofurther development of GNU software. This book is also available inelectronic form; you have the freedom to modify this GNU Manual,like GNU software. Copies published by the Free Software Foundationraise funds for GNU development.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 0596000707Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills