book

Effective awk Programming, 3rd Edition

Name: Effective awk Programming, 3rd Edition
Author: Arnold Robbins
ISBN: 9780596000707

by Arnold Robbins

May 2001

Intermediate to advanced

450 pages

11h 50m

English

O'Reilly Media, Inc.

Read now

Unlock full access

A Note Regarding Supplemental Files
Dedication
Foreword
Preface
History of awk and gawkA Rose by Any Other NameUsing This BookTypographical ConventionsDark CornersThe GNU Project and This BookHow to ContributeAcknowledgments
I. The awk Language and gawk
1. Getting Started with awk
1.1. How to Run awk Programs1.1.1. One-Shot Throwaway awk Programs1.1.2. Running awk Without Input Files1.1.3. Running Long Programs1.1.4. Executable awk Programs1.1.5. Comments in awk Programs1.1.6. Shell-Quoting Issues1.2. Datafiles for the Examples1.3. Some Simple Examples1.4. An Example with Two Rules1.5. A More Complex Example1.6. awk Statements Versus Lines1.7. Other Features of awk1.8. When to Use awk
2. Regular Expressions
2.1. How to Use Regular Expressions2.2. Escape Sequences2.3. Regular Expression Operators2.4. Using Character Lists2.5. gawk-Specific Regexp Operators2.6. Case Sensitivity in Matching2.7. How Much Text Matches?2.8. Using Dynamic Regexps
3. Reading Input Files
3.1. How Input Is Split into Records3.2. Examining Fields3.3. Non-constant Field Numbers3.4. Changing the Contents of a Field3.5. Specifying How Fields Are Separated3.5.1. Using Regular Expressions to Separate Fields3.5.2. Making Each Character a Separate Field3.5.3. Setting FS from the Command Line3.5.4. Field-Splitting Summary3.6. Reading Fixed-Width Data3.7. Multiple-Line Records3.8. Explicit Input with getline3.8.1. Using getline with No Arguments3.8.2. Using getline into a Variable3.8.3. Using getline from a File3.8.4. Using getline into a Variable from a File3.8.5. Using getline from a Pipe3.8.6. Using getline into a Variable from a Pipe3.8.7. Using getline from a Coprocess3.8.8. Using getline into a Variable from a Coprocess3.8.9. Points to Remember About getline3.8.10. Summary of getline Variants
4. Printing Output
4.1. The print Statement4.2. Examples of print Statements4.3. Output Separators4.4. Controlling Numeric Output with print4.5. Using printf Statements for Fancier Printing4.5.1. Introduction to the printf Statement4.5.2. Format-Control Letters4.5.3. Modifiers for printf Formats4.5.4. Examples Using printf4.6. Redirecting Output of print and printf4.7. Special Filenames in gawk4.7.1. Special Files for Standard Descriptors4.7.2. Special Files for Process-Related Information4.7.3. Special Files for Network Communications4.7.4. Special Filename Caveats4.8. Closing Input and Output Redirections
5. Expressions
5.1. Constant Expressions5.1.1. Numeric and String Constants5.1.2. Octal and Hexadecimal Numbers5.1.3. Regular Expression Constants5.2. Using Regular Expression Constants5.3. Variables5.3.1. Using Variables in a Program5.3.2. Assigning Variables on the Command Line5.4. Conversion of Strings and Numbers5.5. Arithmetic Operators5.6. String Concatenation5.7. Assignment Expressions5.8. Increment and Decrement Operators5.9. True and False in awk5.10. Variable Typing and Comparison Expressions5.11. Boolean Expressions5.12. Conditional Expressions5.13. Function Calls5.14. Operator Precedence (How Operators Nest)

6. Patterns, Actions, and Variables
6.1. Pattern Elements6.1.1. Regular Expressions as Patterns6.1.2. Expressions as Patterns6.1.3. Specifying Record Ranges with Patterns6.1.4. The BEGIN and END Special Patterns6.1.4.1. Startup and cleanup actions6.1.4.2. Input/Output from BEGIN and END rules6.1.5. The Empty Pattern6.2. Using Shell Variables in Programs6.3. Actions6.4. Control Statements in Actions6.4.1. The if-else Statement6.4.2. The while Statement6.4.3. The do-while Statement6.4.4. The for Statement6.4.5. The break Statement6.4.6. The continue Statement6.4.7. The next Statement6.4.8. Using gawk’s nextfile Statement6.4.9. The exit Statement6.5. Built-in Variables6.5.1. Built-in Variables That Control awk6.5.2. Built-in Variables That Convey Information6.5.3. Using ARGC and ARGV
7. Arrays in awk
7.1. Introduction to Arrays7.2. Referring to an Array Element7.3. Assigning Array Elements7.4. Basic Array Example7.5. Scanning All Elements of an Array7.6. The delete Statement7.7. Using Numbers to Subscript Arrays7.8. Using Uninitialized Variables as Subscripts7.9. Multidimensional Arrays7.10. Scanning Multidimensional Arrays7.11. Sorting Array Values and Indices with gawk
8. Functions
8.1. Built-in Functions8.1.1. Calling Built-in Functions8.1.2. Numeric Functions8.1.3. String-Manipulation Functions8.1.3.1. More about \ and & with sub, gsub, and gensub8.1.4. Input/Output Functions8.1.5. Using gawk’s Timestamp Functions8.1.6. Bit-Manipulation Functions of gawk8.1.7. Using gawk’s String-Translation Functions8.2. User-Defined Functions8.2.1. Function Definition Syntax8.2.2. Function Definition Examples8.2.3. Calling User-Defined Functions8.2.4. The return Statement8.2.5. Functions and Their Effects on Variable Typing
9. Internationalization with gawk
9.1. Internationalization and Localization9.2. GNU gettext9.3. Internationalizing awk Programs9.4. Translating awk Programs9.4.1. Extracting Marked Strings9.4.2. Rearranging printf Arguments9.4.3. awk Portability Issues9.5. A Simple Internationalization Example9.6. gawk Can Speak Your Language
10. Advanced Features of gawk
10.1. Allowing Nondecimal Input Data10.2. Two-Way Communications with Another Process10.3. Using gawk for Network Programming10.4. Using gawk with BSD Portals10.5. Profiling Your awk Programs
11. Running awk and gawk
11.1. Invoking awk11.2. Command-Line Options11.3. Other Command-Line Arguments11.4. The AWKPATH Environment Variable11.5. Obsolete Options and/or Features11.6. Known Bugs in gawk
II. Using awk and gawk
12. A Library of awk Functions
12.1. Naming Library Function Global Variables12.2. General Programming12.2.1. Implementing nextfile as a Function12.2.2. Assertions12.2.3. Rounding Numbers12.2.4. The Cliff Random Number Generator12.2.5. Translating Between Characters and Numbers12.2.6. Merging an Array into a String12.2.7. Managing the Time of Day12.3. Datafile Management12.3.1. Noting Datafile Boundaries12.3.2. Rereading the Current File12.3.3. Checking for Readable Datafiles12.3.4. Treating Assignments as Filenames12.4. Processing Command-Line Options12.5. Reading the User Database12.6. Reading the Group Database
13. Practical awk Programs
13.1. Running the Example Programs13.2. Reinventing Wheels for Fun and Profit13.2.1. Cutting out Fields and Columns13.2.2. Searching for Regular Expressions in Files13.2.3. Printing out User Information13.2.4. Splitting a Large File into Pieces13.2.5. Duplicating Output into Multiple Files13.2.6. Printing Nonduplicated Lines of Text13.2.7. Counting Things13.3. A Grab Bag of awk Programs13.3.1. Finding Duplicated Words in a Document13.3.2. An Alarm Clock Program13.3.3. Transliterating Characters13.3.4. Printing Mailing Labels13.3.5. Generating Word-Usage Counts13.3.6. Removing Duplicates from Unsorted Text13.3.7. Extracting Programs from Texinfo Source Files13.3.8. A Simple Stream Editor13.3.9. An Easy Way to Use Library Functions
14. Internetworking with gawk
14.1. Networking with gawk14.1.1. gawk’s Networking Mechanisms14.1.1.1. The fields of the special filename14.1.1.2. Comparing protocols14.1.1.3. /inet/tcp14.1.1.4. /inet/udp14.1.1.5. /inet/raw14.1.2. Establishing a TCP Connection14.1.3. Troubleshooting Connection Problems14.1.4. Interacting with a Network Service14.1.5. Setting up a Service14.1.6. Reading Email14.1.7. Reading a Web Page14.1.8. A Primitive Web Service14.1.9. A Web Service with Interaction14.1.9.1. A Simple CGI Library14.1.10. A Simple Web Server14.1.11. Network Programming Caveats14.2. Some Applications and Techniques14.2.1. PANIC: An Emergency Web Server14.2.2. GETURL: Retrieving Web Pages14.2.3. REMCONF: Remote Configuration of Embedded Systems14.2.4. URLCHK: Look for Changed Web Pages14.2.5. WEBGRAB: Extract Links from a Page14.2.6. STATIST: Graphing a Statistical Distribution14.2.7. MOBAGWHO: A Simple Mobile Agent14.3. Related Links
III. Appendixes
A. The Evolution of the awk Language
A.1. Major Changes Between V7 and SVR3.1A.2. Changes Between SVR3.1 and SVR4A.3. Changes Between SVR4 and POSIX awkA.4. Extensions in the Bell Laboratories awkA.5. Extensions in gawk Not in POSIX awkA.6. Major Contributors to gawk
B. Installing gawk
B.1. The gawk DistributionB.1.1. Getting the gawk DistributionB.1.2. Extracting the DistributionB.1.3. Contents of the gawk DistributionB.2. Compiling and Installing gawk on UnixB.2.1. Compiling gawk for UnixB.2.2. Additional Configuration OptionsB.2.3. The Configuration ProcessB.3. Installation on PC Operating SystemsB.3.1. Installing a Prepared Distribution for PC SystemsB.3.2. Compiling gawk for PC Operating SystemsB.3.3. Using gawk on PC Operating SystemsB.4. Reporting Problems and BugsB.5. Other Freely Available awk Implementations
C. Implementation Notes
C.1. Downward Compatibility and DebuggingC.2. Making Additions to gawkC.2.1. Adding New FeaturesC.2.2. Porting gawk to a New Operating SystemC.3. Adding New Built-in Functions to gawkC.3.1. A Minimal Introduction to gawk InternalsC.3.2. Directory and File Operation Built-insC.3.2.1. Using chdir and statC.3.2.2. C code for chdir and statC.3.2.3. Integrating the extensionsC.4. Probable Future Extensions
D. Basic Programming Concepts
D.1. What a Program DoesD.2. Data Values in a ComputerD.3. Floating-Point Number Caveats
E. GNU General Public License
E.1. PreambleE.2. Terms and Conditions for Copying, Distribution, and ModificationE.3. NO WARRANTYE.4. END OF TERMS AND CONDITIONSE.4.1. How to Apply These Terms to Your New Programs
F. GNU Free Documentation License
F.1. ADDENDUM: How to Use This License for Your Documents
Glossary
Index
About the Author
Colophon
Copyright

Content preview from Effective awk Programming, 3rd Edition

Appendix C. Implementation Notes

This appendix contains information mainly of interest to implementors and maintainers of gawk. Everything in it applies specifically to gawk and not to other implementations.

Downward Compatibility and Debugging

See the Section A.5 in Appendix A for a summary of the GNU extensions to the awk language and program. All of these features can be turned off by invoking gawk with the --traditional option or with the --posix option.

If gawk is compiled for debugging with -DDEBUG, then there is one more option available on the command line:

-W parsedebug --parsedebug: Prints out the parse stack information as the program is being parsed.

This option is intended only for serious gawk developers and not for the casual user. It probably has not even been compiled into your version of gawk, since it slows down execution.

Making Additions to gawk

If you find that you want to enhance gawk in a significant fashion, you are perfectly free to do so. That is the point of having free software; the source code is available and you are free to change it as you want (see Appendix E).

This section discusses the ways you might want to change gawk as well as any considerations you should bear in mind.

Adding New Features

You are free to add any new features you like to gawk. However, if you want your changes to be incorporated into the gawk distribution, there are several steps that you need to take in order to make it possible for me to include your changes: ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 0596000707Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Effective awk Programming, 3rd Edition

by Arnold Robbins

Appendix C. Implementation Notes

Downward Compatibility and Debugging