book

Effective awk Programming, 4th Edition

Name: Effective awk Programming, 4th Edition
Author: Arnold Robbins
ISBN: 9781491904619

by Arnold Robbins

March 2015

Intermediate to advanced

560 pages

14h 27m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Dedication
Foreword to the Third Edition
Foreword to the Fourth Edition
Preface
History of awk and gawkA Rose by Any Other NameUsing This BookTypographical ConventionsDark CornersThe GNU Project and This BookHow to Stay CurrentUsing Code ExamplesSafari® Books OnlineHow to Contact UsAcknowledgments
I. The awk Language
1. Getting Started with awk
How to Run awk ProgramsOne-Shot Throwaway awk ProgramsRunning awk Without Input FilesRunning Long ProgramsExecutable awk ProgramsComments in awk ProgramsShell Quoting IssuesQuoting in MS-Windows batch filesDatafiles for the ExamplesSome Simple ExamplesAn Example with Two RulesA More Complex Exampleawk Statements Versus LinesOther Features of awkWhen to Use awkSummary
2. Running awk and gawk
Invoking awkCommand-Line OptionsOther Command-Line ArgumentsNaming Standard InputThe Environment Variables gawk UsesThe AWKPATH Environment VariableThe AWKLIBPATH Environment VariableOther Environment Variablesgawk’s Exit StatusIncluding Other Files into Your ProgramLoading Dynamic Extensions into Your ProgramObsolete Options and/or FeaturesUndocumented Options and FeaturesSummary
3. Regular Expressions
How to Use Regular ExpressionsEscape SequencesRegular Expression OperatorsUsing Bracket ExpressionsHow Much Text Matches?Using Dynamic Regexpsgawk-Specific Regexp OperatorsCase Sensitivity in MatchingSummary
4. Reading Input Files
How Input Is Split into RecordsRecord Splitting with Standard awkRecord Splitting with gawkExamining FieldsNonconstant Field NumbersChanging the Contents of a FieldSpecifying How Fields Are SeparatedWhitespace Normally Separates FieldsUsing Regular Expressions to Separate FieldsMaking Each Character a Separate FieldSetting FS from the Command LineMaking the Full Line Be a Single FieldField-Splitting SummaryReading Fixed-Width DataDefining Fields by ContentMultiple-Line RecordsExplicit Input with getlineUsing getline with No ArgumentsUsing getline into a VariableUsing getline from a FileUsing getline into a Variable from a FileUsing getline from a PipeUsing getline into a Variable from a PipeUsing getline from a CoprocessUsing getline into a Variable from a CoprocessPoints to Remember About getlineSummary of getline VariantsReading Input with a TimeoutDirectories on the Command LineSummary
5. Printing Output
The print Statementprint Statement ExamplesOutput SeparatorsControlling Numeric Output with printUsing printf Statements for Fancier PrintingIntroduction to the printf StatementFormat-Control LettersModifiers for printf FormatsExamples Using printfRedirecting Output of print and printfSpecial Files for Standard Preopened Data StreamsSpecial Filenames in gawkAccessing Other Open Files with gawkSpecial Files for Network CommunicationsSpecial Filename CaveatsClosing Input and Output RedirectionsSummary

6. Expressions
Constants, Variables, and ConversionsConstant ExpressionsNumeric and string constantsOctal and hexadecimal numbersRegular expression constantsUsing Regular Expression ConstantsVariablesUsing variables in a programAssigning variables on the command lineConversion of Strings and NumbersHow awk converts between strings and numbersLocales can influence conversionOperators: Doing Something with ValuesArithmetic OperatorsString ConcatenationAssignment ExpressionsIncrement and Decrement OperatorsTruth Values and ConditionsTrue and False in awkVariable Typing and Comparison ExpressionsString type versus numeric typeComparison operatorsString comparison with POSIX rulesBoolean ExpressionsConditional ExpressionsFunction CallsOperator Precedence (How Operators Nest)Where You Are Makes a DifferenceSummary
7. Patterns, Actions, and Variables
Pattern ElementsRegular Expressions as PatternsExpressions as PatternsSpecifying Record Ranges with PatternsThe BEGIN and END Special PatternsStartup and cleanup actionsInput/output from BEGIN and END rulesThe BEGINFILE and ENDFILE Special PatternsThe Empty PatternUsing Shell Variables in ProgramsActionsControl Statements in ActionsThe if-else StatementThe while StatementThe do-while StatementThe for StatementThe switch StatementThe break StatementThe continue StatementThe next StatementThe nextfile StatementThe exit StatementPredefined VariablesBuilt-in Variables That Control awkBuilt-in Variables That Convey InformationUsing ARGC and ARGVSummary
8. Arrays in awk
The Basics of ArraysIntroduction to ArraysReferring to an Array ElementAssigning Array ElementsBasic Array ExampleScanning All Elements of an ArrayUsing Predefined Array Scanning Orders with gawkUsing Numbers to Subscript ArraysUsing Uninitialized Variables as SubscriptsThe delete StatementMultidimensional ArraysScanning Multidimensional ArraysArrays of ArraysSummary
9. Functions
Built-in FunctionsCalling Built-in FunctionsNumeric FunctionsString-Manipulation FunctionsMore about ‘\’ and ‘&’ with sub(), gsub(), and gensub()Input/Output FunctionsTime FunctionsBit-Manipulation FunctionsGetting Type InformationString-Translation FunctionsUser-Defined FunctionsFunction Definition SyntaxFunction Definition ExamplesCalling User-Defined FunctionsWriting a function callControlling variable scopePassing function arguments by value or by referenceThe return StatementFunctions and Their Effects on Variable TypingIndirect Function CallsSummary
II. Problem Solving with awk
10. A Library of awk Functions
Naming Library Function Global VariablesGeneral ProgrammingConverting Strings to NumbersAssertionsRounding NumbersThe Cliff Random Number GeneratorTranslating Between Characters and NumbersMerging an Array into a StringManaging the Time of DayReading a Whole File at OnceQuoting Strings to Pass to the ShellDatafile ManagementNoting Datafile BoundariesRereading the Current FileChecking for Readable DatafilesChecking for Zero-Length FilesTreating Assignments as FilenamesProcessing Command-Line OptionsReading the User DatabaseReading the Group DatabaseTraversing Arrays of ArraysSummary
11. Practical awk Programs
Running the Example ProgramsReinventing Wheels for Fun and ProfitCutting Out Fields and ColumnsSearching for Regular Expressions in FilesPrinting Out User InformationSplitting a Large File into PiecesDuplicating Output into Multiple FilesPrinting Nonduplicated Lines of TextCounting ThingsA Grab Bag of awk ProgramsFinding Duplicated Words in a DocumentAn Alarm Clock ProgramTransliterating CharactersPrinting Mailing LabelsGenerating Word-Usage CountsRemoving Duplicates from Unsorted TextExtracting Programs from Texinfo Source FilesA Simple Stream EditorAn Easy Way to Use Library FunctionsFinding Anagrams from a DictionaryAnd Now for Something Completely DifferentSummary
III. Moving Beyond Standard awk with gawk
12. Advanced Features of gawk
Allowing Nondecimal Input DataControlling Array Traversal and Array SortingControlling Array TraversalSorting Array Values and Indices with gawkTwo-Way Communications with Another ProcessUsing gawk for Network ProgrammingProfiling Your awk ProgramsSummary
13. Internationalization with gawk
Internationalization and LocalizationGNU gettextInternationalizing awk ProgramsTranslating awk ProgramsExtracting Marked StringsRearranging printf Argumentsawk Portability IssuesA Simple Internationalization Examplegawk Can Speak Your LanguageSummary
14. Debugging awk Programs
Introduction to the gawk DebuggerDebugging in GeneralDebugging Conceptsawk DebuggingSample gawk Debugging SessionHow to Start the DebuggerFinding the BugMain Debugger CommandsControl of BreakpointsControl of ExecutionViewing and Changing DataWorking with the StackObtaining Information About the Program and the Debugger StateMiscellaneous CommandsReadline SupportLimitationsSummary
15. Arithmetic and Arbitrary-Precision Arithmetic with gawk
A General Description of Computer ArithmeticOther Stuff to KnowArbitrary-Precision Arithmetic Features in gawkFloating-Point Arithmetic: Caveat Emptor!Floating-Point Arithmetic Is Not ExactMany numbers cannot be represented exactlyBe careful comparing valuesErrors accumulateGetting the Accuracy You NeedTry a Few Extra Bits of Precision and RoundingSetting the PrecisionSetting the Rounding ModeArbitrary-Precision Integer Arithmetic with gawkStandards Versus Existing PracticeSummary
16. Writing Extensions for gawk
IntroductionExtension LicensingHow It Works at a High LevelAPI DescriptionIntroductionGeneral-Purpose Data TypesMemory Allocation Functions and Convenience MacrosConstructor FunctionsRegistration FunctionsRegistering an extension functionRegistering an exit callback functionRegistering an extension version stringCustomized input parsersCustomized output wrappersCustomized two-way processorsPrinting MessagesUpdating ERRNORequesting ValuesAccessing and Updating ParametersSymbol Table AccessVariable access and update by nameVariable access and update by cookieCreating and using cached valuesArray ManipulationArray data typesArray functionsWorking with all the elements of an arrayHow to create and populate arraysAPI VariablesAPI version constants and variablesInformational variablesBoilerplate CodeHow gawk Finds ExtensionsExample: Some File FunctionsUsing chdir() and stat()C Code for chdir() and stat()Integrating the ExtensionsThe Sample Extensions in the gawk DistributionFile-Related FunctionsInterface to fnmatch()Interface to fork(), wait(), and waitpid()Enabling In-Place File EditingCharacter and Numeric values: ord() and chr()Reading DirectoriesReversing OutputTwo-Way I/O ExampleDumping and Restoring an ArrayReading an Entire FileExtension Time FunctionsAPI TestsThe gawkextlib ProjectSummary
IV. Appendices
A. The Evolution of the awk Language
Major Changes Between V7 and SVR3.1Changes Between SVR3.1 and SVR4Changes Between SVR4 and POSIX awkExtensions in Brian Kernighan’s awkExtensions in gawk Not in POSIX awkCommon Extensions SummaryRegexp Ranges and Locales: A Long Sad StoryMajor Contributors to gawkSummary
B. Installing gawk
The gawk DistributionGetting the gawk DistributionExtracting the DistributionContents of the gawk DistributionCompiling and Installing gawk on Unix-Like SystemsCompiling gawk for Unix-Like SystemsAdditional Configuration OptionsThe Configuration ProcessInstallation on Other Operating SystemsInstallation on PC Operating SystemsCompiling gawk for PC operating systemsTesting gawk on PC operating systemsUsing gawk on PC operating systemsUsing gawk in the Cygwin environmentUsing gawk in the MSYS environmentCompiling and Installing gawk on Vax/VMS and OpenVMSCompiling gawk on VMSCompiling gawk dynamic extensions on VMSInstalling gawk on VMSRunning gawk on VMSThe VMS GNV projectSome VMS systems have an old version of gawkReporting Problems and BugsOther Freely Available awk ImplementationsSummary
C. GNU General Public License
Index
Colophon
Copyright

Content preview from Effective awk Programming, 4th Edition

Chapter 10. A Library of awk Functions

User-Defined Functions describes how to write your own awk functions. Writing functions is important, because it allows you to encapsulate algorithms and program tasks in a single place. It simplifies programming, making program development more manageable and making programs more readable.

In their seminal 1976 book, Software Tools,^[58] Brian Kernighan and P.J. Plauger wrote:

Good Programming is not learned from generalities, but by seeing how significant programs can be made clean, easy to read, easy to maintain and modify, human-engineered, efficient and reliable, by the application of common sense and good programming practices. Careful study and imitation of good programs leads to better writing.

In fact, they felt this idea was so important that they placed this statement on the cover of their book. Because we believe strongly that their statement is correct, this chapter and Chapter 11 provide a good-sized body of code for you to read and, we hope, to learn from.

This chapter presents a library of useful awk functions. Many of the sample programs presented later in this book use these functions. The functions are presented here in a progression from simple to complex.

Extracting Programs from Texinfo Source Files presents a program that you can use to extract the source code for these example library functions and programs from the Texinfo source for this book. (This has already been done as part of the gawk distribution.)

The programs in ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

The AWK Programming Language, 2nd Edition

Publisher Resources

ISBN: 9781491904930Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Effective awk Programming, 4th Edition

by Arnold Robbins

Chapter 10. A Library of awk Functions

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.