bash Cookbook, 2nd Edition

Preface

Every modern operating system has at least one shell, and some have many. Some shells are command line–oriented, such as the shell discussed in this book. Others are graphical, like Windows Explorer or the Macintosh Finder. Some users will interact with the shell only long enough to launch their favorite application, and then never emerge from that until they log off. But most users spend a significant amount of time using the shell. The more you know about your shell, the faster and more productive you can be.

Whether you are a system administrator, a programmer, or an end user, there are certainly occasions where a simple (or perhaps not so simple) shell script can save you time and effort, or facilitate consistency and repeatability for some important task. Even using an alias to change or shorten the name of a command you use often can have a significant effect. We’ll cover this and much more.

As with any general programming language, there is more than one way to do a given task in the shell. In some cases, there is only one best way, but in most cases there are at least two or three equally effective and efficient ways to write a solution. Which way you choose depends on your personal style, creativity, and familiarity with different commands and techniques. This is as true for us as authors as it is for you as the reader. In most cases we will choose a single method and implement it. In a few cases we may choose a particular method and explain why we think it’s the best. We may also occasionally show more than one equivalent solution so you can choose the one that best fits your needs and environment.

There is also sometimes a choice between a clever way to write some code, and a readable way. We will choose the readable way every time because experience has taught us that no matter how transparent you think your clever code is now, 6 or 18 months and 10 projects from now, you will be scratching your head asking yourself what you were thinking. Trust us: write clear code, and document it—you’ll thank yourself (and us) later.

Who Should Read This Book

This book is for anyone who uses a Unix or Linux system, as well as system administrators who may use several systems on any given day. With it, you will be able to create scripts that allow you to accomplish more, in less time, more easily, consistently, and repeatably than ever before.

Anyone? Yes. New users will appreciate the sections on automating repetitive tasks, making simple substitutions, and customizing their environment to be more friendly and perhaps behave in more familiar ways. Power users and administrators will find new and different solutions to common tasks and challenges. Advanced users will have a collection of techniques they can use at a moment’s notice to put out the latest fire, without having to remember every little detail of syntax.

Ideal readers include:

• New Unix or Linux users who don’t know much about the shell, but want to do more than point and click

• Experienced Unix or Linux users and system administrators looking for quick answers to shell scripting questions

• Programmers who work in a Unix or Linux (or even Windows) environment and want to be more productive

• New Unix or Linux sysadmins, or those coming from a Windows environment who need to come up to speed quickly

• Experienced Windows users and sysadmins who want a more powerful scripting environment

This book will only briefly cover basic and intermediate shell scripting—see Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly) and Classic Shell Scripting by Nelson H. F. Beebe and Arnold Robbins (O’Reilly) for more in-depth coverage. Instead, our goal is to provide solutions to common problems, with a strong focus on the “how to” rather than the theory. We hope this book will save you time when figuring out solutions or trying to remember syntax. In fact, that’s why we wrote this book: it’s what we wanted, one we could read through to get ideas, then refer to for practical working examples when needed. That way we wouldn’t have to remember the subtle differences between the shell, Perl, C, and so forth.

This book assumes you have access to a Unix or Linux system (or see Recipes 1.14 through 1.18, or Recipe 15.4) and are familiar with logging in, typing basic commands, and using a text editor. You do not have to be root to use the vast majority of the recipes, though there are a few, particularly dealing with installing bash, where root access will be needed.

About This Book

This book covers bash, the GNU Bourne Again Shell, which is a member of the family of shells that includes the original Bourne shell, sh, the Korn shell, ksh, and the public domain Korn shell, pdksh. While these and other shells such as dash and zsh are not specifically covered, odds are that most of the scripts will work pretty well with them.

You should be able to read this book cover to cover, and also just pick it up and read anything that catches your eye. But perhaps most importantly, we hope that when you have a question about how to do something or you need a hint, you will be able to easily find the right answer—or something close enough—and save time and effort.

A great part of the Unix philosophy is to build simple tools that do one thing well, then combine them as needed. This combination of tools is often accomplished via a shell script because these commands, called pipelines, can be long or difficult to remember and type. Where appropriate, we’ll cover the use of many of these tools in the context of the shell script as the glue that holds the pieces together to achieve the goal.

The first edition of this book was written using OpenOffice.org Writer running on whatever Linux or Windows machine happened to be handy, and kept in Subversion (see Appendix D). The nature of the Open Document Format facilitated many critical aspects of writing this book, including cross-references and extracting code (see Recipe 13.18). That source was later converted to DocBook for production.

For the second edition, we’ve switched to Asciidoc and Git on O’Reilly’s Atlas system, which worked very well. We’re grateful to O’Reilly’s production and tools departments for their help.

GNU Software

bash and many of the other tools we discuss in this book are part of the GNU Project. GNU (pronounced guh-noo, like canoe) is a recursive acronym for “GNU’s Not Unix,” and the project dates back to 1984. Its goal is to develop a free (as in freedom) Unix-like operating system.

Without getting into too much detail, what is commonly referred to as Linux is, in fact, a kernel with various supporting software as a core. The GNU tools are wrapped around it and it has a vast array of other software that may be included, depending on your distribution. However, the Linux kernel itself is not GNU software.

The GNU Project argues that Linux should in fact be called “GNU/Linux,” and it has a good point, so some distributions (notably Debian) do this. Therefore, GNU’s goal has arguably been achieved, though the result is not exclusively GNU.

The GNU Project has contributed a vast amount of superior software, notably including bash. There are GNU versions of practically every tool we discuss in this book, and while the GNU tools are more rich in terms of features and (usually) friendliness, they are also sometimes a little different. We discuss this in Recipe 15.3, though the commercial Unix vendors in the 1980s and 1990s are also largely to blame for these differences.

Enough (several books this size worth) has already been said about all of these aspects of GNU, Unix, and Linux, but we felt that this brief note was appropriate. See http://www.gnu.org for much more on the topic.

A Note About Code Examples

When we show an executable piece of shell scripting in this book, we typically show it in an offset area like this:

$ ls
a.out  cong.txt  def.conf  file.txt  more.txt  zebra.list
$

The first character is often a dollar sign ($) to indicate that this command has been typed at the bash shell prompt. (Remember that you can change the prompt, as described in Recipe 16.2, so your prompt may look very different.) The prompt is printed by the shell; you type the remainder of the line. Similarly, the last line in such an example is often a prompt (the $ again), to show that the command has ended execution and control has returned to the shell.

The pound or hash sign (#) is a little trickier. In many Unix or Linux files, including bash shell scripts, a leading # denotes a comment, and we have used it that way in some of our code examples. But as the trailing symbol in a bash command prompt (instead of $), # means you are logged in as root. We only have one example that is running anything as root, so that shouldn’t be confusing, but it’s important to understand.

When you see an example without the prompt string, we are showing the contents of a shell script. For several large examples we will number the lines of the script, though the numbers are not part of the script.

We may also occasionally show an example as a session log or a series of commands. In some cases, we may cat one or more files so you can see the script and/or datafiles we’ll be using in the example or in the results of our operation, like this:

$ cat data_file
static header line1
static header line2
1 foo
2 bar
3 baz

Many of the longer scripts and functions are available to download as well. See “Using Code Examples” for details. We have chosen to use !/usr/bin/env bash for these examples, where applicable, as that is more portable than the !/bin/bash you will see on Linux or a Mac. See Recipe 15.1 for more details.

Also, you may notice something like the following in many code examples:

# cookbook filename: snippet_name

That means that the code you are reading is available for download in our GitHub repository. You’ll find the code in something like ./chXX/snippet_name, where chXX is the chapter and snippet_name is the name of the file.

Useless Use of cat

Certain Unix users take a positively giddy delight in pointing out inefficiencies in other people’s code. Most of the time this is constructive criticism gently given and gratefully received.

Probably the most common case is the so-called “useless use of cat award” bestowed when someone does something like cat file | grep foo instead of simply grep foo file. In this case, cat is unnecessary and incurs some system overhead since it runs in a subprocess. Another common case would be cat file | tr '[A-Z]' '[a-z]' instead of tr '[A-Z]' '[a-z]' < file. Sometimes using cat can even cause your script to fail (see Recipe 19.8).

But…(you knew that was coming, didn’t you?) sometimes unnecessarily using cat actually does serve a purpose. It might be a placeholder to demonstrate a fragment of a pipeline, with other commands later replacing it (perhaps even cat -n). Or it might be that placing the file near the left side of the code draws the eye to it more clearly than if it were hidden behind a < on the far-right side of the page.

While we applaud efficiency and agree it is a goal to strive for, it isn’t as critical as it once was. We are not advocating carelessness and code bloat, we’re just saying that processors aren’t getting any slower anytime soon. So if you like cat, use it.

A Note About Perl

We made a conscious decision to avoid using Perl in our solutions as much as possible, though there are still a few cases where it makes sense. Perl is already covered elsewhere in far greater depth and breadth than we could ever manage here. And Perl solutions are generally much larger, with significantly more overhead, than ours. There is also a fine line between shell scripting and Perl scripting, and this is a book about shell scripting.

Shell scripting is basically glue for sticking Unix programs together, whereas Perl incorporates much of the functionality of the external Unix programs into the language itself. This makes it more efficient and in some ways more portable, at the expense of being different and making it harder to efficiently run any external programs you still need.

The choice of which tool to use often has more to do with familiarity than with any other reason. The bottom line is always getting the work done; the choice of tools is secondary. We’ll show you many ways to do things using bash and related tools. When you need to get your work done, you get to choose what tools you use.

More Resources

• Perl Cookbook, 2nd Edition, by Nathan Torkington and Tom Christiansen (O’Reilly)

• Programming Perl, 4th Edition, by Larry Wall et al. (O’Reilly)

• Perl Best Practices, by Damian Conway (O’Reilly)

• Mastering Regular Expressions, 3rd Edition, Jeffrey E. F. Friedl (O’Reilly)

• Learning the bash Shell, 3rd Edition, by Cameron Newham (O’Reilly)

• Classic Shell Scripting, Nelson H. F. Beebe and Arnold Robbins (O’Reilly)

Conventions Used in This Book

The following typographical conventions are used in this book:

Plain text

Indicates menu titles, menu options, menu buttons, and keyboard accelerators (such as Alt and Ctrl).

Italic

Indicates new terms, URLs, email addresses, filenames, file extensions, pathnames, directories, and Unix utilities.

Constant width

Indicates commands, options, switches, variables, attributes, keys, functions, types, classes, namespaces, methods, modules, properties, parameters, values, objects, events, event handlers, XML tags, HTML tags, macros, the contents of files, and the output from commands.

Constant width bold

Shows commands or other text that should be typed literally by the user.

Constant width italic

Shows text that should be replaced with user-supplied values.

Note

This icon signifies a general note.

Tip

This icon signifies a tip or suggestion.

Warning

This icon indicates a warning or caution.

Using Code Examples

Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/vossenjp/bashcookbook-examples.

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “bash Cookbook, 2nd Edition, by Carl Albing and JP Vossen. Copyright 2018 Carl Albing and JP Vossen, 978-1-491-97533-6.”

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [e-mail].

O’Reilly Safari

Note

Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals.

Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.

For more information, please visit http://oreilly.com/safari.

We’d Like to Hear from You

Please address comments and questions concerning this book to the publisher:

• O’Reilly Media, Inc.
• 1005 Gravenstein Highway North
• Sebastopol, CA 95472
• 800-998-9938 (in the United States or Canada)
• 707-829-0515 (international or local)
• 707-829-0104 (fax)

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/bash_cookbook_2E.

You can find information about this book, code samples, errata, links, bash documentation, and more at the authors’ site, http://www.bashcookbook.com.

Please drop by for a visit to learn, contribute, or chat. The authors would love to hear from you about what you like and don’t like about the book, what bash wonders you may have found, or lessons you have learned. To comment or ask technical questions about this book, send email to [e-mail].

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.

Find us on Facebook: http://facebook.com/oreilly

Follow us on Twitter: http://twitter.com/oreillymedia

Watch us on YouTube: http://www.youtube.com/oreillymedia

Acknowledgments

Thank you to the GNU Software Foundation and Brian Fox for writing bash. And thank you to Chet Ramey, who has been maintaining and improving bash since around version 1.14 in the early to mid-1990s. More thanks to Chet for answering our questions and for reviewing a draft of this book.

A special thanks, also, to Cameron Newham, who provided some material for the first edition, some of which remains in this edition. We highly recommend his O’Reilly book, Learning the bash Shell, written by Cameron Newham and Bill Rosenblatt.

Reviewers

Many thanks to our reviewers! They all provided valuable feedback, suggestions, and in some cases alternate solutions, pointed out issues we had overlooked, and in general greatly improved the book. Any errors or omissions in this text are ours and not theirs. An excellent example of their wisdom is the correct observation, “That sentence doesn’t know whether it’s coming or going!”

First edition: Yves Eynard, Chet Ramey, William Shotts, Ryan Waldron, and Michael Wang.

Second edition: Chet Ramey, Robert Day, and Arnold Robbins.

O’Reilly

Thanks to the entire team at O’Reilly, without whom this book would not exist for many reasons, and if it did the content wouldn’t be or look nearly as good!

First edition: our editor Mike Loukides, Derek Di Matteo, and Laurel Ruma.

Second edition: our editor Jeff Bleiel, Rachel Head, Kristen Brown, James Fraleigh, Ellen Troutman-Zaig, and Rebecca Demarest in production, and Matthew Hacker and other Tools folks. For a variety of reasons, the second edition required a lot of very painstaking work: thank you all for that. We are so impressed by your dedication to this project, as well as your attention to detail.

From the Authors

Carl

The writing of a book is never a solitary effort, though it has its moments. Thanks to JP for working on this project with me through both editions over many years. Our complementary talents and time schedules have made this a better book than I could have managed alone. Thanks also to JP for his great sysadmin efforts to provide us with some infrastructure. Thanks to Mike for listening to our proposals for a bash cookbook, for pushing us along when we got stuck, and for reining us in when we went crazy. His steady guidance and technical input were much appreciated. My wife, Cynthia, and children (now grown!) have patiently supported me through this process, giving me encouragement and motivation, as well as time and space to work. I thank them wholeheartedly.

But deeper than the immediate task of this book was the background and preparation. I’m greatly indebted to Dr. Ralph Bjork, who made it possible for me to start working with Unix back before almost anyone had ever heard of it. His vision, foresight, and guidance have paid dividends for me longer than I would ever have expected.

My work on this book is dedicated to my parents, Hank and Betty, who have given me every good thing they had to offer—life itself, Christian faith, love, an excellent education, a sense of belonging, and all those good and healthy things one hopes to pass on to one’s own children. I can never thank them enough.

JP

Thanks to Cameron for writing Learning the bash Shell, from which I learned a lot and which was my primary reference until I started this project, and for contributing so much useful material from it. Thanks to Carl for all his work: without him this would have taken four times as long and only been half as good. Thanks to Mike for getting the ball rolling, then keeping it rolling, and for bringing Carl on board. And thanks to both Carl and Mike for their patience with my life and time management issues.

This book is dedicated to Dad, who’d get a kick out of it. He always told me there are only two decisions that matter: what you do and who you marry. I’ve managed to get two for two, so I guess I’m doing pretty well. So this is also dedicated to Karen, for her incredible support, patience, and understanding during this longer than expected process and without whom even computers wouldn’t be as fun. Finally, thanks to Kate and Sam, who contributed greatly to my aforementioned life management issues.