Learning the Korn Shell, 2nd Edition

Preface

The long, tortuous history of the Unix operating system has resulted in systems with all kinds of permutations and combinations of features. This means that whenever you walk up to an unfamiliar Unix system, you need to find out certain things about it in order to use it properly. And even on a given system, you may have a number of choices you can make about what features you want to use.

The most important such decision — if you get to make it — is what shell to use. “Shell” is Unix jargon for the program that allows you to communicate with the computer by entering commands and getting responses. The shell is completely separate from the Unix operating system per se; it’s just a program that runs on Unix. With other systems such as MS-DOS, Microsoft Windows, Macintosh,^[1] OpenVMS, and VM/CMS, the command interpreter or user interface is an integral part of the operating system.

Nowadays there are dozens of different shells floating around, ranging from the original standard, the Bourne shell, to menu-based and graphical interfaces. The most important shells have been the Bourne shell, the C shell, and the Korn shell — the subject of this book.

Korn Shell Versions

Specifically, this book describes the 1993 version of the Korn shell. The 1993 version is distributed with the three major commercial versions of Unix: Solaris, HP-UX, and AIX, albeit as part of the Common Desktop Environment (CDE), in /usr/dt/bin/dtksh. The 1988 version is also available on modern Unix systems, usually as /usr/bin/ksh. There are various other versions, variations, and implementations on other operating systems; these, along with the differences between the 1988 and 1993 versions of the Korn shell are described in Appendix A.

The 1993 version is now available in both source code form and as precompiled executables for many common systems. Downloading and building it is described in Appendix C. The latest, downloadable version of ksh has a number of features not in earlier versions. We cover all of those features, too. We have made an effort to point out when something may not be in an earlier version, but caveat emptor; if you need a feature and your system’s version of the 1993 Korn shell doesn’t have it, you may need to download a prebuilt executable or download the source and build your own executable.

To find out which version you have, type the command set -o emacs, then press CTRL-V. You should see a date followed by a version letter (the letter is generally unimportant). If you do, you have one of the official versions, the 1993 version, the 1988 version, or an older one. But if you don’t, you have a nonstandard version, such as pdksh, the public domain Korn shell discussed in Appendix A.

Summary of Korn Shell Features

The Korn shell is the most advanced of the shells that are “officially” distributed with Unix systems. It’s a backward-compatible evolutionary successor to the Bourne shell that includes most of the C shell’s major advantages as well as a number of new features of its own.

Features appropriated from the C shell include:

Job control

The ability to stop jobs with CTRL-Z and move them to the foreground or background with the fg and bg commands.

Aliases

The ability to define shorthand names for commands or command lines.

Functions

The ability to store your own shell code in memory instead of files. Functions increase programmability and efficiency. (Functions have been common in the Bourne shell for many years.)

Command history

The ability to recall previously entered commands.

The Korn shell’s major new features include:

Command-line editing

This feature allows you to use vi or Emacs-style editing commands on your command lines.

Integrated programming features

The functionality of several external Unix commands, including test, expr, getopt, and echo, has been integrated into the shell itself, enabling common programming tasks to be done more cleanly and without creating extra processes.

Control structures

Additional flow-control structures, especially the select construct, enable easy menu generation.

Debugging primitives

These features make it possible to write tools that help programmers debug their shell code.

Regular expressions

Well known to users of Unix utilities like grep and awk, regular expressions (albeit with a different syntax) have been added to the standard set of filename wildcards and to the shell variable facility.

Advanced I/O features

Several new facilities for control of process I/O, including the ability to do two-way communication with concurrent processes (coroutines), and to connect to network services.

New options and variables

These options and variables give you more ways to customize your environment than the standard Unix shells do.

Increased speed

The Korn shell often executes the same shell program considerably faster than the Bourne shell does.

Security features

Features designed to help protect against “Trojan horses” and other types of break-in schemes.

Major new features in the 1993 version include:

POSIX compliance

Compliance with POSIX, an international standard for portable shell programming, makes it possible to write and use portable shell scripts.

Arithmetic for loops

This new control structure lets you program more naturally when looping a fixed number of times.

Floating-point arithmetic

The ability to use floating-point numbers and new built-in arithmetic functions enrich the shell as a programming language.

Structured variable names

New syntax for variable names provides facilities similar to C structures and Ada records for grouping related items together in a variable.

Indirect variable references

This facility eases shell function programming for manipulating global variables.

Associative arrays

A powerful data-management facility that is similar to those in awk or perl.

Additional text manipulation facilities

There are even more ways to match patterns and substitute variables.

More built-in commands

Additional commands improve efficiency and increase script portability.

Intended Audience

This book is designed to appeal most to casual Unix users who are just above the “raw beginner” level. You should be familiar with the process of logging in, entering commands, and doing simple things with files. Although Chapter 1 reviews concepts such as the tree-like file and directory scheme, you may find that it moves too quickly if you’re a complete neophyte. In that case, we recommend the book Learning the Unix Operating System by Jerry Peek, Grace Todino, and John Strang, published by O’Reilly & Associates, Inc.

If you’re an experienced user, you may wish to skip Chapter 1 altogether. But if your experience is with the C shell, you may find that Chapter 1 reveals a few subtle differences between the Korn and C shells.

No matter what your level of experience, you will undoubtedly learn many things in this book that will make you a more productive Korn shell user — from major features down to details at the “nook-and-cranny” level that you weren’t aware of.

If you are interested in shell programming (writing shell scripts and functions that automate everyday tasks or serve as system utilities), you should find this book useful too. However, we have deliberately avoided drawing a strong distinction between interactive shell use (entering commands during a login session) and shell programming. We see shell programming as a natural, inevitable outgrowth of increasing experience as a user.

Accordingly, each chapter depends on those prior to it, and although the first three chapters are oriented toward interactive use only, subsequent chapters describe interactive user-oriented features in addition to programming concepts.

In fact, if this book has an overriding message, it is this: “The Korn shell is an incredibly powerful and grossly undervalued Unix programming environment. You — yes, you — can write useful shell programs, even if you just learned how to log on last week and have never programmed before.”

Toward that end, we have decided not to spend much time on features of interest exclusively to low-level systems programmers. Concepts like file descriptors, special file types, etc., can only confuse the casual user, and anyway, we figure that those of you who understand such things are smart enough to extrapolate the necessary information from our cursory discussions.

Code Examples

This book is full of examples of shell commands and programs that are designed to be useful in your everyday life as a user, not just to illustrate the feature being explained. From Chapter 4 onwards, we include various programming problems, which we call tasks, that illustrate particular shell programming concepts. Some tasks have solutions that are refined in subsequent chapters. The later chapters also include programming exercises, many of which build on the tasks in the chapter.

You should feel free to use any code you see in this book and pass it along to friends and colleagues. We especially encourage you to modify and enhance it yourself. The code is available from this book’s web site: http://www.oreilly.com/catalog/korn2/.

If you want to try examples but you don’t use the Korn shell as your login shell, you must put the following line at the top of each shell script:

#! /bin/ksh

If your Korn shell isn’t installed as the file /bin/ksh, or if /bin/ksh is the 1988 version, substitute the full pathname for your version of ksh93 in the above.

Chapter Summary

If you want to investigate specific topics rather than read the entire book through, here is a chapter-by-chapter summary:

Chapter 1

Introduces the Korn shell and tells you how to install it as your login shell. It then introduces the basics of interactive shell use, including overviews of the Unix file and directory scheme, standard I/O, and background jobs.

Chapter 2

Discusses the shell’s command history mechanism, including the emacs and vi editing modes and the hist history command.

Chapter 3

Covers ways to customize your shell environment without programming, by using the .profile and environment files. Aliases, options, and shell variables are the customization techniques discussed.

Chapter 4

Introduces shell programming. This chapter explains the basics of shell scripts and functions, and discusses several important “nuts-and-bolts” programming features: string manipulation operators, regular expressions, command-line arguments (positional parameters), and command substitution.

Chapter 5

Continues the discussion of shell programming by describing command exit status, conditional expressions, and the shell’s flow-control structures: if, for, case, select, while, and until.

Chapter 6

Goes into depth about positional parameters and command-line option processing, then discusses special types and properties of variables, such as integer and floating-point arithmetic, the arithmetic version of the for loop, indexed and associative arrays, and the typeset command.

Chapter 7

Gives a detailed description of Korn shell I/O, filling in the information omitted in Chapter 1. All of the shell’s I/O redirectors are covered, along with the shell’s ability to make TCP/IP socket connections and the line-at-a-time I/O commands read, print, and printf. The chapter then discusses the shell’s command-line processing mechanism and the eval command.

Chapter 8

Covers process-related issues in detail. It starts with a discussion of job control and then gets into various low-level information about processes, including process IDs, signals, and traps. The chapter then moves out to a higher level of abstraction to discuss coroutines, two-way pipes, and subshells.

Chapter 9

Discusses various debugging techniques, starting with simple ones like trace and verbose modes and “fake signal” traps. Next, this chapter describes discipline functions. Finally, it presents kshdb, a Korn shell debugging tool that you can use to debug your own code.

Chapter 10

Gives information for system administrators, including techniques for implementing system-wide shell customization, customizing the built-in editors, and features related to system security.

Appendix A

Compares the 1993 Korn shell to several similar shells, including the standard SVR4 Bourne shell, the 1988 Korn shell, the IEEE 1003.2 POSIX shell standard, the CDE Desk Top Korn shell (dtksh), tksh (which blends Tcl/Tk with ksh), the public domain Korn shell (pdksh), the Free Software Foundation’s bash, the Z shell (zsh), and a number of Bourne-style shells (really Unix-emulation environments) for Microsoft Windows.

Appendix B

Contains lists of shell invocation options, built-in commands, predefined aliases, built-in variables, conditional test operators, set command options, typeset command options, and emacs and vi editing mode commands. This appendix also covers the full details for using the getopts built-in command.

Appendix C

Describes how to download the source for ksh93 and build a working executable. This appendix also covers downloading prebuilt executables for a number of different systems.

Appendix D

Presents the licensing terms for the ksh93 source code.

Conventions Used in This Handbook

We leave it as understood that, when you enter a shell command, you press ENTER at the end. ENTER is labeled RETURN on some keyboards.

Characters called CTRL-X, where X is any letter, are entered by holding down the CTRL (or CTL, or CONTROL) key and pressing that letter. Although we give the letter in uppercase, you can press the letter without the SHIFT key.

Other special characters are newline (which is the same as CTRL-J), BACKSPACE (same as CTRL-H), ESC, TAB, and DEL (sometimes labeled DELETE or RUBOUT).

This book uses the following font conventions:

Italic

Used when discussing Unix filenames, external and built-in commands, alias names, command options, shell options, and shell functions. Italic is also used in the text when discussing dummy parameters that should be replaced with an actual value, to distinguish the vi and emacs programs from their Korn-shell modes, and to highlight special terms the first time they are defined.

Constant width

Used for variable names and shell keywords, filename suffixes, and in examples to show the contents of files or the output from commands, as well as for command lines when they are within regular text.

Constant width bold

Used in examples to show interaction between the user and the shell; any text the user types in is shown in constant width bold. For example:

$ pwd
/home/billr/ora/kb
$

Constant width italic

Used in the text and in example command lines for dummy parameters that should be replaced with an actual value. For example:

$ cd 
                        directory

Reverse video

Used in Chapter 2 to show the position of the cursor on the command line being edited. For example:

grep -l Bob < ~pete/wk/names

Note

Indicates a tip, suggestion, or general note.

Warning

Indicates a warning or caution.

Standard Unix utility commands are sometimes mentioned with a number in parentheses (usually 1) following the command’s name. The number refers to the section of the Unix User’s Manual in which you’ll find reference documentation (a.k.a. “the man page”) on the utility in question. For example, grep(1) means the man page for grep in Section 1.

When there is an important difference between the 1988 and 1993 versions of the Korn shell, we refer to them as ksh88 and ksh93 respectively. Most of this book applies to all versions of the 1993 Korn shell. When we need to distinguish among different versions of the 1993 Korn shell, we add the minor release to the name, such as ksh93h, or ksh93l+.

About the Second Edition

The first edition of this book covered the 1988 version of the Korn shell. Shortly after it was published, David Korn released the 1993 version, which included compatibility with the POSIX 1003.2 shell standard, as well as many new features.

Although ksh93 has been slow to spread to the commercial Unix world, source code is now available, so anyone who wants a copy of the latest and greatest version of ksh93 has but to download the source and compile it. With this in mind, we have made ksh93 the focus of the second edition, with a summary of the differences available in an appendix. This edition covers the most recent ksh93 release available at the time of writing, which includes some significant features not found in earlier versions.

The basic structure and flow of the book remains the same, although we have fixed a number of mistakes and typos in the first edition and updated kshdb, the Korn Shell Debugger, to work with ksh93. Appendix A now includes more information about Korn shell work-alikes, both for Unix and Windows systems.

Also included with this edition is a reference card that covers many of the features of ksh93 described in this book. This card is copyrighted by Specialized Systems Consultants, Inc. (SSC), and is reprinted by permission. SSC sells a four-color, 26-panel, 3.5-inch by 8.5-inch version of the card that covers both ksh88 and ksh93 in considerably more detail. See http://www.ssc.com for more information.

We’d Like to Hear From You

Please address comments and questions concerning this book to the publisher:

O’Reilly & Associates, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

[phone] (in the United States or Canada)

[phone] (international or local)

[phone] (fax)

We have a web page for this book, where we list examples, errata, or any additional information. You can access this page at:

http://www.oreilly.com/catalog/korn2/

To comment or ask technical questions about this book, send email to:

[e-mail]

For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our web site at:

http://www.oreilly.com

Acknowledgments

Writing a book from scratch isn’t easy. Updating a book is even harder; the trick is to make it impossible (or at least difficult) for the reader to tell which author wrote what parts. I hope I’ve succeeded in that. I want to acknowledge Bill Rosenblatt for writing the first edition and giving me an excellent body of material with which to work. This is one of the best O’Reilly books I’ve ever read, and it was a pleasure to work with it.

I would like to thank (in alphabetical order) Nelson A. Beebe (University of Utah Mathematics department), Dr. David G. Korn (AT&T Research), Chet Ramey (maintainer of bash), Bill Rosenblatt (GiantSteps/Media Technology Strategies), and Dr. Eugene H. Spafford (Purdue University Computer Science department) for reviewing the book and providing many helpful comments. Mike Loukides, the book’s editor, was very patient with me during several delays in the update. David Chu, of O’Reilly’s editorial staff, did a great job making sure that many of the “nuts and bolts” parts of the project got done, for which I’m grateful.

David Korn, now of AT&T Research Laboratories, and author of the Korn shell, answered a number of questions and provided early access to ksh93l documentation, which helped considerably, as well as prerelease access to ksh93n. Glenn Fowler, also of AT&T Research, helped with compilation issues under GNU/Linux, as well as helping me understand a number of the finer points of ksh usage. Steve Alston provided some improvements to the kshdb debugger in Chapter 9. George Kraft IV provided helpful information on dtksh for Appendix A. Glenn Barry, of Sun Microsystems, provided information on zsh for Appendix A.

Thanks to Phil Hughes, president of SSC, for permission to reprint parts of their ksh reference card.

Several other staff members at O’Reilly contributed to the book, as well: Leanne Soylemez was the production editor and proofreader; Mary Brady and Jane Ellin provided additional quality control; Brenda Miller wrote the index.

Finally, thanks to my wonderful wife Miriam, for not demanding the attention she was entitled to on way too many nights while I worked on this book. Without her, nothing would be worth doing.

Arnold Robbins
Nof Ayalon
ISRAEL

Acknowledgments from the First Edition

Many people contributed to this book in many ways. I’d like to thank the following people for technical advice and assistance: for system administration help, John van Vlaanderen and Alexis Rosen. For information on alternative shells, John (again), Sean Wilson (of MKS), Ed Ravin, Mel Rappaport, and Chet Ramey. For identifying the need for a shell debugger, expertise in SunOS and system security, and, indeed, a significant portion of my career, Hal Stern. For debugger suggestions, Tan Bronson. For humanitarian aid, Jessica Lustig. And much thanks to David Korn for all kinds of great “horse’s mouth” information — and, of course, for the Korn shell itself.

Thanks to our technical reviewers: Jim Baumbach, Jim Falk, David Korn, Ed Miner, Eric Pearce, and Ed Ravin. I especially appreciate the cooperation of Ed and Ed (in that order) during my “Whaddya mean, it doesn’t work?!?” phase.

Several people at O’Reilly & Associates contributed to this effort: Gigi Estabrook and Clairemarie Fisher O’Leary proofread multiple drafts of the manuscript, Kismet McDonough and Donna Woonteiler copyedited the manuscript, Len Muellner implemented the book design macro package, Jennifer Niederst designed the cover and the format of the book, and Chris Reilley created the figures. Finally, an ocean of gratitude to Mike Loukides — editor, motivator, facilitator, constructive nit-picker, and constant voice of reason. He and the other folks at O’Reilly & Associates are some of the most innovative, interesting, and motivated people I’ve ever had the privilege to work with.

—Bill Rosenblatt

---

^[1] At least up until MacOS X. MacOS X has a Unix-like system at its core — Darwin, which is derived from Mach and 4.4-BSD-Lite.