By Don Libes
Price: $34.95 USD
£24.95 GBP

Cover | Table of Contents | Colophon

Table of Contents

Chapter 1: Intro—What Is Expect?

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect is a program to control interactive applications. These applications interactively prompt and expect a user to enter keystrokes in response. By using Expect, you can write simple scripts to automate these interactions. And using automated interactive programs, you will be able to solve problems that you never would have even considered before.

Expect can save you hours of drudgery, and this book will start you on your way. This first chapter is an overview of Expect. I will describe some simple applications and present some short scripts. However, the explanations are not intended to be complete but rather to whet your appetite and give you a taste of the good things to come. In the following chapters, I will revisit all of the concepts mentioned in this chapter and cover them in more detail.

fsck, the UNIX file system check program, can be run from a shell script only with the -y or -n options. The manual defines the -y option as follows:

"Assume a yes response to all questions asked by fsck; this should be used with extreme caution, as it is a free license to continue, even after severe problems are encountered."

The -n option is safer, but almost uselessly so. This kind of interface is inexcusably bad, yet many programs have the same style. ftp, a file transfer program, has an option that disables interactive prompting so that it can be run from a script. But it provides no way to take alternative action should an error occur.

Expect is a tool for controlling interactive programs. It solves the fsck problem, providing all the interactive functionality non-interactively. Expect is not specifically designed for fsck and can handle errors from ftp as well.

The problems with fsck and ftp illustrate a major limitation in the user interface offered by shells such as sh, csh, and others (which I will generically refer to as "the shell" from now on). The shell does not provide a way of reading output from and writing input to a program. This means the shell can run fsck, but only by missing out on some of its useful features. Some programs cannot be run at all from a shell script. For example,

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Ouch, Those Programs Are Painful!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

fsck, the UNIX file system check program, can be run from a shell script only with the -y or -n options. The manual defines the -y option as follows:

"Assume a yes response to all questions asked by fsck; this should be used with extreme caution, as it is a free license to continue, even after severe problems are encountered."

The -n option is safer, but almost uselessly so. This kind of interface is inexcusably bad, yet many programs have the same style. ftp, a file transfer program, has an option that disables interactive prompting so that it can be run from a script. But it provides no way to take alternative action should an error occur.

Expect is a tool for controlling interactive programs. It solves the fsck problem, providing all the interactive functionality non-interactively. Expect is not specifically designed for fsck and can handle errors from ftp as well.

The problems with fsck and ftp illustrate a major limitation in the user interface offered by shells such as sh, csh, and others (which I will generically refer to as "the shell" from now on). The shell does not provide a way of reading output from and writing input to a program. This means the shell can run fsck, but only by missing out on some of its useful features. Some programs cannot be run at all from a shell script. For example, passwd cannot be run without a user interactively supplying the input. Similar programs that cannot be automated in a shell script are telnet, crypt, su, rlogin, and gdb. A large number of application programs are written with the same fault of demanding user input.

Expect was designed specifically to interact with interactive programs. An Expect programmer can write a script describing the dialogue. Then the Expect program can run the "interactive" program non-interactively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

A Very Brief Overview

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect programs can be written in C or C++, but are almost always written using Tcl. Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you will not have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, UNIX programs can be called (exec). I will provide a quick introduction to the language in (p. 23).

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. The expect command describes a list of patterns to watch for. Each pattern is followed by an action. If the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings "welcome", "failed", or "busy", and then it evaluates one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other pattern matches in a certain amount of time.

expect {
    "welcome"            break
    "failed"            abort
    timeout            abort
    "busy"        {
        puts "busy"
        continue
    }
}

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

A First Script—dialback

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument.

spawn tip modem
expect "connected"
send "ATD$argv\r"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with "login:".

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60Kb executable (written in C) that did the same thing!

In (p. 347), I will talk more about the dialback concept and show a different way to do it.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Total Automation

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line. Instead, it interactively prompts for it—twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that—automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern "password:". expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage-return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a non-interactive passwd, but the script has to do it because passwd

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Differing Behavior When Running Non-Interactively

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Some programs behave differently when run interactively and non-interactively, and they do so intentionally. For example, most programs prompt only when running interactively. Non-interactively, prompting is not needed.

A more serious problem occurs when dealing with programs that change the way they buffer output depending on whether they are running interactively or not. Programs using the standard I/O library provided by UNIX automatically buffer their output when running non-interactively. This causes problems when you need to see the output immediately. Expect can make the programs think they are running interactively, thereby resolving the buffering problem.

As another example, shells force the user to press control characters (^Z, ^C) and keywords (fg, bg) to switch jobs. These cannot be used from shell scripts. Similarly, the shell running non-interactively does not deal with history and other features designed solely for interactive use. This presents a similar problem as with passwd earlier. Namely, it is impossible to construct shell scripts which test certain shell behavior. The result is that these aspects of the shell will inevitably not be rigorously tested. Using Expect, it is possible to drive the shell using its interactive job control features. A spawned shell thinks it is running interactively and handles job control as usual.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Partial Automation

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect's interact command turns control of a process over to you so that you can type directly to the process instead of through send commands.

Consider fsck, the UNIX program I mentioned earlier, which checks file system consistency. fsck provides almost no way of answering questions in advance. About all you can say is "answer everything yes" or "answer everything no".

The following fragment shows how a script can automatically answer some questions differently than others. The script begins by spawning fsck and then in a loop answering yes to one type of question and no to another. The \\ prevents the next character from being interpreted as a wildcard. In this example, the asterisk is a wild card but the question mark is not and matches a literal question mark.

while 1 {
    expect {
        eof                     {break}
        "UNREF FILE*CLEAR\\?"   {send "y\r"}
        "BAD INODE*FIX\\?"      {send "n\r"}
        "\\? "                  {interact +}
    }
}

The last question mark is a catch-all. If the script sees a question it does not understand, it executes the interact command, which passes control back to you. Your keystrokes go directly to fsck. When done, you can exit or return control to the script, here triggered by pressing the plus key. If you return control to the script, automated processing continues where it left off.

Without Expect, fsck can be run non-interactively only with very reduced functionality. It is barely programmable and yet it is the most critical of system administration tools. Many other tools have similarly deficient user interfaces. In fact, the large number of these is precisely what inspired the original development of Expect.

The interact command can be used to partially automate any program. Another popular use is for writing scripts that telnet through a number of hosts or front-ends, automatically handling protocols as encountered. When such a script finally reaches a point that you would like to take over, you can do so. For example, you could browse through remote library catalogs this way. Using Expect, scripts can make a number of different library systems seem like they are all connected rather than different and disconnected.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Dangerous, Unfriendly, Or Otherwise Unlikable User Interfaces

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The interact command is just a shorthand for the simplest and most common types of filtering that can be done in interactions. It is possible to build arbitrarily sophisticated mechanisms using the tools in Expect.

For example, commands can be filtered out of interfaces to prevent users from entering commands that you would prefer they not enter. Programs can also be wrapped even more tightly. The adb program, for instance, can crash a UNIX system with a slip of the finger or, more likely, inexperience. You can prevent this from happening by securely wrapping the adb program in a script. Not only does this increase the safety of your system, but your system administrators no longer all have to be masters of its intricacies.

The UNIX dump program is another program with an unlikable interface. For example, dump often guesses incorrectly about the length of a tape and will prompt for a new tape even if one is not needed. An Expect script can be used to respond to dump so that it can continue automatically. A script can answer just this question or any question from dump.

Expect can, in general, be applied to interfaces that you simply do not like for whatever reason. For example, you might like to automate sequences of your favorite game, perhaps because you have long since mastered some of it and would like to practice the end game but without laboriously replaying the beginning each time.

It is possible to customize exactly how much of the underlying programs show through. You can even make interactions entirely invisible so that users do not have to be irritated by underlying programs that they have no interest in anyway. Their attitude is "Use any tool. Just get the job done." And they are right.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Graphical Applications

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect can be combined with Tk, a toolkit for the X Window System. The combination of Expect and Tk is called Expectk. Using Expectk, it is possible to take an existing interactive application and give it a Motif-like X interface without changing any of the underlying program. No recompiling is necessary, and because the underlying programs are not changed, there is no need to retest them again. All your efforts can be focused on the graphical user interface. Making it look pretty is all you have to do.

Figure :

While Expectk will allow you to build X applications, it is limited in the amount it can automate existing X applications. Currently, Expect can automate xterm and other applications that specifically provide automation support, but Expect cannot automate any arbitrary X application.

Expect is also limited in its knowledge of character graphics such as is done by Curses. Nonetheless, with a little scripting, testing and automation of character graphics can be accomplished, and in (p. 453), I will describe one way of doing this. Ultimately, I believe that Expect will do best with this capability built in rather than provided via scripts. However, the requirements and interfaces are not obvious, and further experimentation and design is required. ExpecTerm is an example implementation of a built-in character-graphic capability based on an earlier version of Expect. ExpecTerm is available from the Tcl archive (see page 19).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

A Little More About Tcl

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect's language is general-purpose. It has control structures and data structures so that you can do just about anything. This can make up for the lack of programmability in other programs. For example, most debuggers do not provide very sophisticated methods of programmed control. Typically, you cannot declare variables and are restricted to simple loops.

The core of Expect's language facilities are provided by Tcl, a language used by many other tools. "Tcl" stands for Tool Command Language. It comes as a library intended to be embedded in applications, providing a generic language facility to any application. Tcl solves a long-standing problem in designing application languages.

Actually, most tools do not have very sophisticated internal control languages. The shell represents the extreme—it has a very rich language and can even call upon other programs. Start-up scripts such as .cshrc and .profile can be very complex. Few programs have such flexibility. And that is just as well. Such programs each have their own language and it is daunting to master each new one.

In the middle of the spectrum are programs such as ftp that have a limited ad hoc language—such as permitted in the file .netrc—not very flexible but better than nothing. A lot of programs are designed like ftp—their languages are limited, not extensible, and different from one tool to the next.

At the far end are programs such as telnet which have no control language whatsoever. The bulk of all interactive programs are like ftp and telnet—severely lacking in programmability.

This lack of programmability is understandable. Consider writing an application that controls the printer. It is not worth writing a big language that requires a scanner, a parser, symbol table, etc. The application alone might only be 10Kb. However, it is impossible to predict what uses you might put your program to in the future. Suppose you get another printer. Then your program needs to understand this. Suppose you give your program to someone else and they have two printers but they are of two different types. Suppose they want to reserve one printer for printouts of a special kind for a day—printing checks on payday, for example. You can see that the possibilities are endless. So rather than customizing the application each time and extending the language as required (if you have not boxed yourself into a corner already), it makes sense to use a general-purpose language from the beginning.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Job Control

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Just as you personally can interact with multiple programs at the same time, so can Expect. Analogous to the way you can say fg and bg to switch between processes in a shell, Expect can also switch its attention. Of course, Expect does it a lot more quickly than you can. The result is that Expect can act as "glue" for programs that were never designed to operate with other programs. An amusing example is the original chess program distributed with V7 and BSD UNIX written by Ken Thompson. It was designed to interact with a user—it prompts for moves and echoes user input in algebraic notation. Unfortunately, it does not accept its own output as input. Even if it did, there is no way to pipe both inputs and outputs between two processes simultaneously from the shell.

With a few lines of Expect, it is possible to make one chess process play another, including both the communication and the massaging of the output so that it is acceptable as input. I will show this in (p. 229), along with some more serious uses for these techniques.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Background Processes

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect is useful for automating processes in the background such as through cron. By using Expect with cron, you can not only automate the interactions but also automate starting the process in the first place. For example, you might want to copy over files each night between two networked but otherwise unrelated systems. And you can be arbitrarily selective about choosing files. You might want to copy over only executables that are less than 1 Mb and were created on a Saturday unless the current load exceeds 2.5 and the previous transfer took less than 5 seconds, or some other such complicated set of rules. There are no ftp clients that have such complex front-ends. But you can write Expect scripts to make whatever decisions seem appropriate while driving ftp.

It is even possible to have a cron process interactively contact you while it is in the background. For example, an ftp script may need a password to continue what it is doing. It can search the network for you and then ask you for it. Or you can have it look for a set of users. After getting the password, Expect will go back and use it to complete the task. By having Expect query for passwords, you do not need to embed them in scripts.

Doing backups from cron is another common reason to use Expect. If the backup program needs another tape, an Expect script can tell it to go on (for example, if your tapes are physically longer than the backup program thinks), or it can, again, contact you for assistance.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using Expect With Other Programs

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Most of the examples in this chapter use programs that are common to all UNIX systems. But Expect is not restricted to these programs. You can apply Expect to other programs. Even (gasp!) programs that you have written.

In the previous section, I described how you might automate ftp. But if you have a different program on your system to do file transfer, that's fine. Expect can use it. Expect does not have a built-in mechanism for file transfer, remote login, rebooting your colleague's workstation, or a million other useful things. Instead, Expect uses whatever local utilities you already have. This means that you do not have to learn a new file transfer protocol or a new host-to-host security system. If you use ".rhosts", then that is what Expect will use when it does remote host operations. If you use Kerberos, then Expect will use that. And so on.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using Expect On UNIX

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I have mentioned a number of UNIX programs that can be controlled by Expect. Expect can run any UNIX program, not just interactive ones. So you can invoke programs like cat or ls if you need them too.

Expect can do just about anything that any shell script can do. But that is not the point of Expect. Indeed, Expect scripts are meant to be small. By wrapping badly behaving programs with Expect scripts, you can use them from other scripting languages, such as the shell. The resulting scripts behave just like any UNIX program. Users have no way of knowing that the programs are just scripts.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using Expect On Other Operating Systems

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect makes use of a number of features that are present in all UNIX systems. The family of standards known as "POSIX" describes most but not all of these features. So while Expect can run on any UNIX system, it may have trouble providing all of its features on non-UNIX systems that nonetheless claim strict POSIX compliance. While Expect works just fine on some non-UNIX POSIX systems and can work in a limited way on all POSIX systems, I prefer to be conservative in my claims, so I use the phrase "UNIX systems" when referring to the systems on which Expect runs.

Fortunately, Expect can be used to control other operating systems indirectly. Since Expect is capable of making network connections (through telnet, rlogin, tip, etc.), it can remotely contact other computers even while running on a UNIX computer. In this way, it is very common to use Expect scripts to control non-UNIX computers such as Lisp machines, PostScript printers, modems, pagers, etc.

Testing and setting modems and other network devices (routers, bridges, etc.) is a particularly popular use of Expect. It is possible to write scripts that regularly test sets of modems to make sure that they are functional and that previous users have not left them in a bad state. Such scripts can even simulate a real user—placing a phone call, connecting to a remote host, and even logging in. An Expect script can remember the port numbers and other trivia that users do not bother to note until they have scrolled off the screen. So if a line turns out to be faulty, the script can record this and other information to a log.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using Expect In Real Applications

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect is a real program used in real applications. In my own environment, my colleagues depend on many Expect scripts to do important and critical tasks. We also use Expect to whip up demos and to automate ad hoc programs.

I have already mentioned lots of other uses for Expect. Many more cannot be described without also going into detail about proprietary or one-of-a-kind programs that people are stuck with using every day. But I will mention some companies and examples of how each uses Expect to give you a warm fuzzy feeling that Expect is widely accepted.

3Com does software quality assurance with Expect. Silicon Graphics uses it to do network measurements such as echo response time using telnet. The World Bank uses it to automate file transfers and updates. IBM uses it as part of a tape backup production environment. HP uses it to automate queries to multiple heterogenous commercial online databases. Sun uses it to sweep across their in-house network testing computer security. Martin Marietta uses it to control and extract usage statistics from network routers. Tektronix uses it to test software simulations of test instruments. The National Cancer Institute uses it to administer accounts across multiple platforms. Cisco uses it for network control and testing. Xerox uses it for researcher collaboration. Motorola uses it to control and backup multiple commercial databases. Data General uses it for software quality engineering. The Baha'i World Centre uses it to automate and coordinate data collection and storage from different telephone exchange locations. Amdahl uses it to automatically retrieve stock prices. CenterLine Software uses it for software quality assurance. Encore Computer uses it to simulate 500 users logging into one system at the same time. AT&T uses it to copy files between internal hosts through a firewall to and from external Internet hosts. Sandia National Laboratories uses it to control unreliable processes that need to be watched constantly. Schlumberger uses it to manage a network environment including routers, repeaters, bridges, and terminal servers all from different manufacturers. ComputerVision built an automated testbed using it. This is just a fraction of some of the users and uses of Expect. The whole list is truly astonishing, and it continues to grow rapidly.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Using Expect In Commercial Applications—Legalese

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It is not necessary to license Expect. Most of Expect is in the public domain, but two parts of it have copyrights. The Tcl core and the regular expression engine inside Expect are copyrighted but otherwise freely available, allowing the software to be used for any purpose and without fee, and with few restrictions on the code other than maintaining the copyright internally. The full copyright notices follow:

• Copyright (c) 1987-1993 The Regents of the University of California. All rights reserved.
• Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute this software and its documentation for any purpose, provided that the above copyright notice and the following two paragraphs appear in all copies of this software.
• In no event shall the University of California be liable to any party for direct, indirect, special, incidental, or consequential damages arising out of the use of this software and its documentation, even if the University of California has been advised of the possibility of such damage.
• The University of California specifically disclaims any warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The software provided hereunder is on an "as is" basis, and the University of California has no obligation to provide maintenance, support, updates, enhancements, or modifications.
• Copyright (c) 1986 by University of Toronto. Written by Henry Spencer. Not derived from licensed software.
• Permission is granted to anyone to use this software for any purpose on any computer system, and to redistribute it freely, subject to the following restrictions:

  1. The author is not responsible for the consequences of use of this software, no matter how awful, even if they arise from defects in it.
  2. The origin of this software must not be misrepresented, either by explicit claim or by omission.
  3. Altered versions must be plainly marked as such, and must not be misrepresented as being the original software.

The following notice is required on the NIST-authored portions of Expect:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Obtaining Expect and the Examples

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect may already be installed on your system, typically in /usr/local/bin. If you cannot find Expect, ask your system administrator. If you do not have a system administrator, you can obtain Expect by following the instructions below. Expect requires no special permissions to install, nor does it have to be installed in any particular place. You can even try it out in your own directory.

Expect includes a number of examples, several of which are useful as tools in their own right. Indeed, quite a few have man pages of their own and can be installed along with Expect. If the examples are not installed, you can find them in the example directory of the Expect distribution. Ask your local system administrator where the distribution is.

The examples provided with Expect are subject to change, but below is a list of just a few of the examples. The README file in the example directory contains a complete list as well as full explanations about each of the examples:

Example .

chess.exp	play one chess game against another
dislocate	allow disconnection from and reconnection to background processes
dvorak	emulate a Dvorak keyboard
ftp-rfc	retrieve an RFC from the Internet via anonymous ftp
kibitz	let several people control a program at the same time for remote assistance, group editing, etc.
lpunlock	unhang a printer waiting for a lock
mkpasswd	generate a good random password and optionally run passwd with it
passmass	set a password on many machines simultaneously
rftp	allow recursive get, put, and list from ftp
rlogin-cwd	rlogin with the same current working directory
rogue.exp	find a good game of rogue
timed-read	limit the amount of time a read from the shell can take
timed-run	limit the amount of time for which a program can run
tkpasswd	change passwords in a GUI
tknewsbiff	pop up a window (or play sounds, etc.) when news arrives in selected newsgroups
tkterm	emulate a terminal in a Tk text widget
unbuffer	disable output buffering that normally occurs when program output is redirected
weather	retrieve a weather forecast from the Internet

These and additional examples are available with the Expect distribution. The

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Expect And Tcl Resources

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This book contains a great deal of information about Expect and Tcl. Yet there are other resources that you may find useful. I will occasionally refer to some of these resources. The others are just for extra reading on your own.

The software for Expect, Tcl, and related packages include online manuals often called man pages. This is the definitive reference material. I will occasionally use the phrase Tcl reference material, for example, to refer to the Tcl man pages. As with most references, the reference material does not provide a lot of background or examples. Nevertheless, it is the most crucial documentation and therefore comes with the software itself.

I encourage you to use TkMan for reading man pages. Written by Tom Phelps at the University of California at Berkeley, TkMan provides an extremely pleasant GUI for browsing man pages. I cannot describe all the nice features of TkMan in this small space. Instead I will merely say that I now actually look forward to reading man pages as long as I can do it with TkMan. TkMan is written in Tcl and Tk and offers a splendid example of their power. Instructions on how to obtain TkMan can be found in the Tcl FAQ (see page 19).

Authoritatively written by the author of Tcl and Tk, John Ousterhout's Tcl and the Tk Toolkit (Addison-Wesley, 1994) is really four books in one, all written in a very readable and balanced style. Two of the books introduce Tcl and Tk. The other two describe how to write extensions for Tcl and Tk. If you find yourself writing many Expect scripts or becoming interested in applying Tcl to other projects, I strongly recommend you read this book.

Although Ousterhout's book does not cover all the features of Tcl and Tk, it nonetheless may be the place to turn for your questions left unanswered by Exploring Expect. For example, Tcl and the Tk Toolkit provide a more thorough treatment of some of the exotic features of Tcl. Ousterhout also provides a number of fascinating historical asides as well as some philosophical notes that contrast interestingly with my own.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Exercises

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

1. Using .netrc or << redirection in a shell script, have ftp retrieve a file. Make the script retry if the remote site is too busy but not if the file cannot be found.
2. Count the number of times that people have rewritten ftp to make it more flexible. Use Archie if you need help.
3. Count the number of programs you use that each have a different language for writing scripts or .rc files.
4. Find Expect's example directory online and try out some of the examples. Rewrite one in your favorite language.
5. Think about each keystroke that you press today. How much is the same from one session to the next? How much can be automated?
6. UNIX existed for over 20 years—without Expect. What did people do before? Which of those solutions still make sense today?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Chapter 2: Tcl—Introduction And Overview

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Expect does not have its own special-purpose language. Expect uses Tcl, a popular language for embedding in applications. Tcl provides lots of basic commands such as if/then/else, while, and set. Expect extends the language with commands such as expect and interact.

This chapter is an introduction and overview of Tcl. While not covering all of Tcl, this chapter does provide everything that the rest of the book depends on, and this is enough to write virtually any Expect script. Even if you already know Tcl, you may find it helpful to read this chapter. In this chapter, I will emphasize things about Tcl that you may not have thought much about before.

You probably want to get on with using Expect, and I can understand the urge to skip this chapter in the hopes of learning as little Tcl as possible so you can put Expect to work for you now. Please be patient and it will all fit together that much more easily.

If you do skip this chapter and you find yourself wondering about points in the other chapters, turn back to this chapter and read it.

A few concepts will not be covered here but will be explained as they are encountered for the first time in other chapters. The index can help you locate where each command is first defined and used.

I will occasionally mention when a particular Tcl command or feature is similar to C. It is not necessary that you know C in order to use Tcl, but if you do know it, such statements are clues that you can rely on what you already know from that language.

The types of variables are not declared in Tcl. There is no need since there is only one type: string. Every value is a string. Numbers are strings. Even commands and variables are strings! The following commands set the variable name to the string value "Don", the variable word to the value "foobar", and the variable pi to "3.14159".

set name Don
set word foobar
set pi 3.14159

Variable names, values, and commands are case sensitive. So the variable name is different than Name.

To access a variable's value, prefix the variable name with a dollar sign (

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Everything Is A String

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The types of variables are not declared in Tcl. There is no need since there is only one type: string. Every value is a string. Numbers are strings. Even commands and variables are strings! The following commands set the variable name to the string value "Don", the variable word to the value "foobar", and the variable pi to "3.14159".

set name Don
set word foobar
set pi 3.14159

Variable names, values, and commands are case sensitive. So the variable name is different than Name.

To access a variable's value, prefix the variable name with a dollar sign ($). The following command sets the variable phrase to foobar by retrieving it from the variable word.

set phrase $word

Variable substitutions can occur anywhere in a command, not just at the beginning of an argument. The following command sets the variable phrase2 to the string "word=foobar".

set phrase2 word=$word

You can insert a literal dollar sign by prefixing it with a backslash. The following command sets the variable money to the value "$1000".

set money \$1000

The backslash is also a useful way to embed other special characters in strings. For example, "\t" is a tab and "\b" is a backspace. Most of the Standard C backslash conventions are supported including \ followed by one to three octal digits and \x followed by any number of hex digits. I will mention more of these escapes later.

# stick control-Z in a variable
set controlZ \032
# define control-C
set controlC \x03
# define string with embedded control-Z and tab
set oddword foo\032bar\tgorp

A command beginning with "#" is a comment. It extends to the end of the line. You can think of "#" as a command whose arguments are discarded.

Multiple commands can be strung together if they are separated by a semicolon. A literal semicolon can be embedded by prefacing it with a backslash.

set word1 foo; set word2 bar   ;# two commands
set word3 foo\;bar             ;# one command

The ";#" sequence above is a common way of a tacking comments on the end of a line. The "

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Quoting Conventions

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Tcl separates arguments by whitespace (blanks, tabs, etc.). You can pass space characters by double quoting an argument. The quotes are not part of the argument; they just serve to keep it together. Tcl is similar to the shell in this respect.

set string1 "Hello world"

Double quotes prevent ";" from breaking things up.

set string2 "A semicolon ; is in here"

Keeping an argument together is all that double quotes do. Character sequences such as $, \t, and \b still behave as before.

set name "Sam"
set age 17
set word2 "My name is $name; my age is $age;"

After execution of these three commands, word2 is left with the value "My name is Sam; my age is 17;".

Notice that in the first command Sam was quoted, while in the second command 17 was not quoted, even though neither contained blanks. When arguments do not have blanks, you do not have to quote them but I often do anyway—they are easier to read. However, I do not bother to quote numbers because quoted numbers simply look funny. Anyway, numbers never contain whitespace.

You can actually have whitespace in command names and variable names in which case you need to quote them too. I will show how to do this later, but I recommend you avoid it if possible.

All commands return values. For example, the pid command returns the process id of the current process. To evaluate a command and use its return value as an argument or inside an argument, embed the command in brackets. The bracketed command is replaced by its return value.

set p "My pid is [pid]."

The set command returns its second argument. The following command sets b and a to 0.

set b "[set a 0]"

When it comes to deciding what are arguments, brackets are a special case. Tcl groups everything between matching brackets, so it is not necessary to quote arguments that only have spaces inside brackets. The following are all legal.

set b [set a 0]
set b "[set a 0]"
set b "[set a 0]hello world"
set b [set a 0]hello

After execution of this last command,

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Expressions

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Strings can be treated as expressions having mathematical values. For example, when the string "1+1" is treated as an expression, it has the value 2. Tcl does not automatically treat strings as expressions, but individual commands can. I will demonstrate this in a moment. For now, just remember that expressions are not complete commands in themselves.

Expressions are quite similar to those in the C language. The following are all valid expressions:

1 + 5
1+5
1+5.5
1e10+5.5
(5%3)*sin(4)
1 <= 2
(1 <= 2) || (2 != 3)

All of the usual mathematical operators are available including +, -, *, /, and % (modulo). Many functions exist such as sin, cos, log, and sqrt. Boolean operators include || (or), && (and), and ! (not). The usual comparison operators are available (<= (less than or equal), == (equal), != (not equal), etc.). They return 0 if the expression is false and 1 if it is true.

Whitespace may be used freely to enhance readability. A number of numeric forms are supported including scientific notation as well as octal (any number with a leading 0) and hexadecimal (any number with a leading 0x). Functions such as floor, ceil, and round convert from floating-point to integer notation.

Precedence and associativity follow C rules closely. For instance, the expression "1-2*3" is interpreted as "1-(2*3)" because multiplication is of higher precedence than addition. All binary operators at the same level of precedence are left-associative. This means, for instance, that the expression "1-2-3" is interpreted as "(1-2)-3". Since Tcl expressions rarely become complex, I will omit a lengthy discussion of the numerous levels of precedence, and instead note that you can always use parentheses to override a particular precedence or associativity. See the Tcl reference material for the complete list of operators and their precedences.

Variable values may also be used in expressions.

1 + $age
$argc < 10

Return values can be used in expressions using the bracket notation. For example, an expression to compare the current process id to 0 is:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Braces—Deferring Evaluation

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Braces are similar to double quotes. Both function as a grouping mechanism; however, braces defer any evaluation of brackets, dollar signs, and backslash sequences. In fact, braces defer everything.

set var1 "a$b[set c]\r"
set var2 {a$b[set c]\r}

After evaluation of these two commands, var1 contains an "a" followed by the values of b and c, terminated by a return character. The variable var2 contains the characters "a", "$", "b", "[", "s", "e", "t", " ", "c", "]", "\", and "r".

As with double quotes, the braces are not part of the argument they enclose. They just serve to group and defer. The primary use of braces is in writing control commands such as while loops, for loops, and procedures. These commands need to see the strings without having $ substitutions and bracket evaluations made.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Control Structures

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Control structures change the flow of control. Many of the control structures in Tcl are patterned directly after their C equivalents. Tcl gives you the power to write your own control structures, so if you do not like those of C, you may yet find happiness. I will not describe how to do it, but it is surprisingly easy. (The hard part is designing something that makes sense.)

The while command loops until an expression is false. It looks very similar to a while in the C language. The following while loop computes the factorial of the number stored in the variable count.

set total 1

while {$count > 0} {
    set total [expr $total * $count]
    set count [expr $count−1]
}

The body of the loop is composed of the two set commands between the braces. The body is executed as long as $count is greater than 0.

Taking a step back, the while command has two arguments. The first is the controlling expression. The second is the body. Notice that both arguments are enclosed in braces. That means that no $ substitutions or bracket evaluations are made. For instance, this while command literally gets the string "$count > 0" as its first argument. Similarly, for the body. So how does anything happen?

The answer is that the while command itself evaluates the expression. If true (nonzero), the while command evaluates the body. The while command then re-evaluates the expression. As long as the expression keeps re-evaluating to a nonzero value, the while command keeps re-evaluating the body.

It is useful to compare this with the set command. The set command does not do any evaluation of its second argument. Consider this command:

set count [expr $count−1]

The [expr ...] part is eval