O'Reilly logo

Exploring Expect by Don Libes

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 1. Intro—What Is Expect?

Expect is a program to control interactive applications. These applications interactively prompt and expect a user to enter keystrokes in response. By using Expect, you can write simple scripts to automate these interactions. And using automated interactive programs, you will be able to solve problems that you never would have even considered before.

Expect can save you hours of drudgery, and this book will start you on your way. This first chapter is an overview of Expect. I will describe some simple applications and present some short scripts. However, the explanations are not intended to be complete but rather to whet your appetite and give you a taste of the good things to come. In the following chapters, I will revisit all of the concepts mentioned in this chapter and cover them in more detail.

Ouch, Those Programs Are Painful!

fsck, the UNIX file system check program, can be run from a shell script only with the -y or -n options. The manual defines the -y option as follows:

“Assume a yes response to all questions asked by fsck; this should be used with extreme caution, as it is a free license to continue, even after severe problems are encountered.”

The -n option is safer, but almost uselessly so. This kind of interface is inexcusably bad, yet many programs have the same style. ftp, a file transfer program, has an option that disables interactive prompting so that it can be run from a script. But it provides no way to take alternative action should an error occur.

Expect is a tool for controlling interactive programs. It solves the fsck problem, providing all the interactive functionality non-interactively. Expect is not specifically designed for fsck and can handle errors from ftp as well.

The problems with fsck and ftp illustrate a major limitation in the user interface offered by shells such as sh, csh, and others (which I will generically refer to as “the shell” from now on). The shell does not provide a way of reading output from and writing input to a program. This means the shell can run fsck, but only by missing out on some of its useful features. Some programs cannot be run at all from a shell script. For example, passwd cannot be run without a user interactively supplying the input. Similar programs that cannot be automated in a shell script are telnet, crypt, su, rlogin, and gdb. A large number of application programs are written with the same fault of demanding user input.

Expect was designed specifically to interact with interactive programs. An Expect programmer can write a script describing the dialogue. Then the Expect program can run the “interactive” program non-interactively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

Figure 1-1. 

A Very Brief Overview

Expect programs can be written in C or C++, but are almost always written using Tcl.[4] Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you will not have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, UNIX programs can be called (exec). I will provide a quick introduction to the language in Chapter 2 (p. 23).

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. The expect command describes a list of patterns to watch for. Each pattern is followed by an action. If the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings "welcome“, "failed“, or "busy“, and then it evaluates one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other pattern matches in a certain amount of time.

expect {
    "welcome"            break
    "failed"            abort
    timeout            abort
    "busy"        {
        puts "busy"
        continue
    }
}

A First Script—dialback

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument.

spawn tip modem
expect "connected"
send "ATD$argv\r"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with "login:“.

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60Kb executable (written in C) that did the same thing!

In Chapter 16 (p. 347), I will talk more about the dialback concept and show a different way to do it.

Total Automation

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line. Instead, it interactively prompts for it—twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that—automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern "password:“. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage-return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a non-interactive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command "expect eof" causes the script to wait for the end-of-file in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

Testing

Despite all the reasons against it, suppose you decide to make changes to the passwd source anyway. After recompiling, it is a good idea to test your changes, right? You want to make sure passwd operates correctly when used interactively. Oh, but you cannot test the old interactive half of your new passwd program in a simple shell script—that is the whole reason you modified it in the first place!

This idea of testing interactive programs for correct behavior is another reason why Expect is useful. Even if you never want to automate a program, you may want to test it. passwd is just one example. Your own programs are another. Suppose you write a program that responds immediately to each command or keystroke. You cannot test it simply by piping a file of commands at it. It may discard characters that arrive before they are wanted, it may want a terminal in raw mode, it may want keystrokes such as ^C to activate signals, or you may need to see its responses in order to know how to phrase each subsequent command.

For example, suppose you are writing a debugger. The debugger may lay out a program in memory differently each time the program is recompiled, but the debugger should otherwise function the same (apart from any bugs you are fixing). If you are trying to verify that the debugger correctly handles the symbol table, you might ask for the value of all variables, verifying that each value is the same whether asked by memory address or variable name.

There is no way to embed the commands in a script because the script itself must change each time as elements are laid down in memory differently. For example, gdb, the GNU debugger, accepts the command "print &var" to print the address of var. Here is what an interaction might look like.

(gdb) print &var
$1 = (int *) 0xe008

In response, gdb numbers the output and then prints an equal sign followed by the type and value. It is possible for Expect to ask for and then print the type and value with the following code:

send "print &var\r"
expect "0x*\r" {
    send_user "$expect_out(0,string)\n"
}

The pattern 0x*\r is a pattern that matches the output 0xe008 followed by a carriage return. The "*" in the pattern is a wildcard meaning “match anything”. This is a convenient shortcut to specifying patterns. Later on, I will demonstrate how to be more precise in what you are asking for.

Following the pattern is an action—triggered when the pattern matches. Here the action is just one command, but it could be more than one, even including another expect command.

send_user sends the quoted string back to the user rather than to gdb. The $ in the string indicates that a variable reference follows and that its value is to be substituted in the string. Specifically, the variable expect_out is an array that contains the results of the previous expect. In this case, the results are just what matched the beginning of the pattern "0x*" up to and including the return character.

Expect is useful for more than just testing a debugger. It can be used to test all of the same programs that it automates. For example, the script used to automate passwd can be extended to test it, checking passwd with regard to improper passwords, unusually slow response, signals, and other sorts of problematic behavior.

Differing Behavior When Running Non-Interactively

Some programs behave differently when run interactively and non-interactively, and they do so intentionally. For example, most programs prompt only when running interactively. Non-interactively, prompting is not needed.

A more serious problem occurs when dealing with programs that change the way they buffer output depending on whether they are running interactively or not. Programs using the standard I/O library provided by UNIX automatically buffer their output when running non-interactively. This causes problems when you need to see the output immediately. Expect can make the programs think they are running interactively, thereby resolving the buffering problem.

As another example, shells force the user to press control characters (^Z, ^C) and keywords (fg, bg) to switch jobs. These cannot be used from shell scripts. Similarly, the shell running non-interactively does not deal with history and other features designed solely for interactive use. This presents a similar problem as with passwd earlier. Namely, it is impossible to construct shell scripts which test certain shell behavior. The result is that these aspects of the shell will inevitably not be rigorously tested. Using Expect, it is possible to drive the shell using its interactive job control features. A spawned shell thinks it is running interactively and handles job control as usual.

Partial Automation

Expect’s interact command turns control of a process over to you so that you can type directly to the process instead of through send commands.

Consider fsck, the UNIX program I mentioned earlier, which checks file system consistency. fsck provides almost no way of answering questions in advance. About all you can say is “answer everything yes” or “answer everything no”.

The following fragment shows how a script can automatically answer some questions differently than others. The script begins by spawning fsck and then in a loop answering yes to one type of question and no to another. The \\ prevents the next character from being interpreted as a wildcard. In this example, the asterisk is a wild card but the question mark is not and matches a literal question mark.

while 1 {
    expect {
        eof                     {break}
        "UNREF FILE*CLEAR\\?"   {send "y\r"}
        "BAD INODE*FIX\\?"      {send "n\r"}
        "\\? "                  {interact +}
    }
}

The last question mark is a catch-all. If the script sees a question it does not understand, it executes the interact command, which passes control back to you. Your keystrokes go directly to fsck. When done, you can exit or return control to the script, here triggered by pressing the plus key. If you return control to the script, automated processing continues where it left off.

Without Expect, fsck can be run non-interactively only with very reduced functionality. It is barely programmable and yet it is the most critical of system administration tools. Many other tools have similarly deficient user interfaces. In fact, the large number of these is precisely what inspired the original development of Expect.

The interact command can be used to partially automate any program. Another popular use is for writing scripts that telnet through a number of hosts or front-ends, automatically handling protocols as encountered. When such a script finally reaches a point that you would like to take over, you can do so. For example, you could browse through remote library catalogs this way. Using Expect, scripts can make a number of different library systems seem like they are all connected rather than different and disconnected.

The interact command also provides arbitrarily complex macros. If you find yourself repeatedly typing out long names, you can create a short character sequence to type instead. A simple example is the following interact command, which sends the (meaningless but long and hard to type) string "set def qwk/term=vt100 yhoriz=200" when (the short and easy to type) "y2" is entered. This ability to abbreviate is useful when dealing with interactive programs that require you to enter the same gobbledegook over and over again.

interact {
    "y2" {send "set def qwk/term=vt100 yhoriz=200"}
    "~~d" {send [exec date]}
}

This fragment also sends the current date if you press "~~d“. Arbitrary actions can be invoked, including other expect and interact commands. This example uses the exec command to run the UNIX date command.

Macros can be used in the other direction, too. If the program you are dealing with prints things out that you do not want to see or want to see but in a different way, you can change the appearance entirely.

Dangerous, Unfriendly, Or Otherwise Unlikable User Interfaces

The interact command is just a shorthand for the simplest and most common types of filtering that can be done in interactions. It is possible to build arbitrarily sophisticated mechanisms using the tools in Expect.

For example, commands can be filtered out of interfaces to prevent users from entering commands that you would prefer they not enter. Programs can also be wrapped even more tightly. The adb program, for instance, can crash a UNIX system with a slip of the finger or, more likely, inexperience. You can prevent this from happening by securely wrapping the adb program in a script. Not only does this increase the safety of your system, but your system administrators no longer all have to be masters of its intricacies.

The UNIX dump program is another program with an unlikable interface. For example, dump often guesses incorrectly about the length of a tape and will prompt for a new tape even if one is not needed. An Expect script can be used to respond to dump so that it can continue automatically. A script can answer just this question or any question from dump.

Expect can, in general, be applied to interfaces that you simply do not like for whatever reason. For example, you might like to automate sequences of your favorite game, perhaps because you have long since mastered some of it and would like to practice the end game but without laboriously replaying the beginning each time.

It is possible to customize exactly how much of the underlying programs show through. You can even make interactions entirely invisible so that users do not have to be irritated by underlying programs that they have no interest in anyway. Their attitude is “Use any tool. Just get the job done.” And they are right.

Graphical Applications

Expect can be combined with Tk, a toolkit for the X Window System. The combination of Expect and Tk is called Expectk.[5] Using Expectk, it is possible to take an existing interactive application and give it a Motif-like X interface without changing any of the underlying program. No recompiling is necessary, and because the underlying programs are not changed, there is no need to retest them again. All your efforts can be focused on the graphical user interface. Making it look pretty is all you have to do.

Figure 1-2. 

While Expectk will allow you to build X applications, it is limited in the amount it can automate existing X applications. Currently, Expect can automate xterm and other applications that specifically provide automation support, but Expect cannot automate any arbitrary X application.

Expect is also limited in its knowledge of character graphics such as is done by Curses. Nonetheless, with a little scripting, testing and automation of character graphics can be accomplished, and in Chapter 19 (p. 453), I will describe one way of doing this. Ultimately, I believe that Expect will do best with this capability built in rather than provided via scripts. However, the requirements and interfaces are not obvious, and further experimentation and design is required. ExpecTerm is an example implementation of a built-in character-graphic capability based on an earlier version of Expect. ExpecTerm is available from the Tcl archive (see page 19).

A Little More About Tcl

Expect’s language is general-purpose. It has control structures and data structures so that you can do just about anything. This can make up for the lack of programmability in other programs. For example, most debuggers do not provide very sophisticated methods of programmed control. Typically, you cannot declare variables and are restricted to simple loops.

The core of Expect’s language facilities are provided by Tcl, a language used by many other tools. “Tcl” stands for Tool Command Language. It comes as a library intended to be embedded in applications, providing a generic language facility to any application. Tcl solves a long-standing problem in designing application languages.

Actually, most tools do not have very sophisticated internal control languages. The shell represents the extreme—it has a very rich language and can even call upon other programs. Start-up scripts such as .cshrc and .profile can be very complex. Few programs have such flexibility. And that is just as well. Such programs each have their own language and it is daunting to master each new one.

In the middle of the spectrum are programs such as ftp that have a limited ad hoc language—such as permitted in the file .netrc—not very flexible but better than nothing. A lot of programs are designed like ftp—their languages are limited, not extensible, and different from one tool to the next.

At the far end are programs such as telnet which have no control language whatsoever. The bulk of all interactive programs are like ftp and telnet—severely lacking in programmability.

This lack of programmability is understandable. Consider writing an application that controls the printer. It is not worth writing a big language that requires a scanner, a parser, symbol table, etc. The application alone might only be 10Kb. However, it is impossible to predict what uses you might put your program to in the future. Suppose you get another printer. Then your program needs to understand this. Suppose you give your program to someone else and they have two printers but they are of two different types. Suppose they want to reserve one printer for printouts of a special kind for a day—printing checks on payday, for example. You can see that the possibilities are endless. So rather than customizing the application each time and extending the language as required (if you have not boxed yourself into a corner already), it makes sense to use a general-purpose language from the beginning.

Programs that already use Tcl are all set. They have a nice language from the outset. But many tools we are blessed with were written before Tcl existed. And most of them will not be rewritten. Development of most of these programs has stopped long ago (although for $$$ you can buy new versions). Fortunately, Expect allows you to layer Tcl on top of any interactive program without making any changes whatsoever to the original program. In effect, Expect makes nonprogrammable programs programmable.

Job Control

Just as you personally can interact with multiple programs at the same time, so can Expect. Analogous to the way you can say fg and bg to switch between processes in a shell, Expect can also switch its attention. Of course, Expect does it a lot more quickly than you can. The result is that Expect can act as “glue” for programs that were never designed to operate with other programs. An amusing example is the original chess program distributed with V7 and BSD UNIX written by Ken Thompson. It was designed to interact with a user—it prompts for moves and echoes user input in algebraic notation. Unfortunately, it does not accept its own output as input. Even if it did, there is no way to pipe both inputs and outputs between two processes simultaneously from the shell.

With a few lines of Expect, it is possible to make one chess process play another, including both the communication and the massaging of the output so that it is acceptable as input. I will show this in Chapter 10 (p. 229), along with some more serious uses for these techniques.

Background Processes

Expect is useful for automating processes in the background such as through cron. By using Expect with cron, you can not only automate the interactions but also automate starting the process in the first place. For example, you might want to copy over files each night between two networked but otherwise unrelated systems. And you can be arbitrarily selective about choosing files. You might want to copy over only executables that are less than 1 Mb and were created on a Saturday unless the current load exceeds 2.5 and the previous transfer took less than 5 seconds, or some other such complicated set of rules. There are no ftp clients that have such complex front-ends. But you can write Expect scripts to make whatever decisions seem appropriate while driving ftp.

It is even possible to have a cron process interactively contact you while it is in the background. For example, an ftp script may need a password to continue what it is doing. It can search the network for you and then ask you for it. Or you can have it look for a set of users. After getting the password, Expect will go back and use it to complete the task. By having Expect query for passwords, you do not need to embed them in scripts.

Doing backups from cron is another common reason to use Expect. If the backup program needs another tape, an Expect script can tell it to go on (for example, if your tapes are physically longer than the backup program thinks), or it can, again, contact you for assistance.

Using Expect With Other Programs

Most of the examples in this chapter use programs that are common to all UNIX systems. But Expect is not restricted to these programs. You can apply Expect to other programs. Even (gasp!) programs that you have written.

In the previous section, I described how you might automate ftp. But if you have a different program on your system to do file transfer, that’s fine. Expect can use it. Expect does not have a built-in mechanism for file transfer, remote login, rebooting your colleague’s workstation, or a million other useful things. Instead, Expect uses whatever local utilities you already have. This means that you do not have to learn a new file transfer protocol or a new host-to-host security system. If you use ".rhosts“, then that is what Expect will use when it does remote host operations. If you use Kerberos, then Expect will use that. And so on.

Using Expect On UNIX

I have mentioned a number of UNIX programs that can be controlled by Expect. Expect can run any UNIX program, not just interactive ones. So you can invoke programs like cat or ls if you need them too.

Expect can do just about anything that any shell script can do. But that is not the point of Expect. Indeed, Expect scripts are meant to be small. By wrapping badly behaving programs with Expect scripts, you can use them from other scripting languages, such as the shell. The resulting scripts behave just like any UNIX program. Users have no way of knowing that the programs are just scripts.

Using Expect On Other Operating Systems

Expect makes use of a number of features that are present in all UNIX systems. The family of standards known as “POSIX” describes most but not all of these features. So while Expect can run on any UNIX system, it may have trouble providing all of its features on non-UNIX systems that nonetheless claim strict POSIX compliance. While Expect works just fine on some non-UNIX POSIX systems and can work in a limited way on all POSIX systems, I prefer to be conservative in my claims, so I use the phrase “UNIX systems” when referring to the systems on which Expect runs.

Fortunately, Expect can be used to control other operating systems indirectly. Since Expect is capable of making network connections (through telnet, rlogin, tip, etc.), it can remotely contact other computers even while running on a UNIX computer. In this way, it is very common to use Expect scripts to control non-UNIX computers such as Lisp machines, PostScript printers, modems, pagers, etc.

Testing and setting modems and other network devices (routers, bridges, etc.) is a particularly popular use of Expect. It is possible to write scripts that regularly test sets of modems to make sure that they are functional and that previous users have not left them in a bad state. Such scripts can even simulate a real user—placing a phone call, connecting to a remote host, and even logging in. An Expect script can remember the port numbers and other trivia that users do not bother to note until they have scrolled off the screen. So if a line turns out to be faulty, the script can record this and other information to a log.

Using Expect In Real Applications

Expect is a real program used in real applications. In my own environment, my colleagues depend on many Expect scripts to do important and critical tasks. We also use Expect to whip up demos and to automate ad hoc programs.

I have already mentioned lots of other uses for Expect. Many more cannot be described without also going into detail about proprietary or one-of-a-kind programs that people are stuck with using every day. But I will mention some companies and examples of how each uses Expect to give you a warm fuzzy feeling that Expect is widely accepted.

3Com does software quality assurance with Expect. Silicon Graphics uses it to do network measurements such as echo response time using telnet. The World Bank uses it to automate file transfers and updates. IBM uses it as part of a tape backup production environment. HP uses it to automate queries to multiple heterogenous commercial online databases. Sun uses it to sweep across their in-house network testing computer security. Martin Marietta uses it to control and extract usage statistics from network routers. Tektronix uses it to test software simulations of test instruments. The National Cancer Institute uses it to administer accounts across multiple platforms. Cisco uses it for network control and testing. Xerox uses it for researcher collaboration. Motorola uses it to control and backup multiple commercial databases. Data General uses it for software quality engineering. The Baha’i World Centre uses it to automate and coordinate data collection and storage from different telephone exchange locations. Amdahl uses it to automatically retrieve stock prices. CenterLine Software uses it for software quality assurance. Encore Computer uses it to simulate 500 users logging into one system at the same time. AT&T uses it to copy files between internal hosts through a firewall to and from external Internet hosts. Sandia National Laboratories uses it to control unreliable processes that need to be watched constantly. Schlumberger uses it to manage a network environment including routers, repeaters, bridges, and terminal servers all from different manufacturers. ComputerVision built an automated testbed using it. This is just a fraction of some of the users and uses of Expect. The whole list is truly astonishing, and it continues to grow rapidly.

In addition to this list of internal applications, companies such as IBM, AT&T, and Data General have incorporated Expect into applications that they sell. One of the best known commercial products utilizing Expect is DejaGnu, written by Rob Savoye for Cygnus Support. DejaGnu is a software testing utility designed to simplify the running of large collections of regression tests. Using Tcl and Expect, DejaGnu provides a framework in which tests can be written, run, and analyzed quickly and easily. Due to the power of Expect, DejaGnu is capable of testing both interactive and non-interactive applications, including embedded applications and applications on other hosts to support cross-platform and remote target board development. Cygnus has created freely available DejaGnu test suites for the GNU C and C++ compiler, debugger, assembler, and binary utilities. These test suites can be used to test any similar program or port of such a program, whether it is freely available or proprietary. According to the Free Software Report, Vol. 3, No. 1, “Cygnus supports over 70 platform configurations of the GNU compilers fully tested by DejaGnu. DejaGnu executes 8000 test cases in 16,000,000 documented tests for a typical release.”

Using Expect In Commercial Applications—Legalese

It is not necessary to license Expect. Most of Expect is in the public domain, but two parts of it have copyrights. The Tcl core and the regular expression engine inside Expect are copyrighted but otherwise freely available, allowing the software to be used for any purpose and without fee, and with few restrictions on the code other than maintaining the copyright internally. The full copyright notices follow:

  • Copyright (c) 1987-1993 The Regents of the University of California. All rights reserved.

  • Permission is hereby granted, without written agreement and without license or royalty fees, to use, copy, modify, and distribute this software and its documentation for any purpose, provided that the above copyright notice and the following two paragraphs appear in all copies of this software.

  • In no event shall the University of California be liable to any party for direct, indirect, special, incidental, or consequential damages arising out of the use of this software and its documentation, even if the University of California has been advised of the possibility of such damage.

  • The University of California specifically disclaims any warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The software provided hereunder is on an “as is” basis, and the University of California has no obligation to provide maintenance, support, updates, enhancements, or modifications.

  • Copyright (c) 1986 by University of Toronto. Written by Henry Spencer. Not derived from licensed software.

  • Permission is granted to anyone to use this software for any purpose on any computer system, and to redistribute it freely, subject to the following restrictions:

    1. The author is not responsible for the consequences of use of this software, no matter how awful, even if they arise from defects in it.

    2. The origin of this software must not be misrepresented, either by explicit claim or by omission.

    3. Altered versions must be plainly marked as such, and must not be misrepresented as being the original software.

The following notice is required on the NIST-authored portions of Expect:

  • This software was produced by the National Institute of Standards and Technology (NIST), an agency of the U.S. government, and by statute is not subject to copyright in the United States. Recipients of this software assume all responsibility associated with its operation, modification, maintenance, and subsequent redistribution.

So you can use Expect in other freely-redistributable or commercial packages. You can even use pieces of the code in other software products. Just don’t blame the authors for any problems encountered with the software.

Obtaining Expect and the Examples

Expect may already be installed on your system, typically in /usr/local/bin. If you cannot find Expect, ask your system administrator. If you do not have a system administrator, you can obtain Expect by following the instructions below. Expect requires no special permissions to install, nor does it have to be installed in any particular place. You can even try it out in your own directory.

Expect includes a number of examples, several of which are useful as tools in their own right. Indeed, quite a few have man pages of their own and can be installed along with Expect. If the examples are not installed, you can find them in the example directory of the Expect distribution. Ask your local system administrator where the distribution is.

The examples provided with Expect are subject to change, but below is a list of just a few of the examples. The README file in the example directory contains a complete list as well as full explanations about each of the examples:

Example 1-1. 

chess.exp

play one chess game against another

dislocate

allow disconnection from and reconnection to background processes

dvorak

emulate a Dvorak keyboard

ftp-rfc

retrieve an RFC from the Internet via anonymous ftp

kibitz

let several people control a program at the same time for remote assistance, group editing, etc.

lpunlock

unhang a printer waiting for a lock

mkpasswd

generate a good random password and optionally run passwd with it

passmass

set a password on many machines simultaneously

rftp

allow recursive get, put, and list from ftp

rlogin-cwd

rlogin with the same current working directory

rogue.exp

find a good game of rogue

timed-read

limit the amount of time a read from the shell can take

timed-run

limit the amount of time for which a program can run

tkpasswd

change passwords in a GUI

tknewsbiff

pop up a window (or play sounds, etc.) when news arrives in selected newsgroups

tkterm

emulate a terminal in a Tk text widget

unbuffer

disable output buffering that normally occurs when program output is redirected

weather

retrieve a weather forecast from the Internet

These and additional examples are available with the Expect distribution. The README file in the distribution also describes the location of the Expect archive which holds even more scripts. You can also contribute your own scripts to the archive. Particularly large or sophisticated applications (such as those which combine Expect with other extensions) can be found separately in the Tcl archive (see page 19).

There is a high probability that you already have Expect on your system. Expect is shipped by many vendors with their operating system utilities. Expect can also be found on many software distributions including GCT from Testing Foundations, USENET Software from UUNET, the Sun User Group CD-ROM, the Prime Time Freeware CD-ROM, the Lemis CD-ROM, and others. Even if you do not have one of these, some other user on your system may have already retrieved Expect from an Internet repository.

The entire Expect distribution can be obtained from many sites around the Internet. Your best bet to finding a nearby site is to use an Archie server. Archie maintains a database of thousands of computers across the Internet and what programs they are willing to supply. If you do not know how to use Archie, you can obtain instructions by sending email to archie@archie.sura.net with the word help as the contents of the message.[6]

As this book is being written, Archie reports at least one site in the following countries where Expect can be obtained:

Country

Internet site

Australia

minnie.cs.adfa.oz.au

Austria

ftp.tu-graz.ac.at

Canada

julian.uwo.ca

England

unix.hensa.ac.uk

France

ftp.imag.fr

Germany

ftp.informatik.tu-muenchen.de

Greece

pythia.csi.forth.gr

Ireland

walton.maths.tcd.ie

Israel

cs.huji.ac.il

Japan

akiu.gw.tohoku.ac.jp

Netherlands

svin02.info.win.tue.nl

Norway

ftp.eunet.no

Sweden

ftp.sunet.se

Switzerland

ftp.switch.ch

United States

ftp.cme.nist.gov

Currently, the site ftp.cme.nist.gov always contains the latest version of Expect (or a pointer to it). To retrieve it from there, you can use anonymous ftp or request automatic email delivery.

To get Expect from ftp.cme.nist.gov via ftp, retrieve the file pub/expect/README. This will tell you what other files you need to retrieve and what to do after that. Typically, you need a copy of Expect and a copy of Tcl. You may also want to get a copy of Tk. These can be found in the same directory as Expect.

If you are not directly on the Internet but can send mail, you can request email delivery of the files. Send a message to "library@cme.nist.gov“. The message body should be:

send pub/expect/README

The site ftp.cs.berkeley.edu always contains the latest releases of Tcl and Tk. This site permits anonymous ftp. If you would like to request email from that site, any of the other sites listed above, or any other anonymous ftp sites, you can use an "ftp by mail” agent, such as the one provided by Digital Equipment’s Western Research Laboratories. Complete instructions for using ftp-mail may be retrieved by emailing to . The subject should be "ftpmail" and the message body should be "help“.

Expect And Tcl Resources

This book contains a great deal of information about Expect and Tcl. Yet there are other resources that you may find useful. I will occasionally refer to some of these resources. The others are just for extra reading on your own.

Important Reading Material

The software for Expect, Tcl, and related packages include online manuals often called man pages. This is the definitive reference material. I will occasionally use the phrase Tcl reference material, for example, to refer to the Tcl man pages. As with most references, the reference material does not provide a lot of background or examples. Nevertheless, it is the most crucial documentation and therefore comes with the software itself.

I encourage you to use TkMan for reading man pages. Written by Tom Phelps at the University of California at Berkeley, TkMan provides an extremely pleasant GUI for browsing man pages. I cannot describe all the nice features of TkMan in this small space. Instead I will merely say that I now actually look forward to reading man pages as long as I can do it with TkMan. TkMan is written in Tcl and Tk and offers a splendid example of their power. Instructions on how to obtain TkMan can be found in the Tcl FAQ (see page 19).

Authoritatively written by the author of Tcl and Tk, John Ousterhout’s Tcl and the Tk Toolkit (Addison-Wesley, 1994) is really four books in one, all written in a very readable and balanced style. Two of the books introduce Tcl and Tk. The other two describe how to write extensions for Tcl and Tk. If you find yourself writing many Expect scripts or becoming interested in applying Tcl to other projects, I strongly recommend you read this book.

Although Ousterhout’s book does not cover all the features of Tcl and Tk, it nonetheless may be the place to turn for your questions left unanswered by Exploring Expect. For example, Tcl and the Tk Toolkit provide a more thorough treatment of some of the exotic features of Tcl. Ousterhout also provides a number of fascinating historical asides as well as some philosophical notes that contrast interestingly with my own.

Other Books

Software Solutions in C edited by Dale Schumacher (Academic Press, to appear) includes a chapter by Henry Spencer on the implementation of his regular expression pattern matcher which is used by Tcl and Expect. His explanation of how pattern matching is actually accomplished is lucid and fascinating. This book is intended for C programming experts, but it may provide additional insight on designing efficient patterns and otherwise using patterns effectively.

Practical Programming in Tcl and Tk by Brent Welch (Prentice Hall, to appear) focuses on the more useful parts of Tcl, Tk, and several important extensions. With his own perspective, Welch provides very good explanations of topics that have proven tricky to people even after reading Ousterhout’s book. Welch also illustrates Tcl scripting and C programming issues by way of numerous program fragments, providing many building blocks that can be used in your own applications.

Other Online Documentation

The Tcl Frequently Asked Questions List (FAQ) contains many common questions and answers that somehow do not belong in either the manual pages or books. For example, the FAQ contains lists of Tcl extensions, documents, ftp sites, and of course common questions and answers. The FAQ was created by Larry Virden and is available from the Tcl archive on the Internet site http://ftp.aud.alcatel.com as tcl/docs/tcl-faq.partXXX, where XXX represents part numbers and file types. The file Index in the same directory lists the literal file names. The Tcl archive is maintained by Sam Kimery.

The Tcl FAQ can also be found on a number of other Internet sites. For example, it can be found on rtfm.mit.edu, which contains many other FAQs. The FAQ is also available through World Wide Web (WWW) as "http://www.cis.ohio-state.edu:80/hypertext/faq/usenet/tcl-faq/top.html“. The World Wide Web also provides access to other information on Tcl and Tk. The link "http://www.sco.com/IXI/of_interest/tcl/Tcl.html" contains links to other Tcl material with a focus on World Wide Web-related information such as browsers and HTML converters. Created by Mike Hopkirk, this link contains much other interesting and useful information as well. Another useful link is "http://web.cs.ualberta.ca/~wade/HyperTcl/“. Created by Wade Holst, this link concentrates on Tcl extensions. Included are a jobs database and an idea database. You can register ideas that you are working on and read what others are doing with Tcl. Many other Tcl-related WWW pages can be found in the Tcl FAQ.

A large number of scholarly papers on Expect and Tcl have appeared in journals and conference proceedings. These papers are not useful for people writing simple Expect scripts. Most of these papers are intended for computer scientists and cover topics such as implementation, performance, and comparisons to other methodologies. A Tcl bibliography is available on the Internet site http://ftp.aud.alcatel.com in the directory tcl/docs. The same directory contains other miscellaneous documents such as quick reference cards and essays on miscellaneous topics.

Support

A number of companies and individuals sell support for Tcl. These are described in the Tcl FAQ. Cygnus Support and Computerized Processes Unlimited sell support for Expect as well, and it is likely that other companies and individuals would also offer support if approached. This is not to mean that you will need support if you use Expect; however, it is not uncommon to find that management requires software be commercially supported before it is acceptable. As an aside, it may well be cost effective to have a professional support service solve your problems for you. Support can include modifications at your request, round-the-clock consulting by phone, site visits, and other services.

Cygnus Support
1937 Landings Drive
Mountain View, CA  94043
+1 (415) 903-1400
info@cygnus.com

Computerized Processes Unlimited
4200 S. I-10 Service Rd., Suite 205
Metairie, LA 70006
+1 (504) 889-2784
info@cpu.com

Many questions can be also be answered with the help of the Usenet newsgroup comp.lang.tcl. This newsgroup contains announcements for new releases of Tcl and Tcl extensions. The newsgroup is the right place to post bug reports, fixes, observations, and, of course, humor. Many of the people who read it are experts at Tcl and Expect, and they will answer questions. Simple questions that can be found in a book or the FAQ are discouraged. But challenging problems or requests for advice are welcomed.

The comp.lang.tcl newsgroup can be subscribed to by mail. In addition, there are dozens of mailing lists on particular extensions and aspects of Tcl. All of these are documented in the FAQ.

Exercises

  1. Using .netrc or << redirection in a shell script, have ftp retrieve a file. Make the script retry if the remote site is too busy but not if the file cannot be found.

  2. Count the number of times that people have rewritten ftp to make it more flexible. Use Archie if you need help.

  3. Count the number of programs you use that each have a different language for writing scripts or .rc files.

  4. Find Expect’s example directory online and try out some of the examples. Rewrite one in your favorite language.

  5. Think about each keystroke that you press today. How much is the same from one session to the next? How much can be automated?

  6. UNIX existed for over 20 years—without Expect. What did people do before? Which of those solutions still make sense today?



[4] Tcl is pronounced “tickle”. Some people are uncomfortable using this particular word in a formal setting. In that case, I recommend either saying the letters (i.e., “tee cee ell”) or coming up with a name suitable to your audience (e.g., “macho stud language”). Whatever works for you.

[5] Tk is pronounced “tee kay’”. Expectk is pronounced “ek spec’ tee kay”.

[6] All hostnames and filenames in this chapter are subject to change.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required