Exploring Expect

Total Automation

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line. Instead, it interactively prompts for it—twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that—automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern "password:“. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage-return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a non-interactive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command "expect eof" causes the script to wait for the end-of-file in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

Testing

Despite all the reasons against it, suppose you decide to make changes to the passwd source anyway. After recompiling, it is a good idea to test your changes, right? You want to make sure passwd operates correctly when used interactively. Oh, but you cannot test the old interactive half of your new passwd program in a simple shell script—that is the whole reason you modified it in the first place!

This idea of testing interactive programs for correct behavior is another reason why Expect is useful. Even if you never want to automate a program, you may want to test it. passwd is just one example. Your own programs are another. Suppose you write a program that responds immediately to each command or keystroke. You cannot test it simply by piping a file of commands at it. It may discard characters that arrive before they are wanted, it may want a terminal in raw mode, it may want keystrokes such as ^C to activate signals, or you may need to see its responses in order to know how to phrase each subsequent command.

For example, suppose you are writing a debugger. The debugger may lay out a program in memory differently each time the program is recompiled, but the debugger should otherwise function the same (apart from any bugs you are fixing). If you are trying to verify that the debugger correctly handles the symbol table, you might ask for the value of all variables, verifying that each value is the same whether asked by memory address or variable name.

There is no way to embed the commands in a script because the script itself must change each time as elements are laid down in memory differently. For example, gdb, the GNU debugger, accepts the command "print &var" to print the address of var. Here is what an interaction might look like.

(gdb) print &var
$1 = (int *) 0xe008

In response, gdb numbers the output and then prints an equal sign followed by the type and value. It is possible for Expect to ask for and then print the type and value with the following code:

send "print &var\r"
expect "0x*\r" {
    send_user "$expect_out(0,string)\n"
}

The pattern 0x*\r is a pattern that matches the output 0xe008 followed by a carriage return. The "*" in the pattern is a wildcard meaning “match anything”. This is a convenient shortcut to specifying patterns. Later on, I will demonstrate how to be more precise in what you are asking for.

Following the pattern is an action—triggered when the pattern matches. Here the action is just one command, but it could be more than one, even including another expect command.

send_user sends the quoted string back to the user rather than to gdb. The $ in the string indicates that a variable reference follows and that its value is to be substituted in the string. Specifically, the variable expect_out is an array that contains the results of the previous expect. In this case, the results are just what matched the beginning of the pattern "0x*" up to and including the return character.

Expect is useful for more than just testing a debugger. It can be used to test all of the same programs that it automates. For example, the script used to automate passwd can be extended to test it, checking passwd with regard to improper passwords, unusually slow response, signals, and other sorts of problematic behavior.