Exploring Expect

Tcl—A Little History

Underneath Expect is Tcl, a small and simple language designed to be embedded in applications. Much of the reason Expect appears so coherent and well thought out is actually due to Tcl.

Tcl is a tour de force. It is powerful yet elegant, drawing a fine line between primitives and extensibility, and between simplicity and overkill. Tcl allowed me to concentrate on the application requirements of Expect. Tcl will allow you to call and mix the Expect primitives in all sorts of interesting ways to control your applications.

The intent behind Tcl matches Expect’s philosophy perfectly. Expect doesn’t need a specialized language. Any generic but extensible control language would have sufficed. However, at the time that I was thinking about writing Expect, no such extensible control language existed. I was irritated at the thought of having to create a language specifically for such a simple application. There was simply no justification for designing yet another language.

I had been thinking about writing an Expect-like program after helping Scott Paisley write a program to automate the initial login and command in a telnet session. The program understood only a few simple commands. For instance, find waited for a single fixed string to arrive. Every session ended in a permanent sort of interact with a single telnet process. There were no variables and no flow control commands such as if. And the program used pipes instead of ptys. The program solved our immediate problem, but it had a lot of special-case coding and I thought it could be generalized. This was in 1987.

For a long time I considered borrowing a shell, integrating the Expect primitives into it, and then re-releasing it. However, shells are not intended to be used this way and I did not have any interest in maintaining a shell once I had stuck my hands in it. At that time, shells were renowned for being messy beasts. For example, the C shell was well known for not having consistent and robust parsing. And the Bourne shell had stimulated the Obfuscated C Code Contest—a contest that actually celebrates and revels in torturous code. I wanted something else.

I was fortunate to be able to attend the 1990 Winter USENIX Conference in Washington DC (January 22-26). I had been thinking about writing something like Expect for several years, and I decided to go there and ask some wizards about what they did for portable language facilities. To my delight, there was a talk at the conference addressing that very topic.

John Ousterhout, a professor at the University of California at Berkeley, had designed a language for embedding into applications—my very need! By the middle of the talk, I knew I wanted to try using it. At the end of talk, when he said it was freely available, I swooned.

Four days later, I had a copy of Tcl. I couldn’t believe it. It was not only everything John had promised, it was also easy to use and well documented. By February 7 (eight days after first downloading Tcl), I had a working, albeit primitive, Expect. It had the core of what Expect has today: send, expect, spawn, interact, plus a logging function. At that time, the compiled Tcl code was about 48K, while Expect added on another 12K. The idea of a control language being larger than the application seemed peculiar, but it worked too well to go back to the old ways of designing ad hoc interfaces.

Even with minimal functionality, it clearly suggested some interesting uses and I thought that it might be nice to inform others. I had a USENIX Call-For-Papers hanging next to my workstation. It said:

“The final deadline for receipt of submissions is February 7, 1990. Abstracts received after this deadline will not be considered.”

It was February 8, one day after the deadline. I sat down, banged out the requested extended abstract, and sent it in on the 9th.^[3] It was accepted with top marks by all the referees and appeared at the very next conference. No person was more astonished by this rapid publication describing a Tcl-based application than John. (John published another Tcl-related paper at the 1991 Winter USENIX Conference—leading to Tcl papers at three consecutive USENIX conferences and a high profile very early in its life.)

During 1990 and 1991, my local ftp server distributed over 2500 copies of Tcl, all for the purpose of running Expect. I like to think that Expect was a catalyst in the success of Tcl, but even without Expect, Tcl would have caught on eventually. Tcl is now used by thousands of applications and millions of users. Some of the Tcl extensions (Tk, in particular) stand on their own, and like Expect, allow people to do things more easily than before and in many cases things that they would never even have tried.

Since writing Expect, I’ve used Tcl to build several other applications and extensions. While none of them are as general purpose as Expect, the results are wonderful. Even for building specialized tools, Tcl is a joy to work with. I hope never to go back to yacc and lex again.

While the focus here is Expect, not Tcl, I believe that this book is worth reading even if you are learning Tcl for other reasons—perhaps contemplating putting it to use in your own applications. Expect is a good example of what can be accomplished with Tcl. Plus there are many techniques—such as the debugger and the signal handler—that while invented for Expect, can be applied to just about any other Tcl application. By reusing my efforts (and in some cases, avoiding my mistakes), it will be that much easier when designing your own Tcl-based applications.

---

^[3] The abstract was finished in a couple of hours, but I delayed a day to contemplate several different names for the software. While “Expect” has worked out well, another leading contender was “Sex” (for either “Super Exec” or “Send-Expect”) which was obviously much more memorable—a key factor for audience attendance in a multi-track conference—plus “intercourse with other programs” seemed reasonably descriptive of what the software did.