Managing IMAP | O'Reilly Media

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

IMAP stands for Internet Mail Access Protocol. For much of the Internet email system administrator community, it also stands for flexibility, speed, and power. These attributes come from abilities like being able to store all a user’s mail centrally, not demand that she store copies of it on each workstation from which she wants to access her mail. IMAP users also store their mail in an arbitrary number of server-side mailboxes, each of which they can move messages into or out of with any IMAP client. When an IMAP user checks her mail, her client need only download some of the header for each message, not the entire message. When she sees messages in her index she wants to retrieve, she can decide which parts of the 13-part message she wants to download, and which she doesn’t. These are capabilities that no other standardized mail access protocol permits.

Before we dive into a more detailed discussion of IMAP, though, let’s talk about Internet mail in general. Much of this discussion is a definition of terms. In defining those terms, however, we’re discussing the language that is the bedrock of Internet electronic mail.

The Internet Mail Model, like the Internet itself, is a collection of standardized components all acting with a common goal. In the case of email, the goal is to provide the framework for carrying electronic messages between one user and another. Each of the end users may be on very different platforms. Their respective sites may have vast geographic, technological, and social differences. Those differences demand that the framework be at once both robust and flexible. The Internet’s email framework consists of agents, mailstores, and standards. It may help you to reference Figure 1-1 as you read the chapter. The figure shows how the agents, mailstores, and standards work together.

Figure 1-1: Email cycle of life

The software programs that handle Internet messages are called agents. There are three types of Internet messaging agents: the Mail Transport Agent (MTA), Mail Delivery Agent (MDA), and Mail User Agent (MUA).

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The Internet Mail Model, like the Internet itself, is a collection of standardized components all acting with a common goal. In the case of email, the goal is to provide the framework for carrying electronic messages between one user and another. Each of the end users may be on very different platforms. Their respective sites may have vast geographic, technological, and social differences. Those differences demand that the framework be at once both robust and flexible. The Internet’s email framework consists of agents, mailstores, and standards. It may help you to reference Figure 1-1 as you read the chapter. The figure shows how the agents, mailstores, and standards work together.

Figure 1-1: Email cycle of life

The software programs that handle Internet messages are called agents. There are three types of Internet messaging agents: the Mail Transport Agent (MTA), Mail Delivery Agent (MDA), and Mail User Agent (MUA).

MTA

An MTA (Figure 1-2 ) is a program that transmits and receives messages between messaging sites. The sending MTA accepts messages from end user client software and transmits it to a receiving MTA. The receiving MTA receives messages from the sending MTA, determines whether or not the recipient resides locally on the receiving MTA (server) system, and then hands off the message for delivery. If the message is destined for a user on the receiving MTA’s system, then the receiving MTA hands the message off to a Message Delivery Agent (MDA) such as /bin/mail. If the user is not on the local system, then the receiving MTA acts as a sending MTA to pass the message on to the MTA on the remote system.

Figure 1-2: The MTA

Figure 1-3 shows typical Internet message headers. Each “Received” header line represents transit through a separate MTA. MTAs do not touch the mailstore. They delegate that work to the MDA.

Figure 1-3: Typical Internet email message header

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Unlike closed commercial mail systems, Internet messaging is defined by a series of specifications that are free and open for all. Consequently, an Internet messaging system can be built using a variety of products from several vendors, with assuring that each product will interoperate with all the other products. This is especially important because the open standards defining the Internet itself make for a highly complex environment in which each component of a messaging model must know what to expect of the others. Communication standardization is the soul of the Internet.

The Internet repeatedly proves the fact that there is no problem so large that it cannot be solved by the principle of “divide and conquer.” A browse through some of the messaging-related RFCs from the early 1970s shows how early ARPAnet engineers struggled to send email back and forth. Early on, they relied on the weak model embraced by many modern-day closed-source solutions: email as a file-copying program. RFC 469 (circa 1973) kicks around the idea of an email infrastructure based on passing files around using FTP. Even in those early discussions, innovative ideas were alluded to, such as active links to other documents, redirection to central document repositories, permanent email archives, and content from arbitrary non-textual sources. Those ideas suggested the need for a hierarchy of standards and protocols.

Before too long, however, the problem of how to best implement email was divided and conquered. As we’ve seen, a special-purpose protocol (SMTP) was developed exclusively for transport of the messages from one place to another. Other protocols were developed for accessing the mail once it arrived at the destination mailstore (IMAP and POP). Standards were developed for the format of messages themselves and for encapsulating the payload of those messages (MIME). With the standards in hand, it was an entirely manageable task for the various parts that make up Internet email to interoperate, because the means of doing so was a widely discussed and published set of industry standards. System administrators no longer needed to worry about whether or not the sendmail, QMail, and Postfix MTAs would talk to each other. Nor did they need to concern themselves with whether any IMAP-compliant client would be able to retrieve mail from their UW IMAP, Cyrus, or proprietary IMAP server.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The stage is set, and now we’re ready to introduce the players.

Think about what happens when mail is sent by a user on a PC using Netscape to someone who uses the text-mode, Unix-based MUA, PINE. As seen in Figure 1-7 , the sender (Netscape user) sends the message by SMTP to the MTA running on the sender’s ISP’s mail server. Once the mail arrives at the mail server, the MTA asks the MDA to store the message in the local message store. When the recipient reads his mail, he runs PINE on the mail server itself, which views his INBOX as a local file. Although in the example, no network mail access protocol is involved while reading the mail, you might think of the mail access protocol in this case as a way of converting the mail into something the client can understand. In those terms, the mail access protocol can be thought of as being hardcoded in the client.

Figure 1-7: Mail routing example

An agent is a program that performs a task on behalf of a human user, either directly at the human’s behest or indirectly under instruction from another agent. Usually, a chain of agents work in concert to get the information from its point of origin to its intended destination. Unlike the previous examples, Internet email agents perform their duties in the open and usually with the full consent and support of all the participants. Here are some examples of some of the more popular Internet email agents. We’ll go into more detail about some of these later, but it’s always good to solidify theory with some real-world examples as soon as possible.

MTAs

Any discussion of Internet MTAs could also easily be entitled “Sendmail and some alternatives.” The old slogan that “nobody ever got fired for buying IBM” could be adapted equally well to sendmail. Leave it to the Internet community, though, not to leave well enough alone. Recently, viable contenders have come onto the scene, such as Qmail and Postfix.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter looks at what IMAP is and what distinguishes it from other mail access protocols. We discuss briefly where IMAP is now and where it’s headed.

IMAP is a way of accessing electronic mail that is stored on a central server.

Certainly, this statement is true, but there’s more to it than that. More precisely, IMAP is a way to retrieve messages from one or more mailboxes on a central server, without ever having to download a single message to local hard disk. The messages remain on the server at all times.

By design, IMAP was intended to provide the same level of functionality for mailbox and message access and management that exists with a mailbox located on a local hard drive. Consequently, IMAP has server operations, such as “search for messages matching such-and-such criteria,” that are normally associated with mail clients.

You can see the advantages of IMAP very clearly if you work from several computers (e.g., home computer, office computer, and laptop). With IMAP, you don’t have to wonder which computer you were on when you downloaded and read a given message. You know it’s still on the server.

With the right IMAP client, you can do all of the following:

Learn when new messages arrive in any of your mailboxes
Share your mailboxes with anyone or everyone
Move messages from one mailbox to another
Mark messages with flags (such as “Important”) that are preserved between IMAP sessions

Another distinguishing feature of IMAP is that, not only is your INBOX stored in a central location, but your mail folders (mailboxes, in IMAP parlance) are stored in a central location on the IMAP server as well. As long as you’ve got IMAP client software, not only can you read your incoming email from just about anywhere on the Net, but you can access all the mail you archived in your mailboxes, too.

One feature that sets IMAP apart is that it supports not one, but three interaction models.

IMAP has three models for interacting with your mail: offline, online, and disconnected. The three interaction models are at the core of why, on its own merit, IMAP is the most powerful and flexible mail access protocol. The models are formally defined in RFC 1733 (

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

IMAP is a way of accessing electronic mail that is stored on a central server.

Certainly, this statement is true, but there’s more to it than that. More precisely, IMAP is a way to retrieve messages from one or more mailboxes on a central server, without ever having to download a single message to local hard disk. The messages remain on the server at all times.

By design, IMAP was intended to provide the same level of functionality for mailbox and message access and management that exists with a mailbox located on a local hard drive. Consequently, IMAP has server operations, such as “search for messages matching such-and-such criteria,” that are normally associated with mail clients.

You can see the advantages of IMAP very clearly if you work from several computers (e.g., home computer, office computer, and laptop). With IMAP, you don’t have to wonder which computer you were on when you downloaded and read a given message. You know it’s still on the server.

With the right IMAP client, you can do all of the following:

Learn when new messages arrive in any of your mailboxes
Share your mailboxes with anyone or everyone
Move messages from one mailbox to another
Mark messages with flags (such as “Important”) that are preserved between IMAP sessions

Another distinguishing feature of IMAP is that, not only is your INBOX stored in a central location, but your mail folders (mailboxes, in IMAP parlance) are stored in a central location on the IMAP server as well. As long as you’ve got IMAP client software, not only can you read your incoming email from just about anywhere on the Net, but you can access all the mail you archived in your mailboxes, too.

One feature that sets IMAP apart is that it supports not one, but three interaction models.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

IMAP has three models for interacting with your mail: offline, online, and disconnected. The three interaction models are at the core of why, on its own merit, IMAP is the most powerful and flexible mail access protocol. The models are formally defined in RFC 1733 (Distributed Electronic Mail Models in IMAP4 ). Let’s take look at the three models.

In the online model, messages are left on the mail server and are manipulated remotely by mail client programs. The mail client maintains an open connection to the server for the duration of the session, that is, until the user decides to end the session.

Whether you prefer the online model has much to do with where you think the higher resource burden should be placed: on the client or on the server. If you believe, as we do, that resources are best centralized when possible, then you’re likely to prefer the online model. If you believe that centralized resources should be conserved whenever possible, even at the expense of a greater investment on the part of the user, then you’re likely to prefer the offline model. Watch out, though, because placing resource demands on the server requires attention and monitoring of system administrators lest things get out of hand. The online model can put a burden on memory and CPU resources if users tend to keep several mailboxes open, or if mail clients are poorly behaved in some way that results in multiple sessions per user.

In situations where many users share the same PC (such as a kiosk or computer lab), the online model is the only viable alternative. The same holds true in cases where one user reads mail from many different machines.

An advantage of the online model is that it works splendidly over slow or high-latency communication lines. Let’s take IMAP operating in online mode as an example. Early IMAP testing was over a 2,400 bps modem, and a design goal was that it be no less tolerable than running a host-based program over a dialup connection. In fact, IMAP ended up being more tolerable due to client-based caching of data (such as when the message browser redraws locally instead of over the wire). Even over a slow Internet connection, the online model lets a user manage a shockingly large amount of mail. Suppose a user receives a lot of mail and likes to keep all of it. He might easily have thousands of messages spread across hundreds of mailboxes. When that user dials in to his Internet service provider over a 56 Kbps connection, he definitely does not want to download all his mail in order to read just one particular mail message. Using the online model, he would simply connect to the server and select the message he needed. The only time required would be the time it takes the server to access the message and display a copy of the message in his mail client. Furthermore, with IMAP one can usually select to download an attachment

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The short answer is that IMAP solves many of the problems inherent in legacy mail systems. Since the earliest days of networking, the best and the brightest in the computer field have looked for successively better ways to corral individual correspondence into threads of dialog that could easily be used online. This section puts IMAP in the context of several legacy mail models and its competing mail access protocol, POP.

Host-based email was the original model for email. If you’ve been using email for many years, you probably started by using this model. In the host-based email model, the MUA, MTA, and mailstore reside on the same physical machine. Figure 2-1 illustrates the host-based email model.

Figure 2-1: Host-based email

Host-based email is a study in single-points-of-failure. Because all functions are carried out by a single system, divisions of responsibility between different parts of the system are unclear. If one part craters, there’s a much greater risk that it’ll cause the rest of the system to crater. On a host-based email system, the actions of any of the three essential components (MUA, MTA, and mailstore) impact the performance of the others. Host-based email systems are frequently general-purpose compute servers or Internet servers as well. Most enterprise mail systems that have been around for many years began in this model, the shortcomings of which necessitated the development of standardized mail access protocols.

Depending on how you approach it, the host-based email model is either the most or the least secure model around. Some might consider it the most secure because in a single draconian sweep, you clamp down on the security of the mailstore, transport agent, and user agent in one fell swoop. Of course, this is subject to the Primary Firewall Fallacy: a firewall tricks you into believing that everything on the outside is dirty and everything on the inside is clean. Although host-based email may give you a great deal of control over who can get to your mail jewels, from the outside it provides relatively little protection of your users from one another.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If you look at which mail access protocols have popular market share, there are only two players on the field: POP and IMAP. POP essentially created the market. It also created a need for a protocol with a great many more features. IMAP filled that need.

POP is the granddaddy of standardized mail access protocols. Although recently you could say that there was more POP client software than IMAP client software, now you can say only that there’s more well-implemented POP client software than well-implemented IMAP client software. Therein lies the continuing value of POP: its simplicity. POP is a humble protocol. It doesn’t keep track of a variety of message states, it doesn’t allow the user to search through her mailbox, and it doesn’t even facilitate storing messages in a number of mailboxes. It does one thing and one thing only: it makes the messages available to the user to download to her local machine on demand.

For some users, the limitations of POP are just fine. They use one and only one machine to read their mail, and they download all the messages from their mail server to that one machine and subsequently delete them from the server. In the early days of POP, users wanting to read their mail from more than one machine would have found themselves downloading duplicate copies for their email to each machine from which they wanted to view that mail. At some point, when they thought all the various machines had caught up with their new mail, they finally would delete the mail from the server. Fortunately, though, most modern POP server implementations offer the option of leaving mail on the server after it’s downloaded.

For POP users, storing mail in mailboxes means storing mail in mailboxes on the local MUA machine. Searching the mailbox likewise means searching on the client machine. Internet Service Providers (ISPs) for the most part have embraced POP as their mail access protocol of choice because of its modest demands on the mail server. POP is undemanding on the mail server because all the real work is done on your local machine. Mail storage, mail searching, and usually address books are all local data on your workstation.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In the not-so-recent past, IMAP was largely thought of as an overly complex method for managing email, and POP was considered just fine. However, nearly everyone supports or promises to support IMAP these days. That makes the size of the development investment in IMAP possibly second only to HTTP. You would think this would be a good thing. As you’ll read later, however, at least on the MUA side of the house, there’s some sloppy implementation going on.

This actually ends up being a good thing. In an open-standard marketplace, the more users who reject the one or two large-market-share apps in preference to the plethora of smaller-market apps, the more diversity there is in the market. Some of the best IMAP implementation going on right now is from companies you probably have never heard of before. If the one or two market leaders hadn’t given away their market share, the IMAP market would be as locked up as the PC operating system market.

With the added complexity of IMAP comes greater flexibility. With flexibility comes a broader base of technologies on which it can be implemented, ranging from interactive voice response to wireless personal digital assistants (PDAs). In one sense, the success of a standard might be measured by the number of related standards it inspires.

The glory days of IMAP are just beginning. Here’s a brief list of where we stand as of this writing:

Dozens of IMAP implementations exist for nearly every desktop and server platform.
IMAP is being used as a backend for web-based email interfaces, further adding to its ubiquity.
The IMAP client market for PDAs, notably Palm OS and Windows CE, is just starting to take off, with a handful of clients for each.
While users may not be storming the ISP Bastille asking for IMAP by name, the feature list they are demanding fits IMAP to a tee.
Commercial Internet messaging products now incorporate IMAP support as a core feature.
University messaging environments now regard IMAP as standard operating procedure.
IMAP is starting to be offered on architectures like cellular phones, but the jury is still out on how useful IMAP will be in those environments.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

For nearly a decade, there have been only two appreciable open source IMAP servers: the University of Washington’s IMAP server and Carnegie Mellon University’s Cyrus IMAP server. Both are robust, time- and user-tested servers with an install base to rival any commercial alternative.

We will go into more detail in later chapters about the interesting histories and specific features of each server, but let’s take a brief look at them for the time being.

The University of Washington server is the reference implementation of IMAP. It was written by Mark Crispin, the inventor of IMAP. It was started in 1988 at Stanford University as a C rewrite of the original Interlisp client and DEC-20 assembly language server. When Crispin changed jobs for the University of Washington late in 1988, the IMAP project went with him.

The University of Washington IMAP server strives for compatibility with existing Unix systems. If you’ve stored your mail in a given format on a Unix system over the course of the past few decades, chances are that the UW server can read that format out of the box.

The UW server has a number of interesting experimental extensions, such as multiple append and server-based sorting and threading. The UW server also supports an extensive list of international character sets.

The UW server is very modular—it is easy to add support for another mailbox format or SASL authenticator by writing a code module and relinking.

The CMU Cyrus Server is a component of a project called Project Cyrus. Started in 1994, the Cyrus project was started because the management overhead of running the existing proprietary system was getting to be too high. CMU was unable to keep up with client development in particular and wanted to use commercial off-the-shelf (COTS) mail clients. Project Cyrus was created to provide a next-generation messaging system that relied heavily on Internet standards and was highly scalable as well as modular, supported disconnected mode, and enabled freedom from legacy architecture. Cyrus was a rejection of the idea of basing an email system on software that just copied files around from place to place in a filesystem. Historically, the lack of reliability of such systems has only been exceeded by their lack of scalability.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

All the standards and related documents in this section are Request For Comment (RFCs) documents. There is a plethora of IMAP-related Internet Drafts (would-be standards that are still working their way through the approval process), but they change so rapidly that they won’t be mentioned here. Everyone has favorite RFC archives. Here are ours. Each of the sites lets you choose the geographically closest archive from which to retrieve your documents:

Internet RFC documents (http://www.nexor.com/info/rfc/index/rfc.htm)
Internet Drafts (http://www.nexor.com/info/internet-drafts/id.html )

A word about these documents—they’re the epitome of “hit the ground running” docs. They’re meant to be terse and narrow, much like their Unix manpage cousins. Despite their laissez-faire name, RFCs are the canonical standards documents of the Internet. If you have a bet with someone and need an indisputable source to settle the argument, turn to the RFC.

Table 2-2 is a snapshot of the current RFC standards related to IMAP. For a more comprehensive list, you can do a database search on “IMAP” at one of the previously mentioned URLs. In Table 2-2, the most important document is the Internet Message Access Protocol Version 4rev1, RFC 2060, by Mark Crispin. Consider it the defining document of the core features of IMAP. Second behind it would be RFC 2683 (IMAP4 Implementation Recommendations), which is necessary to read to understand the IMAP “folklore.”

Table : IMAP-Related RFCs and Internet Drafts
RFC Number	Authors	Title
1731	J. Myers	IMAP4 Authentication Mechanisms
1732	M. Crispin	IMAP4 Compatibility with IMAP2 and IMAP2bis
1733	M. Crispin	Distributed Electronic Mail Models in IMAP4
2060	M. Crispin	Internet Message Access Protocol Version 4rev1
2061	M. Crispin	IMAP4 Compatibility with IMAP2bis
2086	J. Myers	IMAP4 ACL extension
2087	J. Myers	IMAP4 QUOTA extension
2088	J. Myers	IMAP4 non-synchronizing literals
2095	J. Klensin, R. Catoe, P. Krumviede	IMAP/POP AUTHorize Extension for Simple Challenge/Response
2177	B. Leiba	IMAP4 IDLE command

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter covers the conceptual middle ground between a layman’s understanding of the IMAP protocol and complete coverage, as contained in RFC 2060 (IMAP4rev1). This chapter will provide enough information to arm you to troubleshoot most IMAP problems and evaluate most clients, but not enough to write your own client or troubleshoot some of the stickiest dilemmas. For those situations, you would be much better off using the RFC 2060 documentation and the RFCs for the extensions your server professes to use.

This section covers how a client talks to a server, the details of what an IMAP session looks like, and how we captured that information on our network.

You may occasionally see IMAP referred to as a “line-oriented” protocol. All this means is that the conversation between the IMAP client and server is transmitted in the form of character strings that end with CRLF. Line-oriented protocol sessions are easy to follow: a command is sent as a line of text to the server, and the server returns its response as a line of text. Line-oriented protocols are easy to learn and understand for the very same reason.

In fact, commands that make up a line-oriented protocol are frequently so understandable that a user can use the commands to masquerade as an IMAP client. As an example, consider just about anyone with access to a version of telnet that lets you specify the target port. The Telnet protocol, by definition, uses TCP port 23, but most Telnet software can usually be directed at alternative ports. This means that a user can telnet to TCP port 25 and trick an SMTP server into interacting with him as if she were an SMTP client. It’s just as easy to for a user to telnet to TCP port 143 (IMAP) and interact with an IMAP server as if he were an IMAP client.

Why do we bother telling you this? Line-oriented protocol spoofing, apart from being a common hacking tool, is a helpful troubleshooting tool. If you’re having difficulty getting a given client and server to talk, one approach is to take the client out of the equation. If you spoof IMAP from a

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This section covers how a client talks to a server, the details of what an IMAP session looks like, and how we captured that information on our network.

You may occasionally see IMAP referred to as a “line-oriented” protocol. All this means is that the conversation between the IMAP client and server is transmitted in the form of character strings that end with CRLF. Line-oriented protocol sessions are easy to follow: a command is sent as a line of text to the server, and the server returns its response as a line of text. Line-oriented protocols are easy to learn and understand for the very same reason.

In fact, commands that make up a line-oriented protocol are frequently so understandable that a user can use the commands to masquerade as an IMAP client. As an example, consider just about anyone with access to a version of telnet that lets you specify the target port. The Telnet protocol, by definition, uses TCP port 23, but most Telnet software can usually be directed at alternative ports. This means that a user can telnet to TCP port 25 and trick an SMTP server into interacting with him as if she were an SMTP client. It’s just as easy to for a user to telnet to TCP port 143 (IMAP) and interact with an IMAP server as if he were an IMAP client.

Why do we bother telling you this? Line-oriented protocol spoofing, apart from being a common hacking tool, is a helpful troubleshooting tool. If you’re having difficulty getting a given client and server to talk, one approach is to take the client out of the equation. If you spoof IMAP from a telnet session, you can observe the commands being sent to and from the server and classify problems as either client or server problems.

The primary difference between IMAP and other line-oriented TCP protocols like SMTP, NNTP, and POP3 is that each command from the client is preceded by a short alphanumeric string, called a tag. The purpose of the tag is to help the IMAP client keep track of which response goes with which command. Once a connection is opened, all tags generated by the client must be unique until the connection is closed. When the server responds to a command, it attaches the tag that the client sent to its response, as illustrated in Figure 3-1 .

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Here we discuss the components of IMAP to prepare us for our blow-by-blow protocol trace later in the chapter.

As we saw in Chapter 2, What Is IMAP? , IMAP clients can operate in one of three modes: offline mode, online mode, or disconnected mode. While you’re not likely to see any IMAP operations take place between the client and server that say, “I’m an online session” or “I’m on offline session,” these modes are quite an important part of IMAP.

First of all, the session can be in one of four states. Commands and responses are, in most cases, valid only in certain states (e.g., you can’t issue the command to select a mailbox unless you’re in the authenticated state). Although two or more sessions with a given mailbox can each be in different states, no single session can be in more than one state simultaneously. For example, when your client initially connects to the server and you’re in the non-authenticated state, you authenticate successfully, leave the non-authenticated state, and enter the authenticated state. Here’s a description of the four states. A diagram of the relationship between the states is shown in Figure 3-2 .

Non-authenticated state: The non-authenticated state begins immediately when a connection starts.
Authenticated state: The authenticated state begins when the client authenticates successfully. It may also begin after an error in selecting a mailbox.
Selected state: The selected state begins once a mailbox has been selected successfully.
Logout state: The logout state begins when the client sends the LOGOUT command or the server unilaterally decides to close the connection (e.g., when the session reaches the inactivity time-out limit). The logout state lasts only long enough for the server to close the TCP connection.

Figure 3-2: IMAP state diagram

Although there are extensions to provide attributes to mailboxes (such as access control lists), in the core IMAP protocol all the interesting attributes are associated with the messages themselves, not the mailboxes.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

In this section we show and describe an actual IMAP session play-by-play. Our goal is to familiarize you with the most common IMAP operations. Knowing the operations will help you quickly troubleshoot problems independently of the client or server you’re working with.

In our examples, we use the tcpflow program (see Chapter 18, IMAP Tools , for information on where to obtain tcpflow) to examine the IMAP client-server interactions. tcpflow is a special protocol-analysis program that permits you to watch a conversation take place between a TCP-based client and server. tcpdump or your favorite protocol analysis program would work equally well.

Our session was generated in a Telnet session to the IMAP port on the server:

% telnet localhost imap

Commands are shown in bold, responses from the server are shown in plaintext:

               A00001 CAPABILITY
* CAPABILITY IMAP4REV1 MAILBOX-REFERRALS LOGIN-REFERRALS AUTH=CRAM-MD5
A00001 OK Completed

Each command is preceded by an arbitrary tag. As we mentioned earlier, the tag is an arbitrary string that the server “tags” its responses with. The purpose is to help the client keep track of which response goes with which command. Appendix C, IMAP Commands , provides a complete list of IMAP commands. The commands used in the sample session are, for the most part, self-explanatory. In any case, they’re discussed in the explanation following Example 3-1, which shows the client/server conversation during an actual IMAP session and discusses each request and response.

Example . Simple IMAP Session

ROOT@Server # tcpflow -c 'host Client and port 143'
tcpflow[3000]: listening on le0
Server: * OK Server Cyrus IMAP4 v1.5.19 server ready

Client: 00000000 CAPABILITY007
Server: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS \\
        X-NON-HIERARCHICAL-RENAME NO_ATOMIC_RENAME UNSELECT
        00000000 OK Completed

Client: 00000001 LOGIN dianna "xxxxxxxx"
Server: 00000001 OK User logged in

Client: 00000002 SELECT INBOX
Server: * FLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen)
        * OK [PERMANENTFLAGS (\\Answered \\Flagged \\Draft \\Deleted \\Seen \\*)]
        * 1 EXISTS
        * 0 RECENT
        * OK [UNSEEN 1]
        * OK [UIDVALIDITY 929804083]
        00000002 OK Completed

Client: 00000003 NOOP
Server: 00000003 OK Completed

Client: 00000004 FETCH 1 FLAGS
Server: * 1 FETCH (FLAGS (  ))
        00000004 OK Completed

Client: 00000005 FETCH 1 UID
Server: * 1 FETCH (UID 26888)
        00000005 OK Completed

Client: 00000006 FETCH 1 (ENVELOPE BODY.PEEK[HEADER.FIELDS \\
        (Path Message-ID Newsgroups Followup-To References)] INTERNALDATE \\
        RFC822.SIZE FLAGS)
Server: * 1 FETCH (FLAGS (  ) INTERNALDATE "5-Mar-2000 10:58:50 -0600" \\
        RFC822.SIZE 853 ENVELOPE ("Sun, 5 Mar 2000 11:00:52 -0600 (CST)" \\
        "Testing" ((NIL NIL "drm" "nec.unt.edu")) \\
        ((NIL NIL "drm" "nec.unt.edu")) ((NIL NIL "drm" "nec.unt.edu")) \\
        ((NIL NIL "dianna" "europa.acs.unt.edu")) NIL NIL NIL \\
        "<Pine.GS4.4.10.10003051100260.3034-100000@nec.unt.edu>") \\
        BODY[HEADER.FIELDS (Path Message-ID Newsgroups Followup-To References)] \\
        {70} Message-ID: Pine.GS4.4.10.10003051100260.3034-100000@nec.unt.edu 

        )
        00000006 OK Completed

Client: 00000007 FETCH 1 (BODYSTRUCTURE FLAGS)
Server: * 1 FETCH (FLAGS (  ) BODYSTRUCTURE ("TEXT" "PLAIN" \\
        ("CHARSET" "US-ASCII") NIL NIL "7BIT" 29 2 NIL NIL NIL))
        00000007 OK Completed

Client: 00000008 FETCH 1 BODY.PEEK[HEADER.FIELDS (Resent-Date Resent-From \\
        Resent-To Resent-cc Resent-Subject)]
Server: * 1 FETCH (BODY[HEADER.FIELDS (Resent-Date Resent-From Resent-To \\
        Resent-cc Resent-Subject)] {2}

        )
        00000008 OK Completed

Client: 00000009 FETCH 1 BODY[1]
Server: * 1 FETCH (FLAGS (\\Seen) BODY[1] {29}
        This is the message body.

        )
        00000009 OK Completed

Client: 0000000a STORE 1 +Flags (\\DELETED)
Server: * 1 FETCH (FLAGS (\\Deleted \\Seen))
        0000000a OK Completed

Client: 0000000b NOOP
Server: 0000000b OK Completed

Client: 0000000c EXPUNGE
Server: * 1 EXPUNGE
        * 0 EXISTS
        * 0 RECENT
        0000000c OK Completed

Client: 0000000d NOOP
Server: 0000000d OK Completed

Client: 0000000e LOGOUT
Server: * BYE LOGOUT received
        0000000e OK Completed

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Having an IMAP server is all fine and good, but it does you little good if you can’t actually read your mail. That, of course, is where the client comes in.

A microscopic examination of the strengths and weaknesses of each IMAP client on the market is somewhat beyond the scope of this book and beyond the capacity of a book publishing cycle, anyway. We’re currently on the threshold of an IMAP explosion in the Internet messaging industry. The features and bugs in any given IMAP client change with the phase of the moon. All that notwithstanding, here’s our version of a whirlwind tour of the best freely available (or at least freely demo-able) clients.

This section begins by listing and describing a set of features that might be found in the ideal IMAP client. The features are cross-referenced with popular IMAP clients.

We cover Windows, Mac, and Unix clients separately; the results of the evaluations are shown in Table 4-1 . We looked for the following features in each of the clients. We reviewed only whether the feature exists and actually works or not—not how well it works.

Free: Is the client available completely free of charge?
LDAP: Does the client support remote directory lookups via LDAP?
ACAP: Does the client support storage and retrieval of client options via ACAP?
IMSP: Does the client support storage and retrieval of client options via IMSP?
NNTP: Does the client support reading Usenet news via NNTP?
MIME: Does the client handle MIME attachments? In other words, does a MIME attachment show up as a legitimate attachment with some indication of its MIME subtype (text/plain, application/postscript, or audio/basic, for example)?
SSL: Does the client support SSL IMAP sessions? This of course is only useful if you’ve built SSL support into your IMAP server.
Kerberos v4: Does the client support Kerberos authentication?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This section begins by listing and describing a set of features that might be found in the ideal IMAP client. The features are cross-referenced with popular IMAP clients.

We cover Windows, Mac, and Unix clients separately; the results of the evaluations are shown in Table 4-1 . We looked for the following features in each of the clients. We reviewed only whether the feature exists and actually works or not—not how well it works.

Free

Is the client available completely free of charge?

LDAP

Does the client support remote directory lookups via LDAP?

ACAP

Does the client support storage and retrieval of client options via ACAP?

IMSP

Does the client support storage and retrieval of client options via IMSP?

NNTP

Does the client support reading Usenet news via NNTP?

MIME

Does the client handle MIME attachments? In other words, does a MIME attachment show up as a legitimate attachment with some indication of its MIME subtype (text/plain, application/postscript, or audio/basic, for example)?

SSL

Does the client support SSL IMAP sessions? This of course is only useful if you’ve built SSL support into your IMAP server.

Kerberos v4

Does the client support Kerberos authentication?

CRAM MD5

Does the client support CRAM message digests?

Disconnected

Does the client support disconnected mode? This item should be taken with a grain of salt. Some clients claim to support disconnected mode simply based on allowing a user to read mail while disconnected from the server. They do not, however, allow other basic actions (e.g., deleting messages) to be performed while the user is offline. In these cases, we gave the client a “no” on this point. The criteria that determine whether or not a client truly supports disconnected mode include:

The user can read messages while disconnected from the IMAP server.
Changes automatically synchronize when the user goes back online.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

PINE (Program for Internet News and Email) is a popular character-based IMAP client. PINE was developed at the University of Washington, home of IMAP itself. PINE is available for nearly all Unix platforms. It is also available for Windows machines in a version called PC PINE. PINE was designed to be easy to use. While it has retained its simplicity of operation over many years of development, it has also grown into a very highly configurable, robust client equally suited to both novices and power users. It’s worth mentioning that there are other character- based IMAP clients for Unix, but they are not covered here because none even comes close to matching PINE’s feature-richness. The Elm mailer was once widely preferred over PINE for its simple interface. Although it supports IMAP, it has come to be considered a holdover from the good old days of “old-school Unix.”

The Windows version of PINE, PC PINE, has all the features of Unix PINE, but runs under Windows or DOS. PC PINE has the added benefit of allowing you to attach files that reside on your computer’s local disk to email messages without the need to first upload those files to another host. PC PINE, although it doesn’t appear to on a first look, does support drag-and-drop attachments (try dragging a document into a Compose window).

The PINE home page is located at http://www.washington.edu/PINE/.

PINE supports the following Internet protocols and specifications: IMAP4, SMTP (Simple Mail Transport Protocol), NNTP (Network News Transport Protocol) for reading Usenet news within PINE, and MIME (Multipurpose Internet Mail Extensions). PINE has the option of supporting LDAP and Kerberos, if it’s built to include the appropriate libraries. Instructions for integrating LDAP and Kerberos are provided in the PINE online documentation.

PINE offers many features, including:

Personal address books.
Context-specific online help.
A command to grab an address from a message and add it to the address book.
A message composer with easy-to-use default editor and built-in spell checker. The composer expands addresses as they are entered and provides direct access to the address book.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Star Office is a freeware gem. It is an “office suite” of productivity applications, with file formats compatible with Microsoft Office applications. Star Office includes an email client, Star Mail, which supports IMAP. Star Office is free to private, non-commercial users and students. In addition to its native Windows 95/98/NT operating environment, Star Office is also available for other operating systems, including OS/2, Linux, and Solaris.

Star Mail is a component of the Star Office Suite and cannot be installed separately—if you want to use Star Mail, you have to install the whole Star Office Suite. That could be problematic if you’re low in disk space—the Star Office download for Windows is 80 MB; installed, it requires around 160 MB of disk.

The interface is somewhat non-intuitive to those who are regular users of Microsoft products. While this is not necessarily a liability, expect some users to encounter a slight learning curve over the first few times they use the interface.

For more information or to download a free copy of Star Office, visit http://www.sun.com/products/staroffice/.

Client-side filtering

Star Mail is one of the few IMAP clients that, at the time of this writing, provides client-side filtering.

Import filters

Star Office has tools for importing data from Messenger and Outlook Express.

Remote LDAP searching

Remote address book feature allows you to search Bigfoot, WhoWhere, and Switchboard, but not your own site’s LDAP directory.

Advanced searching

In addition to standard searching of message headers, Star Mail permits searches through levels of nested folders.

Performance enhancements for slow connections

To avoid wasting time downloading large messages, Star Mail allows the user to set a threshold on the size of messages and attachments. If the user attempts to open a message exceeding that threshold, he will be warned and asked whether to cancel the action or continue.

If you are running Star Office for the first time, it may come up in full-screen mode. If you prefer to run Star Office in a window, then press Alt-V to bring up the View menu and unselect “Full Screen.”

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Netscape Messenger is the mail client component of the popular Netscape Communicator package. Messenger’s popularity is due in large part to the fact that it’s part of the Communicator package—many Netscape users use it out of convenience, rather than for its merits. In any case, it’s a widely used client, and for that reason is mentioned here. All in all, Netscape Messenger is adequate for the majority of users who want a quick, easy, and free way to read their email and manage addresses. It lacks the functionality that power IMAP users look for in a mail client.

Netscape is available for a variety of platforms, including Windows 95/98/NT, Macintosh, and several flavors of Unix. Complete information and the latest version of the Communicator package are available at Netscape’s download site: http://download.netscape.com/.

Supports SSL

One of the most desirable features of Messenger is that it supports SSL. If your IMAP server is enhanced to support SSL, Netscape Messenger can be used to avoid sending passwords in cleartext over your network.

Folders are stored on the IMAP server

Unlike its primary competitor, Outlook Express, Netscape stores all folders on the IMAP server, including outgoing mail that is automatically saved into a “sent mail” folder and, also, drafts.

Performs well over slow connections

To improve performance over slow connections, Messenger allows the user to choose to download only messages that are below a certain message size threshold. Additionally, attachments are downloaded on demand, allowing a message with a large attachment to be downloaded quickly.

Superior LDAP support

LDAP support is excellent in Messenger. Messenger has a single interface for searching both personal and remote LDAP address books. The address book interface supports search not only by name, but by city, organization, phone number, and, believe it or not, by soundex. The client also supports integration of custom LDAP schema for searching. The user can drag and drop the results of an LDAP search into her local address book. LDAP directories can be replicated and used in offline mode, and Netscape claims that the local LDAP address book scales to over 200,000 names.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Outlook Express (OE) comes bundled with Microsoft’s Internet Explorer browser. Like Netscape Messenger, OE is popular by virtue of being free.

One drawback is that, in order to install Outlook Express, you’ll have to install the entire Internet Explorer (IE) package. That may be daunting to those who don’t have copious hard drive space to spare. The “standard” installation, which is the most basic installation you can use and still install OE, requires 72 MB of disk. The disk space required to run IE/OE once it’s installed is 47 MB.

Although many people prefer Netscape Messenger over OE, OE generally behaves better with IMAP servers. Overall, OE works well for users who are looking for a bare-bones IMAP client that is quick, intuitive, easy to set up for the first time, and easy to use. Internet Explorer and OE are available for Windows 95/98/NT, Macintosh, and Unix. To download a free copy of Outlook Express, visit http://www.microsoft.com/ie/.

Ease of use

Outlook Express is incredibly easy to install, set up, switch to from other email clients, and use. In fact, OE has a facility for importing mail client configurations and address books from other mail clients (the next section, “Configuration ,” shows you how).

Supports SSL

A great benefit of using Outlook Express is that it supports SSL. You may be concerned about sending passwords in cleartext over your network. Your users can use OE and connect via SSL to your SSL-enabled IMAP server—and passwords will cross the net encrypted.

Advanced searching

OE offers more than just simple searches on message headers. OE allows you to search for a message in a folder and all of its sub-folders. In addition to standard searching by message header fields (sender, recipient, subject line, date), you can also search the message body and certain types of message attachments for a text string.

Performance over slow connections

To improve performance, OE downloads mail in the background, leaving you free to do other tasks while your INBOX is being populated. You also have the option of setting a size threshold on messages to prevent long downloads if you’re on a slow connection.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Mulberry is a powerful IMAP client for Windows and Macintosh that fully implements the feature set of IMAP and the IMAP extensions. Mulberry is a product of Cyrusoft, Inc., the company co-founded by the former project manager of CMU’s Project Cyrus, Matt Wall, and Cyrus Daboo, the original developer of Mulberry. Cyrusoft’s philosophy focuses on true adherence to open standards. Cyrusoft seeks to provide a client that provides a full set of features that help the user read mail more productively and efficiently.

Mulberry is popular in the IMAP world, especially at educational institutions. It’s been in use at over 1,000 sites since 1995, in 22 countries, with over 300,000 users.

Cyrusoft has been very diligent in providing regular updates for both the Windows and Mac versions of their software simultaneously. They state their commitment to both versions up front, and they have historically stuck with it. You don’t have to worry that the Mac client will go away next year. Having both your Mac and Windows users on the same release of the software will make life easier for your end user support staff. Some of the future development plans for Mulberry include Unix clients (Solaris and Linux, initially), Palm OS clients, and Windows CE clients.

Mulberry is a hidden treasure. Mulberry performs well, is stable, and fully implements IMAP. Mulberry’s ACL editor is unmatched and is a valuable resource to Cyrus users. Once we got used to Mulberry’s interface, we were hooked.

Mulberry’s user interface has a different look and feel from your average Windows application, but it’s powerful, friendly, and easy to configure. Detailed information on Mulberry is available at Cyrusoft’s web site: http://www.cyrusoft.com/. Information at that site includes feature lists, free demo for download, FAQ, general information on IMAP, and Mulberry documentation for download.

Performance

Mulberry was designed not only to provide a pleasing user interface, but also to provide high performance. In fact, the folks at Cyrusoft claim that Mulberry is the “fastest IMAP client around,” with the fastest connect time, fastest download of MIME attachments, and fastest message display time of any client available. We didn’t do any quantitative benchmarks to confirm that claim, but from a strictly qualitative standpoint, Mulberry is definitely the fastest GUI client evaluated in this chapter. Mulberry minimizes memory and disk space requirements by implementing a “plug-in” architecture—users are given the option of discarding features they don’t use, thereby claiming back some disk space and memory overhead. Mulberry does not include a web browser, a news reader, or any other program that has nothing to do with email, which makes Mulberry smaller still in terms of disk space and memory requirements. To avoid wasting time downloading large messages, Mulberry allows the user to set a threshold on the size of messages and attachments. If the user attempts to open a message exceeding that threshold, she will be warned and asked whether to cancel the action or continue.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|

Table of Contents

MTA

MTAs

Client-side filtering

Import filters

Remote LDAP searching

Advanced searching

Performance enhancements for slow connections

Supports SSL

Folders are stored on the IMAP server

Performs well over slow connections

Superior LDAP support

Ease of use

Supports SSL

Advanced searching

Performance over slow connections

Performance