qmail

Chapter 1. Internet Email

Despite being one of the oldest applications on the Internet, email remains the Net’s “killer application” for most users. For users’ email to be sent and delivered, two kinds of programs have to work together, a mail user agent (MUA)^[1] that a person uses to send and read mail, and a mail transfer agent (MTA) that moves the mail from server to server. Qmail is a modern MTA for Unix and Unix-like systems.

Before diving into the details of qmail, it’s a good idea to closely examine some of the basics of Internet email that apply to all MUAs and MTAs. Common terms like envelope and mailbox have special meanings in Internet mail parlance, and both the structure of mail messages and the path that messages take through the mail system are carefully defined. The essential documents are RFC 2821, which defines the Simple Mail Transfer Protocol (SMTP) used to move mail from one place to another, and RFC 2822, which defines the format of mail messages. These RFCs were published in April 2001, updating the original RFCs 821 and 822 published in 1982. (All RFCs are available online at http://www.rfc-editor.org.)

For many years, the only widely used MTA for Unix and Unix-like systems was the venerable sendmail, which has been around in one form or another for 20 years. As a result, many people assume that whatever sendmail does is correct, even when it disagrees with the RFCs or has unfortunate consequences. So even if you’re familiar with sendmail (indeed, especially if you’re familiar with sendmail), at least skim this chapter so we all can agree on our terminology.

Mail Basics

The Internet’s SMTP mail delivers a message from a sender to one or more recipients. The sender and recipients are usually people, but may also be mailing lists or other software agents. From the point of view of the mail system, the sender and each recipient are addresses. The message is a sequence of lines of text. (RFC 2821 uses the word “mailbox” as a synonym for “address” and “content” for the message.)

Addresses

All email addresses have the simple form local-part@domain. The domain, the part after the at-sign, indirectly identifies a host to which mail should be delivered (although the host rarely has the same name as the domain). The local-part, the part before the at-sign, identifies a mailbox within that domain.

The set of valid domains is maintained by the Internet’s Domain Name System (DNS). Every domain is a sequence of names separated by dots, such as example.com. The names in email domains consist of letters, digits, and hyphens. (If current efforts to internationalize domain names ever settle down, the set of valid characters will probably become larger.)

The local-part is interpreted only by the host that handles the address’s domain. In principle, the mailbox can contain any characters other than an at-sign and angle brackets, but in practice, it is usually limited to letters, digits, and a small set of punctuation such as dots, hyphens, and underscores. Upper- and lowercase letters are equivalent in domains. It’s up to the receiving mail host whether upper- and lowercase are equivalent in local parts, although most mail software including qmail treats them as equivalent.

Addresses appear in two different contexts: “envelope” data that is part of an SMTP transaction defined by RFC 2821, or in the header of a message defined by RFC 2822. In an SMTP envelope, addresses are always enclosed in angle brackets and do not use quoting characters or permit comments. In message headers, the address syntax is considerably more flexible. An address like "Fred.Smith"@example.com ( Fred Smith ) is valid in message headers but not in SMTP. (The form [e-mail] is valid in either.)^[2]

Envelopes

Every message handled by SMTP has an envelope containing the addresses of the sender and recipients). Often the envelope addresses match the addresses in the To: and From: headers in the message, but they don’t have to match. There are plenty of legitimate reasons why they might not.

The envelope sender address is primarily used as the place to send failure reports (usually called bounce messages) if message can’t be delivered. If the sender address is null (usually written in angle brackets as <>), any failure reports are discarded. Bounce messages are sent with null envelope senders to avoid mail loops if the bounce message can’t be delivered. The sender address doesn’t affect normal mail delivery.

The envelope recipient address(es) control where a message is to be delivered. Usually a message starts out with the envelope recipients matching the ones on the To: and Cc: lines, but as a message is routed through the network, the addresses change. If, for example, a message is sent to [e-mail] and [e-mail], the copy sent to the host handling example.com will only have able’s address in the envelope and the one sent to the host handling domain.com will only have baker’s address. In many cases a user will have a different internal than external address—for example, mail to [e-mail] is delivered to [e-mail], in which case the envelope recipient address is changed at the place where the mail is received for the original address and readdressed to the new one.

Messages

An Internet email message has a well specified format defined in RFC 2822. The message consists of lines of text, each ended by a carriage-return line-feed pair. All of the text must be seven-bit ASCII. (The 8BITMIME extension to SMTP permits characters with the high bit as well but still doesn’t permit arbitrary binary data. If you want to send binary material as email, you must encode it using MIME encodings.)

The first part of the message is the header. Each header line starts with a tag that says what kind of header it is, followed by a colon, usually some whitespace, and then the contents of the header line. If a header is too long to fit on one line, it can be split into multiple lines. The second and subsequent lines start with whitespace to identify them as continuations. Every message must have From: and Date: header lines, and most have other headers such as To:, Cc:, Subject:, and Received:. The contents of some headers (such as Date:) are in a strictly defined format, while the contents of others (such as Subject:) are entirely arbitrary.

Some mail programs are more careful than others to create correct headers. (Many, for example, put invalid time zones in Date: headers.) Qmail is quite careful when it creates headers at the time a new message is injected into the mail system, but doesn’t look at or change message headers on messages that are transported through the system. The only change it makes to existing messages is to add Received: and Delivered-To: headers at the top, to chronicle the message’s path through the system.

The headers are separated from the body of the message by an empty line. The body can contain any arbitrary text, subject to a rarely enforced limit of 998 characters per line. The message must end with CR/LF, that is, no partial line at the end.

Lines

Every line in a message must end with CR/LF, the two hex bytes 0D 0A. This simple sounding requirement has caused a remarkable amount of confusion and difficulty over the years. Different computer operating systems use different conventions for line endings. Some use CR/LF, including all of Microsoft’s systems and a string of predecessors from CP/M to the 1960s era TOPS-10. Unix and Unix-like systems use LF. Macintoshes use CR, just to be different.

Regardless of the local line-ending convention, messages sent and received via SMTP have to use CR/LF, and the MTA has to translate from local to CR/LF when sending mail and back from CR/LF to local when receiving mail. Unfortunately, a common bug in some MTAs has been to forget to make this translation, typically sending bare LFs rather than CR/LF. Furthermore, RFC 822 said nothing about what a bare CR or LF in a mail message means. Some MTAs (sendmail, notably) treat a bare LF the same as CR/LF. Others treat it as any other data character. Qmail rejects incoming SMTP mail containing a bare CR or LF on the theory that it’s impossible to tell what the sender’s intent was, and RFC 2822 agrees with qmail that a bare CR or LF is forbidden. (It’s easy enough to tweak qmail’s SMTP daemon to accept bare LF, of course, if you really want to. See Chapter 6.)

Mailstore

The mailstore is the place where messages live between the time that qmail or another MTA delivers them and the user picks them up. Often, it’s also the place where the user saves messages after reading them.

I divide mailstores into two varieties: transparent and opaque. A transparent mailstore is one that an MUA can directly access as files, while an opaque one requires a network protocol to access. (As you might expect, there’s considerable overlap between the two, with an MUA running on the system where the mail is stored using a user’s mailstore as transparent and one running on a PC elsewhere using the same mailstore as opaque.)

A mailstore has several jobs beyond receiving messages. It must:

• Maintain a little per-message status information, such as whether a message is read, answered, or deleted

• Make it possible to group messages into multiple folders

• Make it possible to delete messages and move them from folder to folder

Transparent Mailstore

Unix systems have had a variety of mailstore file formats over the years. The oldest and still most popular is mbox, a format invented in two minutes in the 1970s for an early Unix mail program, and largely unchanged since then. An mbox is just a text file with the messages one after another. Each message is preceded by a From line and followed by a blank line. The From line looks like this:

From [e-mail] Wed Oct 06 19:10:49 1999

The address is usually (but not always) the envelope sender of the following message, and the timestamp is the time the message was added to the mailbox. Although it’s easy to add a new message to an mbox, it’s difficult to manipulate messages in the middle of a mailbox, and sharing a mailbox reliably between two processes is very tricky due to problems with file locking on disks shared over a network. Mboxes have been surprisingly durable considering their nearly accidental origins and their drawbacks, discussed in more detail in Chapter 10.

The MH mail system, developed at the RAND corporation in the 1980s, used a more sophisticated mailstore that made each mailbox a directory, with each message a separate file in the directory. Separate files made it easier to move messages around within mailboxes but still didn’t solve the locking problems.

Qmail introduced Maildir, a mailbox format that uses three directories per mailbox to avoid any need for locking beyond what the operating system provides. Maildirs are covered in detail in Chapter 10.

Opaque Mailstore

Opaque mailstores became popular when PCs started to gain dial access to the Internet, and users started running mail programs on the PCs rather than using Telnet to connect to shared servers and running mail programs there. The two popular opaque schemes are Post Office Protocol (POP3 for Version 3), and Internet Message Access Protocol, (IMAP4, pronounced eye-map, for Version 4).

POP3

POP3 is by far the most popular scheme used to deliver mail to PC clients. It is a fairly simple scheme that lets client systems download mail messages from servers. A client program connects to the POP server, sends user and password information, and then usually downloads all the waiting mail and deletes it from the server. It is possible for the client to leave the mail on the server, for people who check their mail from multiple places and want to receive all the mail on their primary computer even if they’ve peeked at it from somewhere else. POP3 can also assign unique ID strings (UIDs) to messages so that client programs can check to see which messages on the server haven’t been seen before. (Despite these features, IMAP is usually better suited for people who read mail from more than one place.)

Qmail comes with a POP3 server that uses Maildirs for its internal mailstore. You can also use Qualcomm’s popular qpopper that uses mbox mailboxes or the POP server from the Courier mail package that uses Maildirs. See Chapter 13.

IMAP4

IMAP is a scheme that lets client software manipulate messages and mailboxes on the mail server. It is much more powerful than POP at the cost of being much more complex as well. The client can tell the IMAP server to copy messages in either direction between client and server, create folders, move messages among folders, search for text strings in messages and mailboxes, and just about any other function that a mail client could possibly do to a message or mailbox.

The goal of IMAP is to allow client programs to manipulate mailboxes on the server just as though they were on the client system. This makes it possible for users to leave all their mail on the server so that they see a consistent view of their mail no matter from where they check it.

Qmail does not come with an IMAP server, but several IMAP servers work with qmail. The original IMAP server from the University of Washington uses mbox mailboxes, while the Courier IMAP server, part of the Courier MTA package, and the newer binc IMAP server use Maildirs.

The Structure of Internet Mail

Now that we’ve seen all the pieces of Internet mail, let’s put them together and watch the typical path of a message as it’s sent from one person to another.

First the sender runs a MUA, such as Pine or Eudora, and creates the message. Then a click of the Send button (or the equivalent) starts it on its way by passing it to the MTA (most likely qmail if you’re reading this book), a process known as submitting the message. If the MUA is running on the same computer as the MTA, the MUA submits the message by running the MTA’s injection component with the message as an input file. If the MUA is running on a separate computer, such as a Windows PC, the MUA makes a network connection to the computer running the MTA, and transfers the message using SMTP or a minor variant of SMTP called SUBMIT that’s specifically intended for host-to-host message submission.

Either way, the MTA receives the message envelope with the sender and recipient addresses and the message text. Typically the MTA fixes up the header lines in the submitted message so that they comply with RFC 2822, then looks at the domain parts of each recipient address. If the domain is one that the MTA handles locally, the MTA can deliver the message immediately. In the more common case that it’s not, the MTA has to send the message over the Net.

To figure out where to send the message, the MTA consults the DNS. Every domain that receives mail has an MX (Mail eXchanger) record in DNS identifying the host that receives mail for the domain.^[3] Once the MTA has found the MX host for a domain, it opens an SMTP connection to the MX host and sends the message to it. In some cases, the MX host uses SMTP to forward the message again if, for example, the MX host is a firewall that passes mail between MTAs on a private network and the rest of the Internet.

Eventually the message arrives at a host where the MTA knows how to deliver mail to the recipient domain. Then the MTA looks at the local part of the recipient address to figure out where to deliver the mail. In the simple case that the address is a user mailbox, the MTA either deposits the message directly into the mailstore or, more likely, calls a local delivery agent program to deliver the mail. (On Unix, a popular local delivery agent is procmail, which does mail sorting and filtering as well as delivery.) Depending on the MUA that the recipient user has, the MUA may read the message directly from a transparent mailstore on the mail server, or use POP or IMAP to read the mail on a client PC.

A domain can have more than one MX record in its DNS. Each MX record contains a numeric value known as the preference or distance along with the name of a host. Sending systems try the MX host with the lowest distance first, and if that MX host can’t be contacted, successively higher distances until one answers or it runs out of MXes. If there are several MX hosts at the same distance, it tries them all in any order before going on to hosts at a higher distance. If the sending host can’t contact any of the MXes, it holds onto the message and retries later.

When the Internet was less reliable, backup MXes with a higher distance than the main MX were useful to receive mail for a domain when the main MX was unavailable, and then send it to the main MX when it came back. Now, backup MXes are only marginally useful, because sending hosts retry mail for at least a few days before giving up. They wait until the main MX is available and then deliver the mail. Multiple MXes at the same distance are still quite useful for busy domains. Large ISPs often have a dozen or more MXes to share the incoming mail load.

---

^[1] Popular MUAs include pine and mutt on Unix systems, and Eudora, Netscape, Outlook, and Outlook Express on PCs and Macs.

^[2] Sendmail has often confused the two address contexts and has accepted message header formats in SMTP, both causing and masking a variety of bugs.

^[3] Well, they’re supposed to at least. For backward compatibility with pre-1980 mail systems, if a domain has no MX record but does have an A record containing a numeric IP address, the mail system uses that instead.