In the sections above, I’ve tried to highlight some of the most important aspects of Mercurial’s design, to illustrate that it pays careful attention to reliability and performance. However, the attention to detail doesn’t stop there. There are a number of other aspects of Mercurial’s construction that I personally find interesting. I’ll detail a few of them here, separate from the “big ticket” items above, so that if you’re interested, you can gain a better idea of the amount of thinking that goes into a well-designed system.
When appropriate, Mercurial will store both snapshots and deltas in compressed form. It does this by always trying to compress a snapshot or delta, but only storing the compressed version if it’s smaller than the uncompressed version.
This means that Mercurial does “the right thing” when storing a file whose native form is compressed, such as a zip archive or a JPEG image. When these types of files are compressed a second time, the resulting file is usually bigger than the once-compressed form, and so Mercurial will store the plain zip or JPEG.
Deltas between revisions of a compressed file are usually larger than snapshots of the file, and Mercurial again does “the right thing” in these cases. It finds that such a delta exceeds the threshold at which it should store a complete snapshot of the file, so it stores the snapshot, again saving space compared to a naive delta-only approach.
When storing revisions on disk, Mercurial uses the
“deflate” compression algorithm (the same one used by
the popular zip archive format), which balances
good speed with a respectable compression ratio. However, when transmitting revision data over a
network connection, Mercurial uncompresses the compressed revision
data.
If the connection is over HTTP, Mercurial
recompresses the entire stream of data using a compression algorithm
that gives a better compression ratio (the Burrows-Wheeler algorithm
from the widely used bzip2 compression
package). This combination of algorithm and compression of the
entire stream (instead of a revision at a time) substantially
reduces the number of bytes to be transferred, yielding better
network performance over most kinds of network.
If the connection is over ssh, Mercurial doesn’t recompress the stream, because ssh can already do this itself. You can tell Mercurial to always use ssh’s compression feature by editing the .hgrc file in your home directory as follows:
[ui] ssh = ssh -C
Appending to files isn’t the whole story when it comes to guaranteeing that a reader won’t see a partial write. If you recall Figure 4-2, revisions in the changelog point to revisions in the manifest, and revisions in the manifest point to revisions in filelogs. This hierarchy is deliberate.
A writer starts a transaction by writing filelog and manifest data, and doesn’t write any changelog data until those are finished. A reader starts by reading changelog data, then manifest data, followed by filelog data.
Since the writer has always finished writing filelog and manifest data before it writes to the changelog, a reader will never read a pointer to a partially written manifest revision from the changelog, and it will never read a pointer to a partially written filelog revision from the manifest.
The read/write ordering and atomicity guarantees mean that Mercurial never needs to lock a repository when it’s reading data, even if the repository is being written to while the read is occurring. This has a big effect on scalability; you can have an arbitrary number of Mercurial processes safely reading data from a repository all at once, no matter whether it’s being written to or not.
The lockless nature of reading means that if you’re sharing a repository on a multi-user system, you don’t need to grant other local users permission to write to your repository in order for them to be able to clone it or pull changes from it; they only need read permission. (This is not a common feature among revision control systems, so don’t take it for granted! Most require readers to be able to lock a repository to access it safely, and this requires write permission on at least one directory, which of course makes for all kinds of nasty and annoying security and administrative problems.)
Mercurial uses locks to ensure that only one process can write to a repository at a time (the locking mechanism is safe even over filesystems that are notoriously hostile to locking, such as NFS). If a repository is locked, a writer will wait for a while to retry if the repository becomes unlocked, but if the repository remains locked for too long, the process attempting to write will time out after a while. This means that your daily automated scripts won’t get stuck forever and pile up if a system crashes unnoticed, for example. (Yes, the timeout is configurable, from zero to infinity.)
As with revision data, Mercurial doesn’t take a lock to read the dirstate file; it does acquire a lock to write it. To avoid the possibility of reading a partially written copy of the dirstate file, Mercurial writes to a file with a unique name in the same directory as the dirstate file, then renames the temporary file atomically to dirstate. The file named dirstate is thus guaranteed to be complete, not partially written.
Critical to Mercurial’s performance is the avoidance of seeks of the disk head, since any seek is far more expensive than even a comparatively large read operation.
This is why, for example, the dirstate is stored in a single file. If there were a dirstate file per directory that Mercurial tracked, the disk would seek once per directory. Instead, Mercurial reads the entire single dirstate file in one step.
Mercurial also uses a “copy on write” scheme when cloning a repository on local storage. Instead of copying every revlog file from the old repository into the new repository, it makes a “hard link,” which is a shorthand way to say “these two names point to the same file.” When Mercurial is about to write to one of a revlog’s files, it checks to see if the number of names pointing at the file is greater than one. If it is, more than one repository is using the file, so Mercurial makes a new copy of the file that is private to this repository.
A few revision control developers have pointed out that this idea of making a complete private copy of a file is not very efficient in its use of storage. While this is true, storage is cheap, and this method gives the highest performance while deferring most book-keeping to the operating system. An alternative scheme would most likely reduce performance and increase the complexity of the software, but speed and simplicity are key to the “feel” of day-to-day use.
Because Mercurial doesn’t force you to tell it when you’re modifying a file, it uses the dirstate to store some extra information so it can determine efficiently whether you have modified a file. For each file in the working directory, it stores the time that it last modified the file itself, and the size of the file at that time.
When you explicitly hg add, hg remove, hg rename, or hg copy files, Mercurial updates the dirstate so that it knows what to do with those files when you commit.
The dirstate helps Mercurial to efficiently check the status of files in a repository:
When Mercurial checks the state of a file in the working directory, it first checks a file’s modification time against the time in the dirstate that records when Mercurial last wrote the file. If the last modified time is the same as the time when Mercurial wrote the file, the file must not have been modified, so Mercurial does not need to check any further.
If the file’s size has changed, the file must have been modified. If the modification time has changed but the size has not, only then does Mercurial need to actually read the contents of the file to see if it has changed.
Storing the modification time and size dramatically reduces the number of read operations that Mercurial needs to perform when we run commands like hg status. This results in large performance improvements.
Get Mercurial: The Definitive Guide now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
