Classic Shell Scripting

How Big Can Unix Files Be?

The size of Unix files is normally constrained by two hard limits: the number of bits allocated in the inode entry to hold the file size in bytes, and the size of the filesystem itself. In addition, some Unix kernels have manager-settable limits on file sizes. The data structure used on most Unix filesystems to record the list of data blocks in a file imposes a limit of about 16.8 million blocks, where the block size is typically 1024 to 65,536 bytes, settable, and fixed at filesystem-creation time. Finally, the capacity of filesystem backup devices may impose further site-dependent limits.

Most current Unix filesystems use a 32-bit integer to hold the file size, and because the file-positioning system calls can move forward or backward in the file, that integer must be signed. Thus, the largest-possible file is 231 - 1 bytes, or about 2GB.^[19] Until about the early 1990s, most disks were smaller than that size, but disks containing 100GB or more became available by about 2000, and by combining multiple physical disks into a single logical disk, much larger filesystems are now feasible.

Unix vendors are gradually migrating to filesystems with 64-bit size fields, potentially supporting about 8 billion gigabytes. Just in case you think that might not be enough in the near future, consider that writing such a file once at the currently reasonable rate of 10MB/s would take more than 27,800 years! This migration is decidedly nontrivial because all existing software ...