Chapter 1
Parallel I/O at HPC Facilities
Galen Shipman
Oak Ridge National Laboratory
Modern high performance computing facilities deliver computational and data
resources to an ever broadening set of scientific and engineering domains.
What was once a cottage industry just two decades ago, HPC is now integral
to studies ranging from biology to materials science and everything in between.
These domains are using simulation at ever increasing scales driven by increas-
ingly higher resolution models with more accurate representation of physical
processes. This confluence has resulted in the build-out of extreme-scale HPC
facilities with compute platforms composed of hundreds of thousands of com-
pute cores, high performance networking infrastructures, and parallel I/O en-
vironments capable of scaling to terabytes/second of I/O bandwidth while
providing tens of petabytes of capacity.
To meet the required performance and capacity levels of modern HPC fa-
cilities, these large-scale parallel I/O environments may comprise thousands
or tens of thousands of hard disk drives, thousands of networking ports, and
several hundred storage servers. These components are then integrated into
usable systems through complex software stacks beginning with parallel file
system technologies such as IBM’s General Parallel File System (GPFS) ,
Lustre, Panasas, and the Parallel Virtual File System (PVFS). These parallel
file systems have come to be relied upon as both a reliable and high perfor-
mance persistent storage infrastructure. To maintain compatibility with the
broadest base of applications while enabling high performance, each of these
systems provides a POSIX interface while supporting concurrent access to
one or more files within the namespace from one or more compute nodes in
the HPC environment. This ability to provide parallel access while retaining
POSIX semantics has fueled the adoption of parallel file systems across HPC.
While POSIX provides a ubiquitous interface for applications, the appli-
cation developer is burdened with the increasingly complex task of map-
ping complex data structures that may span hundreds or even thousands
of distributed memory compute nodes into one or more files. Many applica-
tions therefore make use of higher-level I/O libraries such as the Hierarchical
Data Form (HDF), the Network Common Data Format (NetCDF), or the
3
4 High Performance Parallel I/O
Adaptable I/O System (ADIOS) to bridge this gap between complex dis-
tributed in-memory data structures to the underlying parallel file system.
These libraries provide both a higher-level semantic than a simple byte stream
while also optimizing parallel I/O operations to the underlying parallel file
system environment.
Complementing these high performance parallel I/O environments are
large-scale archival storage systems such the High Performance Storage Sys-
tem (HPSS). These systems provide HPC facilities with the ability to manage
hundreds of petabytes of tape and/or disk storage for long-term data storage.
These systems balance performance and capacity by supporting automated
tiering of data between tape and disk while providing high performance read-
ing and writing to and from the archive via parallel data movement nodes. In
some cases, these large-scale archives are closely integrated within the paral-
lel I/O environment through hierarchical storage management capabilities of
the parallel file system and the archival storage system allowing datasets to
automatically transition between these two distinct storage environments.
Part I of this book details the parallel I/O environments of a number of
representative high performance computing facilities. While the specific tech-
nologies employed may differ from facility to facility, a great degree of com-
monality exists across these environments. In Chapter 2 the National Energy
Research Scientific Computing Center (NERSC) facility is detailed including
their use of multiple parallel file system environments for large-scale com-
pute platforms and a more general-purpose center-wide storage environment.
Chapter 3 provides an overview of the National Center for Supercomputing
Applications (NCSA), the first site in the United States to achieve over 1
Terabyte/sec of sustained parallel I/O bandwidth. The Argonne Leadership
Computing Facility (ALCF) is detailed in Chapter 4, unique in its use of PVFS
at such a significant scale. Chapter 5 provides an overview of the Livermore
Computing Center notable for their work to integrate the ZFS file system
with the Lustre parallel file system. The Los Alamos National Laboratory
HPC facility is highlighted in Chapter 6, an early proponent of center-wide
accessible parallel file systems and the incorporation of burst buffers within
the parallel I/O environment. Finally, we conclude Part I on HPC facilities
with an overview the Texas Advanced Computing Center, an early adopter of
commodity storage systems and software-based RAID within the parallel I/O
environment.

Get High Performance Parallel I/O now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.