Dependable Computing

10Checkpointing Large‐Scale Systems

10.1 Introduction

This chapter focuses on checkpointing of high‐performance scientific applications running on large‐scale supercomputers, in which the number of processors is on the order of a hundred thousand. Although much work has been done in checkpointing for distributed applications, the high‐performance parallel‐computing community has not studied this problem in detail. While some tools for automated checkpointing were developed by CRAY for supercomputing applications based on synchronous checkpointing, such tools were of limited use due to their high custom design of the checkpointing/recovery support. Additionally, the diverse nature of the MPI support, network and storage support also makes it challenging to maintain and upgrade the checkpointing and recovery tools. Hence, the availability of transparent tools for challenging emerging applications is few and far between. However, that is changing with the advent of large‐scale parallel machines. In these machines, it is not uncommon to have a processor fail every few hours, which is much shorter than the running time of many computational science applications. Here, application‐level support for checkpointing with no modifications or OS‐level hooks is much more the state of practice. The focus is on checkpointing generally multi‐threaded applications and the goal is to ensure, to the extent possible, the checkpointing software is independent of the OS/MPI and other system facilities. ...