Distributed Computing with Python

Chapter 6. Python on an HPC Cluster

In this chapter, we are going to look at yet another way of deploying our distributed Python applications. The method we are going to look at is the use of a High Performance Computing (HPC) cluster (also called a supercomputer), such as the multimillion dollar (or Euros), room-filling machines commonly found in evil overlord lairs.

Real HPC clusters are typically found in universities and national labs and are not what a typical start-up or small company would operate, mostly due to their cost. They are generally quite large systems with possibly hundreds of thousands of CPU cores spread over thousands of machines.

Oftentimes, the maximum size of HPC clusters a center can afford is determined by the amount of ...