Hands-On Software Engineering with Python

Multiprocessing and HPC in Python

High-performance computing (HPC), quite simply, is the use of parallel processing during the execution of an application to spread the computational load across multiple processors, often across multiple machines. There are several MPC strategies to choose from, ranging from custom applications that leverage local multiprocessor computer architecture through to dedicated MPC systems, such as Hadoop or Apache Spark.

In this chapter, we will explore and apply different Python capabilities, building from executing a baseline algorithm against elements in a dataset one element at a time, and look at the following topics:

• Building parallel processing approaches that exploit locally available multiprocessor architectures, ...