Distributed Machine Learning with Python

Chapter 7: Implementing Model Parallel Training and Serving Workflows

In this chapter, we will discuss how to implement a simple model parallelism pipeline. As opposed to data parallelism, where each GPU holds a full copy of a model, in model parallelism, we need to split a model properly among all GPUs in use.

Before diving into the details, we'll qualify our discussion with the following assumptions about both hardware and workload:

• We will use homogenous GPUs for the same model parallel training or serving job.
• Each model training or serving task will occupy the whole hardware exclusively, which means there will be no preemption or interruption during the running of our model training or serving task.
• For GPUs within a machine, they are ...