Accelerate Model Training with PyTorch 2.X

10

Training with Multiple GPUs

Undoubtedly, the computing power provided by GPUs is one of the factors that’s responsible for boosting the deep learning area. If a single GPU device can accelerate the training process exceedingly, imagine what we can do with a multi-GPU environment.

In this chapter, we will show you how to use multiple GPUs to accelerate the training process. Before describing the code and launching procedure, we will dive into the characteristics and nuances of the multi-GPU environment.

Here is what you will learn as part of this chapter:

• The fundamentals of a multi-GPU environment
• How to distribute the training process among multiple GPUs
• NCCL, the default backend for distributed training on NVIDIA GPUs

Technical requirements ...