Deep Learning with PyTorch Quick Start Guide

Distributed environments

Sometimes, data and computing resources are not available on a single physical machine. This requires protocols for exchanging tensor data over a network. With distributed environments, where computations can occur on different kinds of physical hardware over a network, there are a large number of considerations—for example, network latencies or errors, processor availability, scheduling and timing issues, and competing processing resources. In an ANN, it is essential that calculations are produced in a certain order. The complex machinery for the assigning and timing of each computation across networks of machines and processors in each machine is, thankfully, largely hidden in PyTorch using higher-level interfaces. ...