book

Distributed Machine Learning with Python

Name: Distributed Machine Learning with Python
Author: Guanhua Wang
ISBN: 9781801815697

by Guanhua Wang

April 2022

Intermediate to advanced

284 pages

5h 53m

English

Packt Publishing

Read now

Unlock full access

Distributed Machine Learning with Python
ContributorsAbout the authorAbout the reviewers
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Section 1 – Data Parallelism
Chapter 1: Splitting Input Data
Single-node training is too slowThe mismatch between data loading bandwidth and model training bandwidthSingle-node training time on popular datasetsAccelerating the training process with data parallelismData parallelism – the high-level bitsStochastic gradient descent Model synchronization Hyperparameter tuningGlobal batch sizeLearning rate adjustmentModel synchronization schemesSummary
Chapter 2: Parameter Server and All-Reduce
Technical requirementsParameter server architectureCommunication bottleneck in the parameter server architectureSharding the model among parameter serversImplementing the parameter serverDefining model layersDefining the parameter serverDefining the workerPassing data between the parameter server and workerIssues with the parameter server The parameter server architecture introduces a high coding complexity for practitionersAll-Reduce architectureReduceAll-Reduce Ring All-ReduceCollective communication BroadcastGatherAll-GatherSummary
Chapter 3: Building a Data Parallel Training and Serving Pipeline
Technical requirements The data parallel training pipeline in a nutshellInput pre-processing Input data partitionData loadingTrainingModel synchronizationModel updateSingle-machine multi-GPUs and multi-machine multi-GPUsSingle-machine multi-GPUMulti-machine multi-GPUCheckpointing and fault toleranceModel checkpointingLoad model checkpointsModel evaluation and hyperparameter tuningModel serving in data parallelismSummary
Chapter 4: Bottlenecks and Solutions
Communication bottlenecks in data parallel trainingAnalyzing the communication workloadsParameter server architectureThe All-Reduce architectureThe inefficiency of state-of-the-art communication schemesLeveraging idle links and host resourcesTree All-ReduceHybrid data transfer over PCIe and NVLinkOn-device memory bottlenecksRecomputation and quantizationRecomputationQuantizationSummary
Section 2 – Model Parallelism
Chapter 5: Splitting the Model
Technical requirementsSingle-node training error – out of memoryFine-tuning BERT on a single GPUTrying to pack a giant model inside one state-of-the-art GPUELMo, BERT, and GPTBasic conceptsRNNELMoBERTGPTPre-training and fine-tuningState-of-the-art hardwareP100, V100, and DGX-1NVLinkA100 and DGX-2NVSwitchSummary
Chapter 6: Pipeline Input and Layer Split
Vanilla model parallelism is inefficientForward propagationBackward propagationGPU idle time between forward and backward propagationPipeline inputPros and cons of pipeline parallelismAdvantages of pipeline parallelismDisadvantages of pipeline parallelismLayer splitNotes on intra-layer model parallelismSummary

Chapter 7: Implementing Model Parallel Training and Serving Workflows
Technical requirementsWrapping up the whole model parallelism pipelineA model parallel training overviewImplementing a model parallel training pipelineSpecifying communication protocol among GPUsModel parallel servingFine-tuning transformersHyperparameter tuning in model parallelismBalancing the workload among GPUsEnabling/disabling pipeline parallelismNLP model servingSummary
Chapter 8: Achieving Higher Throughput and Lower Latency
Technical requirementsFreezing layersFreezing layers during forward propagationReducing computation cost during forward propagationFreezing layers during backward propagationExploring memory and storage resourcesUnderstanding model decomposition and distillationModel decompositionModel distillationReducing bits in hardwareSummary
Section 3 – Advanced Parallelism Paradigms
Chapter 9: A Hybrid of Data and Model Parallelism
Technical requirementsCase study of Megatron-LMLayer split for model parallelismRow-wise trial-and-error approachColumn-wise trial-and-error approachCross-machine for data parallelismImplementation of Megatron-LMCase study of Mesh-TensorFlowImplementation of Mesh-TensorFlowPros and cons of Megatron-LM and Mesh-TensorFlowSummary
Chapter 10: Federated Learning and Edge Devices
Technical requirementsSharing knowledge without sharing dataRecapping the traditional data parallel model training paradigmNo input sharing among workersCommunicating gradients for collaborative learningCase study: TensorFlow FederatedRunning edge devices with TinyMLCase study: TensorFlow LiteSummary
Chapter 11: Elastic Model Training and Serving
Technical requirementsIntroducing adaptive model trainingTraditional data parallel training Adaptive model training in data parallelismAdaptive model training (AllReduce-based)Adaptive model training (parameter server-based)Traditional model-parallel model training paradigmAdaptive model training in model parallelismImplementing adaptive model training in the cloudElasticity in model inferenceServerlessSummary
Chapter 12: Advanced Techniques for Further Speed-Ups
Technical requirementsDebugging and performance analyticsGeneral concepts in the profiling resultsCommunication results analysisComputation results analysisJob migration and multiplexingJob migrationJob multiplexingModel training in a heterogeneous environmentSummary
Why subscribe?
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Distributed Machine Learning with Python

Chapter 6: Pipeline Input and Layer Split

In this chapter, we will continue our discussion about model parallelism. Compared to data parallelism, model parallelism training often takes more GPUs/accelerators. Thus, system efficiency plays an important role during model parallelism training and inference.

We limit our discussion with the following assumptions:

We assume the input data batches are the same size.
In multi-layer perceptrons (MLPs), we assume they can be calculated with general matrix multiply (GEMM) functions.
For each NLP job, we run it exclusively over a set of accelerators (for example, GPUs). This means there is no interference from other jobs.
For each NLP job, we use the same type of accelerator (for example, GPUs).
GPUs ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Interpretable Machine Learning with Python

Publisher Resources

ISBN: 9781801815697

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Distributed Machine Learning with Python

by Guanhua Wang

Chapter 6: Pipeline Input and Layer Split

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.