book

Distributed Machine Learning Patterns

Name: Distributed Machine Learning Patterns
Author: Yuan Tang
ISBN: 9781617299025

by Yuan Tang

January 2024

Intermediate to advanced

248 pages

6h 29m

English

Manning Publications

Audiobook available

Read now

Unlock full access

Distributed Machine Learning Patterns
Copyright
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this book?How this book is organized: A roadmapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1 Basic concepts and background
1 Introduction to distributed machine learning systems
1.1 Large-scale machine learning1.1.1 The growing scale1.1.2 What can we do?1.2 Distributed systems1.2.1 What is a distributed system?1.2.2 The complexity and patterns1.3 Distributed machine learning systems1.3.1 What is a distributed machine learning system?1.3.2 Are there similar patterns?1.3.3 When should we use a distributed machine learning system?1.3.4 When should we not use a distributed machine learning system?1.4 What we will learn in this bookSummary
Part 2 Patterns of distributed machine learning systems
2 Data ingestion patterns
2.1 What is data ingestion?2.2 The Fashion-MNIST dataset2.3 Batching pattern2.3.1 The problem: Performing expensive operations for Fashion MNIST dataset with limited memory2.3.2 The solution2.3.3 Discussion2.3.4 Exercises2.4 Sharding pattern: Splitting extremely large datasets among multiple machines2.4.1 The problem2.4.2 The solution2.4.3 Discussion2.4.4 Exercises2.5 Caching pattern2.5.1 The problem: Re-accessing previously used data for efficient multi-epoch model training2.5.2 The solution2.5.3 Discussion2.5.4 Exercises2.6 Answers to exercisesSection 2.3.4Section 2.4.4Section 2.5.4Summary
3 Distributed training patterns
3.1 What is distributed training?3.2 Parameter server pattern: Tagging entities in 8 million YouTube videos3.2.1 The problem3.2.2 The solution3.2.3 Discussion3.2.4 Exercises3.3 Collective communication pattern3.3.1 The problem: Improving performance when parameter servers become a bottleneck3.3.2 The solution3.3.3 Discussion3.3.4 Exercises3.4 Elasticity and fault-tolerance pattern3.4.1 The problem: Handling unexpected failures when training with limited computational resources3.4.2 The solution3.4.3 Discussion3.4.4 Exercises3.5 Answers to exercisesSection 3.2.4Section 3.3.4Section 3.4.4Summary
4 Model serving patterns
4.1 What is model serving?4.2 Replicated services pattern: Handling the growing number of serving requests4.2.1 The problem4.2.2 The solution4.2.3 Discussion4.2.4 Exercises4.3 Sharded services pattern4.3.1 The problem: Processing large model serving requests with high-resolution videos4.3.2 The solution4.3.3 Discussion4.3.4 Exercises4.4 The event-driven processing pattern4.4.1 The problem: Responding to model serving requests based on events4.4.2 The solution4.4.3 Discussion4.4.4 Exercises4.5 Answers to exercisesSection 4.2Section 4.3Section 4.4Summary

5 Workflow patterns
5.1 What is workflow?5.2 Fan-in and fan-out patterns: Composing complex machine learning workflows5.2.1 The problem5.2.2 The solution5.2.3 Discussion5.2.4 Exercises5.3 Synchronous and asynchronous patterns: Accelerating workflows with concurrency5.3.1 The problem5.3.2 The solution5.3.3 Discussion5.3.4 Exercises5.4 Step memoization pattern: Skipping redundant workloads via memoized steps5.4.1 The problem5.4.2 The solution5.4.3 Discussion5.4.4 Exercises5.5 Answers to exercisesSection 5.2Section 5.3Section 5.4Summary
6 Operation patterns
6.1 What are operations in machine learning systems?6.2 Scheduling patterns: Assigning resources effectively in a shared cluster6.2.1 The problem6.2.2 The solution6.2.3 Discussion6.2.4 Exercises6.3 Metadata pattern: Handle failures appropriately to minimize the negative effect on users6.3.1 The problem6.3.2 The solution6.3.3 Discussion6.3.4 Exercises6.4 Answers to exercisesSection 6.2Section 6.3Summary
Part 3 Building a distributed machine learning workflow
7 Project overview and system architecture
7.1 Project overview7.1.1 Project background7.1.2 System components7.2 Data ingestion7.2.1 The problem7.2.2 The solution7.2.3 Exercises7.3 Model training7.3.1 The problem7.3.2 The solution7.3.3 Exercises7.4 Model serving7.4.1 The problem7.4.2 The solution7.4.3 Exercises7.5 End-to-end workflow7.5.1 The problems7.5.2 The solutions7.5.3 Exercises7.6 Answers to exercisesSection 7.2Section 7.3Section 7.4Section 7.5Summary
8 Overview of relevant technologies
8.1 TensorFlow: The machine learning framework8.1.1 The basics8.1.2 Exercises8.2 Kubernetes: The distributed container orchestration system8.2.1 The basics8.2.2 Exercises8.3 Kubeflow: Machine learning workloads on Kubernetes8.3.1 The basics8.3.2 Exercises8.4 Argo Workflows: Container-native workflow engine8.4.1 The basics8.4.2 Exercises8.5 Answers to exercisesSection 8.1Section 8.2Section 8.3Section 8.4Summary
9 A complete implementation
9.1 Data ingestion9.1.1 Single-node data pipeline9.1.2 Distributed data pipeline9.2 Model training9.2.1 Model definition and single-node training9.2.2 Distributed model training9.2.3 Model selection9.3 Model serving9.3.1 Single-server model inference9.3.2 Replicated model servers9.4 The end-to-end workflow9.4.1 Sequential steps9.4.2 Step memoizationSummary
index

Content preview from Distributed Machine Learning Patterns

Part 1 Basic concepts and background

This part of the book will provide some background and concepts related to distributed machine learning systems. We will start by discussing the growing scale of machine learning applications (given users’ demand for faster responses to meet real-life requirements), machine learning pipelines, and model architectures. Then we will talk about what a distributed system is, describe its complexity, and introduce one concrete example pattern that’s often used in distributed systems.

In addition, we will discuss what distributed machine learning systems are, examine similar patterns that are often used in those systems, and talk about some real-life scenarios. At the end of this part, we will take a glance at what ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617299025Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Distributed Machine Learning Patterns

by Yuan Tang

Part 1 Basic concepts and background

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.